BooleanPerceptronClassifier with batchSize > 1 can silently lose weight updates, so it trains a wrong model

BooleanPerceptronClassifier trains in its constructor: every term starts with its total term frequency as its weight, each labelled training document is classified once, and the weights of a misclassified document's terms are corrected. The weights used for classification live in an FST that is rebuilt from a pending map. With batchSize > 1 the FST is rebuilt only when a misclassified document sits at position 0, batchSize, 2 x batchSize, ..., never after the last document, and each correction is computed from the last FST build and written over the pending value. So corrections after the last rebuild are dropped when training clears the map (always including every correction in the last batch except one made by its first document), whether an earlier one survives depends on whether a later, unrelated document at a batch boundary is misclassified, and several corrections of one term between two rebuilds keep only one. Trained on "bb" and "aa aa", both labelled false, with bias 1.5 and batchSize 2, the classifier still answers true for "aa aa" (score 0.81); batchSize 1 answers false. It is silent, deterministic, the same with assertions on and off, and present in every release that has the class (5.0.0 to 10.6.0: 21 executed, the others checked by their source, 10.6.0 included) and main. batchSize 1, the only value Lucene's tests use, is unaffected. Accumulating on the pending map and rebuilding at each batch end and once after the loop fixes it; the repro and the whole classification test suite pass with that change.

• BooleanPerceptronClassifier (lucene/classification, @lucene.experimental): a perceptron that assigns true or false to a text. assignClass(text) sums the weights of the text's tokens and answers output >= bias, with score 1 - exp(-|bias - output| / bias) (BooleanPerceptronClassifier.java:222-240).
• Training happens in the constructor (:81-168). Every term of the text field starts with its index-wide totalTermFreq as its weight (:116-123; like every term statistic it also counts unlabelled, filtered-out and deleted documents). The training documents, the hits of the class-field query that have a stored text and a stored class field (:130-148), are visited once, in hit order (with no training query, doc id order); each is classified, and a misclassified one moves the weights of its terms by its term frequency, towards the right answer, clamped at 0 (:140-166, :191-201).
• Pending map and FST. The weights live in a pending SortedMap ("TODO : remove this map as soon as we have a writable FST", :116), and classification reads an FST rebuilt from that map (updateFST, :208-220).

1. The training loop calls updateWeights only for a misclassified document and passes it the rebuild flag batchCount % batchSize == 0 (:155-162); batchCount counts every training document (:164). So the FST is rebuilt only when a misclassified document sits at position 0, batchSize, 2 x batchSize, ...: a boundary reached by a correctly classified document flushes nothing.
2. updateWeights computes each new weight from Util.get(fst, term), the last FST build, and writes it over the pending value (:196-200); it rebuilds only if the flag is set (:203-205). So several corrections of one term between two rebuilds keep only the last one.
3. Training ends with weights.clear(); // free memory while waiting for GC (:167) and no rebuild, so whatever is pending is dropped.
4. batchSize 1 rebuilds after every correction, so the FST always reflects the map (LFV-395's ordering defect aside) and nothing is lost. With batchSize >= the number of training documents, at most the first document's correction is applied (only if that document is misclassified).

• Who is exposed. Direct users of BooleanPerceptronClassifier who pass batchSize > 1, the value the parameter exists for. Nothing in Lucene constructs the class outside its tests, and before the 2021 repository split Solr's classification update processor offered only the k-NN and naive Bayes classifiers.
• Default vs opt-in. The public constructor has no default batchSize; the caller picks it. batchSize 1 is unaffected. No index-size prerequisite: two documents are enough.
• Silent and deterministic. The same index and settings always give the same model, with assertions on and off; when it is wrong, nothing reports it. Not every run loses a correction: in the finder's harness 314,860 of the 458,124 batchSize > 1 configurations differed from the reference. A one-pass perceptron is expected to be imperfect, so a worse model is easy to miss.
• How much is lost depends on the data. It is not a rule that a larger batch loses more: with "bb", "aa aa", "dd", "cc cc" (all false), assignClass("aa aa") is wrong with batchSize 2 and 4 but right with 3, where the misclassified "cc cc" at position 3 triggers a rebuild. In the finder's bounded-exhaustive harness (tiny training sets) the query checks whose class or score differs from a reference perceptron grow from 1,002,793 for batchSize 2 to 1,697,228 for 3 and 1,785,810 for 5, of 3,054,160 each; they count differing answers, not lost corrections. An audit probe with 12 short spam/ham training texts, StandardAnalyzer and bias 4 answered spam for 4 of its 10 queries ("meeting tomorrow", "team lunch", "project notes", "the meeting") with batchSize 8, where a classifier that applies every batch answers ham; with the fix all 10 agree.
• Cost. Measured on one synthetic 4,000-document probe only: stock batchSize > 1 trained in at most 0.54x of the batchSize 1 time (it skips rebuilds by losing updates), the fix in at most 0.76x (a shared, loaded host; relative order only; memory not measured). By construction both rebuild the FST at most 1 + ceil(N / batchSize) times for N training documents, against at most 1 + N for batchSize 1.
• How likely. Plausible but quiet: nobody has reported it since 5.0.0 (2015).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<false> but was:<true>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.classification.TestBooleanPerceptronBatchSize.testUpdateOfSecondDocumentInBatchIsLost(TestBooleanPerceptronBatchSize.java:55)

• The contract. "@param batchSize the size of the batch of docs to use for updating the perceptron weights" (BooleanPerceptronClassifier.java:74). The ordinary reading is that each batch's updates reach the weights; the javadoc does not say when, or what happens to a partial last batch.
• The design, from the 2013 commit. The introducing commit called the no-argument constructor "Default constructor, no batch updates of FST" (batchSize 1): batching was meant to group FST rebuilds, not to decide which updates count. The pending map is a staging copy ("TODO : remove this map as soon as we have a writable FST", :116), and weights.clear(); // free memory while waiting for GC (:167) treats it as already applied.
• The original patch applied every update. The only attachment of LUCENE-4818 (March 2013) read the pending weight from the map and rebuilt the FST after every update; batching, the read from the last FST build and the boundary-only rebuild arrived together in the commit.
• Nothing documents dropped updates. No javadoc, CHANGES or MIGRATE entry mentions it. All five constructions in Lucene's tests pass batchSize 1 (TestBooleanPerceptronClassifier.java:39, :54, :70, :85; TestConfusionMatrixGenerator.java:194), which is why it never showed.
• An unrelated later document decides whether a computed correction is kept. Whether the correction for "aa aa" reaches the model changes with the label of "cc cc": not because "cc cc" teaches anything about "aa", but because only its misclassification triggers a rebuild. A staleness trade-off would delay an update, not keep or drop it.

The strongest "works as designed" case, built by the intended-lens review (another model family) and the devil's advocate: the class is @lucene.experimental; the javadoc defines no mini-batch equation, no flush point and no rule for the last partial batch; batchSize could be read as a lazy FST refresh that accepts stale weights; a one-pass perceptron need not classify its training documents correctly; and only batchSize 1 was ever tested.

Why it fails: experimental status allows API changes, not a parameter that silently drops training signal. A lazy refresh would delay updates, not keep or drop one depending on a later document's label, and it would still refresh at the end. The repro compares with batchSize 1 on the same data, so it does not rely on training-set accuracy, and the expected answers hold under online and mini-batch rules alike. The missing tests explain why nobody noticed; they are not evidence of intent. Verdict: a defect (adjudication: confirmed, high; every verifier: real).

Documents inside a batch are classified with the FST as it stood at the batch start (mini-batch behaviour). The fix also moves the rebuild points: stock rebuilds during the correction of a misclassified document at position 0, b, 2b, ..., so that correction is already visible to the next document; the fix rebuilds after positions b - 1, 2b - 1, ... and once after the loop. batchSize 1 behaves exactly as before. A committer may instead prefer to drop batching or deprecate batchSize, since only batchSize 1 was ever tested. Workaround until a fix ships: pass batchSize 1. Side note, from reading the code only: batchSize 0 throws ArithmeticException and a null batchSize NullPointerException at the first misclassified document; a fix could validate batchSize >= 1.

In the training loop, mark a batch as pending when a correction is made and rebuild the FST when batchCount % batchSize == 0 after the increment, then once more after the loop if anything is pending. In updateWeights, read the previous weight from the pending map (Double previousValue = weights.get(termString);) instead of Util.get(fst, term), and drop its boolean updateFST parameter and its own rebuild. +13/-14 in BooleanPerceptronClassifier.java.

        if (modifier != 0D) {
          updateWeights(termVectors, scoreDoc.doc, assignedClass, weights, modifier);
          pending = true;
        }
        batchCount++;
        if (pending && batchCount % batchSize == 0) {
          updateFST(weights); // end of a batch: apply its accumulated updates
          pending = false;
        }
      }
    }
    if (pending) {
      updateFST(weights); // apply the last, partial batch
    }

Trade-offs

• Cost: at most one rebuild per batch that has a correction, the partial last batch included (at most 1 + ceil(N / batchSize) builds, against at most 1 + N for batchSize 1). On the 4,000-document probe it trained in at most 0.76x of the batchSize 1 time, slower than today's batchSize > 1 (at most 0.54x), which skips rebuilds by losing updates; one synthetic workload, memory not measured.
• The per-update clamp at 0 (LUCENE-6853) is kept; a committer may prefer to clamp once per batch, which the javadoc does not specify.
• Models trained with batchSize > 1 change; today's models are the wrong ones.

What the checks validated

• Scratch worktree of main a9779ef3c193, gradle: the new test passes 3 of 3; the whole :lucene:classification:test passes with assertions on and off (13 suites, 46 tests, 0 failures, 1 @Monster test skipped).
• :lucene:classification:check -x test passes (checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest).
• Standalone runner with the fix shadowed: this doc's test (-ea, -da), the finder's (2/2) and the independent reproducer's (3/3) pass; on 5.0.0 the same minimal fix in a shadowed class passes the finder's test.
• The finder's harness, re-run with the fixed class shadowed in, found the classifier matching its mini-batch reference (clamp at 0 per update) in all 12,216,640 checks.

Read the pending map and rebuild after every misclassified document, ignoring batchSize, as the original LUCENE-4818 patch did.

Trade-offs

Simplest, and identical to batchSize 1, which Lucene's tests cover. Every caller then pays one FST rebuild (linear in the vocabulary) per correction, the cost batching was added to avoid, and batchSize does nothing. Not run as a separate change.

Treat the class as batchSize 1 only.

Trade-offs

Fits an experimental class never tested with batchSize > 1. Deprecation alone changes no behaviour, and removing the parameter breaks source compatibility; callers get batchSize 1 behaviour (slower training, a correct model) only once a constructor without batchSize, or one that forces 1, exists and they move to it. Not run.

Add updateFST(weights) before weights.clear().

Trade-offs

Not recommended: it fixes the lost last batch and the later-document dependence, but not the overwrite of corrections inside a batch (test 3). Not run.

Method: semi-formal (the hunt's tier for tool-guided finds that are not a formal tool's own output): bounded-exhaustive translation validation, a hunt unit (classification, k-NN and utils against reference implementations), 2026-10-27, stock main jars.

• The harness ran the real BooleanPerceptronClassifier on every training set of 1-3 documents over a 3-word vocabulary plus 20,000 seeded sets of 4-7 documents, with batchSize 1, 2, 3 and 5, automatic and fixed bias, and 20 query texts (every combination of 1-3 vocabulary words plus one word not in the vocabulary): 12,216,640 checks.
• It compared each answer with a reference perceptron written from the class javadoc and with a transcription of the code's own update rule. batchSize 1 matched the reference everywhere; the transcription matched the real classifier everywhere; 314,860 of the 458,124 batchSize > 1 configurations differed from the reference.
• The signal came from the reference run, the root cause from the transcription. It was then reduced to a public-API JUnit test, and independent AI tools reproduced, reviewed and judged it (see the checks).