SimpleNaiveBayesDocumentClassifier applies a field boost such as title^10 only on its first call

SimpleNaiveBayesDocumentClassifier documents that its field names "can contain boosting indication e.g. title^10" (SimpleNaiveBayesDocumentClassifier.java:63-64). On every assignClass(Document) or getClasses(Document[, max]) call, analyzeSeedDocument splits "title^10" into the name and the boost, keeps the boost in a map made for that call, and then writes the bare name back with textFieldNames[i] = fieldName (:170). That array is the caller's own varargs array, which the parent class keeps by reference (SimpleNaiveBayesClassifier.java:95). From the second call on it holds "title", no ^ is left, and every boost is 1: the classifier silently returns exactly the unboosted scores and classes. On a six-document index the same document is fruit (0.99994) on call 1 and vehicle (0.55373) on call 2; on upstream's testBoostedDocumentClassification data (title^100) the class flips from batman to the video-game class. A second classifier built from the same String[] after that is unboosted even on its own first call. The write-back came with the class in LUCENE-6631, so every release from 6.0.0 to 10.5.2 and main is affected. Parsing each configured entry per call into local arrays, without writing back, fixes it; the whole lucene/classification test suite passes with that change.

• SimpleNaiveBayesDocumentClassifier (lucene/classification, @lucene.experimental): classifies a Document against an index whose documents carry a class field. For each class it adds, per configured field and per value of that field, the class's log prior plus the field's average word log-likelihood times the field's boost (SimpleNaiveBayesDocumentClassifier.java:118-131), then normalizes the scores to sum to 1.
• Field boost: the constructor's textFieldNames "can contain boosting indication e.g. title^10" (:63-64); title^10 multiplies the title's word log-likelihood by 10 (the class log-prior, added once per field value, is not boosted).
• textFieldNames: the varargs array passed to the constructor; the parent SimpleNaiveBayesClassifier keeps it as a protected final String[] without copying (SimpleNaiveBayesClassifier.java:55, :95).

1. Every assignClass(Document) and both getClasses(Document[, max]) overloads call assignNormClasses (:77-103), which creates two fresh maps for the call, tokens per field and boost per field (:108-109), and calls analyzeSeedDocument (:115).
2. analyzeSeedDocument (:148-172) reads textFieldNames[i] (:154); if it contains ^, it splits it into the name and the boost (:157-161); it stores tokens and boost under the bare name (:168-169) and then writes the bare name back: textFieldNames[i] = fieldName (:170). The write-back exists so that the scoring loop, which iterates textFieldNames again (:121), finds the bare name in the maps (:122, :128).
3. On the next call the array holds title: no ^, so the boost stays 1 (:155), and the title's log-likelihood is multiplied by 1 (:125-128). The result is exactly the unboosted classifier's.
4. The array is the caller's, so once the first classifier has classified a document, a second one built from the same String[] starts unboosted, and the caller sees its own configuration rewritten.

Index: three fruit documents (titles "apple banana", "apple cherry", "banana cherry"; bodies "sweet juice tree", "sweet garden tree", "juice garden") and three vehicle documents (titles "car truck", "truck bus", "bus car"; bodies "wheel engine road fuel", "engine wheel fuel road", "road engine wheel"). New document: title "apple banana cherry", body "engine wheel road fuel".

Per field (average word log-likelihood): title, fruit -1.386 vs vehicle -2.485 (fruit ahead by 1.099); body, fruit -2.741 vs vehicle -1.426 (vehicle ahead by 1.314); equal priors. Times 10 the title wins by 9.67, times 1 the body wins by 0.22.

Rows 1, 2 and 5 come from this doc's test on main and 10.5.2, assertions on and off. Call 3 and the array come from this doc's configuration probe on main and the independent reproducer's test on main, 10.5.2 and 9.12.3 (the array after one call). The upstream-fixture row ran on main (and earlier on 10.5.2).

• Who is exposed. Applications that use SimpleNaiveBayesDocumentClassifier with the documented field^boost syntax and classify more than one document with one instance, or build several classifiers from one String[]. Only a fresh classifier with a fresh array per document escapes, which is what the upstream tests do.
• Default vs opt-in. It needs the opt-in lucene-classification module, this one @lucene.experimental classifier and a configured boost; codec, IndexWriterConfig and index size do not matter (the state is in the classifier object). ConfusionMatrixGenerator is not affected: it calls the text path assignClass(String), which documents no boost support.
• Deterministic and silent. In sequential use, once a first call has completed on an index with class terms, the second call and every later one are unboosted (an independent reproducer: 30 of 30 failures over 10 iterations with a random seed). No exception, no log line, normalized and plausible scores.
• Concurrent use (executed). With 8 threads sharing one array or one classifier, every one of 96,000 results was either the boosted or the unboosted result (1 to 8 boosted per round of 1,200 calls, depending on timing), with no exception or hang (a probe of those runs, not a thread-safety proof; nothing promises thread safety).
• Possible downstream reach (code reading only, not executed). Solr's classification update processor with algorithm=bayes splits inputFields once and passes the same array to every new SimpleNaiveBayesDocumentClassifier; if that reading is right, in sequential use boosted inputFields (documented by Solr, e.g. title^1.5) apply only to the first document classified through that shared array. The factory's lifetime and concurrent first calls were not checked, and nothing was run in Solr.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[fruit 0.9989943793309103]> but was:<[vehicle 0.7933884297520658]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.classification.document.TestSimpleNaiveBayesDocumentClassifierBoostReuse.testFieldBoostAppliesOnEveryCall(TestSimpleNaiveBayesDocumentClassifierBoostReuse.java:64)

• Contract. Constructor javadoc (SimpleNaiveBayesDocumentClassifier.java:63-64): the field names "can contain boosting indication e.g. title^10". No single-use limit in the constructor, DocumentClassifier (DocumentClassifier.java:24-66), package-info.java, MIGRATE.md or CHANGES.txt; the interface classifies "the given Document" and is shaped for repeated calls.
• Tests. TestSimpleNaiveBayesDocumentClassifier.testBoostedDocumentClassification (:114-138) asserts that title^100 gives batman and that without the boost "wrong classification will appear"; it builds a new classifier per assertion, so the second-call path has been untested since 2015. TestKNearestNeighborDocumentClassifier reuses one instance for two documents (:33-45), and ClassificationTestBase.checkOnlineClassification calls assignClass twice on one classifier (with updateSampleIndex() between the calls) and asserts the same class and score (:121-144).
• Sibling and precedent. KNearestNeighborDocumentClassifier.knnSearch parses field^boost into locals on every call and never writes back (:126-146). LUCENE-7991 (#9039, 7.2.0, CHANGES.txt:7763-7765): "KNearestNeighborDocumentClassifier.knnSearch no longer applies a previous boosted field's factor to subsequent unboosted fields", a stale-boost bug in the sibling, fixed as a bug.
• History. The class, the write-back and the boosted test all came in LUCENE-6631 (2015); the private comment "This is done only one time for the Seed Document" (:139-140) comes from a patch revision described as "improved seed document analysis to happen only one time", that is once per call instead of once per class; nothing in that thread discusses reusing a classifier or rewriting the array.

The strongest "intended" case: the classifier and the DocumentClassifier interface are @lucene.experimental; the private javadoc says the analysis "is done only one time"; every upstream test builds a new classifier per call, so instances could be read as single-use; the parent class documents "NO boosting supported per field"; and no javadoc promises that the caller's array is left alone.

Why it fails: @lucene.experimental is about API stability, not an option that silently stops working; "only one time" means once per call, since the maps are rebuilt on every call (:108-109), so the code re-derives the boost per call by design and only the write-back defeats it; the parent's "NO boosting" covers the parent's text path, while this class's constructor allows title^10; the sibling classifier and the shared test base reuse instances, and LUCENE-7991 treated stale boost state as a bug. A reusable classifier that gives another class for the same document on the second call is not a plausible design. Verdict: a defect (adjudication: confirmed, high; the independent reproducer and the devil's advocate: real; the other-model-family intended-lens review: real, medium, intended argument weak). No javadoc promises the array is left unchanged.

Each call then parses the configured strings afresh, scores one entry per configured field with its own boost, and leaves the caller's array alone. Configurations without a repeated field name give results bit-identical to today's first call (run for {title, body} and {title^10, body}; the general case follows from the code). The verification's smaller sketch (loop over the parsed map's keys, drop the write-back) also fixes the reported bug and passes the suite, but it scores a repeated field name only once, which changes the answer of an unboosted configuration such as {"title", "title", "body"} (fruit 0.70741 today, vehicle 0.55373 with it). Parsing once in the constructor into private arrays would work too, and maintainers may prefer it or an immutable internal (name, boost) list: this doc's fix is one validated option, not the only answer. Workaround until a fix ships: build a new classifier with a new array for every call (inline varargs or names.clone()), as the upstream tests do.

assignNormClasses allocates String[] fieldNames, float[] fieldBoosts and List<List<String[]>> fieldTokensArrays sized like textFieldNames; analyzeSeedDocument parses as today and fills entry i instead of the two maps and the write-back; the scoring loop runs over i. The unused LinkedHashMap import goes and the private javadoc says textFieldNames is left unchanged. +21/-16 in SimpleNaiveBayesDocumentClassifier.java; private methods only.

    String[] fieldNames = new String[textFieldNames.length];
    float[] fieldBoosts = new float[textFieldNames.length];
    List<List<String[]>> fieldTokensArrays = new ArrayList<>(textFieldNames.length);
    ...
        for (int i = 0; i < fieldNames.length; i++) {
          String fieldName = fieldNames[i];
          List<String[]> tokensArrays = fieldTokensArrays.get(i);
          ...  calculateLogLikelihood(fieldTokensArray, fieldName, term, docsWithClassSize)
                   * fieldBoosts[i];
  // analyzeSeedDocument, instead of the map puts and textFieldNames[i] = fieldName:
      fieldNames[i] = fieldName;
      fieldBoosts[i] = boost;
      fieldTokensArrays.add(tokenizedValues);

Trade-offs

• Repeated field names change meaning. Today every copy of a repeated name gets the boost of its last entry, even on call 1 ({"title^10", "title", "body"} scores like {"title", "title", "body"}, fruit 0.70741); with this fix each entry keeps its own boost, and both tested orders give the same score (fruit 0.99998); in larger configurations another order could still change low bits (floating-point sums). No upstream test or javadoc covers repeated names.
• Parsing (split, Float.parseFloat) still runs on every call, as today. A malformed boost throws the same exception as today on the first call that parses it; unlike stock, which may already have rewritten earlier entries, the array stays untouched. title^10^2 is still read as title^10, as today. As today, the boost multiplies only the log-likelihood, not the class log-prior added per field value.
• No API change; configurations without repeated names are bit-identical to today's first call (executed for {title, body} and {title^10, body}; the general case follows from the code).

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 2 of 2 on stock (assertions on and off) and passes with the fix; with the fix the new test, TestSimpleNaiveBayesDocumentClassifier and TestKNearestNeighborDocumentClassifier: 3 suites, 9 tests, 0 failures, assertions on and off; the whole :lucene:classification:test, 13 suites and 45 tests, 0 failures, 1 @Monster skip, assertions on and off; :lucene:classification:check -x test (google-java-format, ecj lint, forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test (-ea, -da), the finder's test (2 of 2), the independent reproducer's test (3 of 3) and the upstream-data test (3 of 3, call 2 = batman, 1.0) pass; a configuration probe shows the per-entry behaviour above.

for (String fieldName : fieldName2tokensArray.keySet()) at :121 and no textFieldNames[i] = fieldName (+1/-2).

Trade-offs

Fixes the reported bug, but a repeated field name is scored once: {"title", "title", "body"}, which has no boost, changes from fruit 0.70741 to vehicle 0.55373, and it keeps the position of a name's first occurrence but the tokens and boost of its last: {"title^10", "title", "body"} loses its boost (vehicle 0.55373), {"title", "title^10", "body"} keeps it (fruit 0.99994). Acceptable only with a documented "field names must be unique".

What the checks validated

The new test passes (standalone, -ea); the whole :lucene:classification:test (45 tests, assertions on) passes; the configuration probe shows the changed results above.

Parse textFieldNames into private final name and boost arrays in the constructor (a defensive copy) and use them on every call.

Trade-offs

Same scores as the recommended option, and later changes to the caller's array no longer reach the classifier. A malformed boost (title^x, or title^ with nothing after it) then fails at construction instead of the first call. Not run.

Build new SimpleNaiveBayesDocumentClassifier(..., "title^10", "body") (inline varargs make a new array each time) or pass names.clone() for every classification, as the upstream tests do.

Trade-offs

Application-side only; costs a classifier per call; not a fix.

Method: code reading (the hunt's "non-formal" tier), a hunt unit (classification), 2026-10-26, stock main jars.

• The finder was transcribing SimpleNaiveBayesDocumentClassifier's score pipeline into a Z3 inventory of the classifiers' division and log sites, and read textFieldNames[i] = fieldName in analyzeSeedDocument. The model covers a zero-token division with a boost factor, not the write-back.
• It wrote a 2-test public-API JUnit test (TestSimpleNaiveBayesDocumentClassifierBoostReuse), which failed on main with assertions on and off and on 10.5.2 and 9.12.3, and a two-line fix sketch that passed it and the upstream classification tests.
• Then independent AI tools reproduced, reviewed and judged it (see the checks), and the history step ran the test on eight selected releases back to 6.0.0.