BM25NBClassifier assigns a label that has no training document

BM25NBClassifier makes every term of the class field a candidate label, including a label whose documents are all outside the documented training query or all deleted but not yet merged away. It scores each label as a log prior plus one log likelihood per input word, each the log of the top BM25 score of a search in which the label term is a MUST clause. For a label with no live, training-query-matching document both searches find nothing, and the code falls back to log prior 0 and word probability 1: the label scores exactly 0, the score of a label with probability 1. Trained labels add logs of BM25 scores, which are negative when those scores are below 1, so with few labels the untrained one comes first: on five documents assignClass("apple") returns heldout (0.568) instead of fruit. No error is raised. Every release checked from 7.0.0 to 10.5.2 and main has the code, with ordinary short inputs from 8.0.0; skipping labels whose prior search finds nothing fixes it.

• Label: a term of the class field, for example fruit. BM25NBClassifier takes every term of that field as a candidate (MultiTerms.getTerms, BM25NBClassifier.java:124).
• Training query: the constructor's query, documented as "a Query to eventually filter the docs used for training the classifier" (:72-73). Both searches below add it as a MUST clause.
• Score of a label: log prior + the sum of log likelihoods of the input words (:134).
  • Prior: the top BM25 score of label MUST + training query MUST (:222-231).
  • Likelihood of a word: the top BM25 score of label MUST + word SHOULD + training query MUST (:205-220). The word is optional, so a trained label always has a hit.
• Result: the scores are turned into probabilities with a softmax, best first (:147-173); assignClass returns the first (:96).

1. When a search finds nothing, the code falls back to log prior 0 (:230) and word probability 1, so log likelihood 0 (:219).
2. Because the label term is MUST in both searches, only a label with no live, training-query-matching document ever reaches these fallbacks. It scores exactly 0, the score of a label with probability 1.
3. A trained label scores the sum of 1 + t logs of BM25 scores for t input words. A single BM25 term score on a StringField label is idf / 2.2 with the default k1 = 1.2, which is below 1 unless the label covers fewer than about 11% of the documents. So with few, large labels the trained totals are negative and the untrained label wins.

A BM25 score can exceed 1: in "apple pear", "pear" scores 1.1314 (log +0.124) for fruit, yet fruit's total is -0.609 and heldout still wins (0.591). Deleted variant, no training query: fruit -1.172, vehicle -1.843, gone 0, so gone 0.681. After forceMergeDeletes() the term gone is dropped and "apple" is fruit (0.667).

• Who is exposed. Applications that classify with BM25NBClassifier (@lucene.experimental) when a label is in the class field but has no live, training-query-matching document: one index holding a train/test split in which some label occurs only in test documents, or a retired label whose documents were all deleted and whose segment has not been merged yet. Both use public API and default settings.
• How often. It depends on the number of labels and on the corpus. Every input tried in the two small repro corpora (5 of 5 here, 6 of 6 in an independent sports/cooking corpus). In a synthetic sweep with K balanced labels of 200 documents per label and one deleted extra label: 34% of inputs at K = 2 (accuracy 0.90 to 0.64), 4% at K = 3, 0% at K = 5, 10 and 20 (the same 34% and 4% with the extra label outside the training query). In a second sweep with 5 documents per label and the extra label outside the training query: every input at K = 2, 3 and 4, 2 of 8 at K = 6, 2 of 10 at K = 8, none at K = 10, 15 and 20. So no sweep shows a wrong top answer with 10 or more labels; where the untrained label does not win, it still gets a non-zero share in getClasses.
• The input matters. Each extra input word adds a negative log to every trained label and 0 to the untrained one: in the 200-documents sweep, appending 5 unknown words to each input raised the share sent to the untrained label from 34% to 98% at K = 2 and from 4% to 49.3% at K = 3 (still 0% at K = 5 and 10).
• Silent. No exception, with assertions on or off; the answer is a real label from the index with a plausible probability.
• Not affected. Indexes where every label has at least one live training document.
• Downstream. No known caller: Solr's classification update processor builds only the k-NN and simple naive Bayes document classifiers, and a read-only code search found no occurrence of BM25NBClassifier on the default branches of Solr, Elasticsearch and OpenSearch.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[frui]t> but was:<[heldou]t>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.classification.TestBM25NBUntrainedClass.testClassExcludedByTrainingQueryIsNotAssigned(TestBM25NBUntrainedClass.java:56)

• Contract. Class javadoc: "A classifier approximating naive bayes classifier by using pure queries on BM25" (BM25NBClassifier.java:43). Constructor javadoc: "a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used" (:72-73). Package javadoc: "Uses already seen data (the indexed documents) to classify an input".
• Design intent. The introducing issue #8874 (LUCENE-7823): "uses BM25 scoring as a probability measure of both prior and likelihood". The issue that added the training query, #6348 (LUCENE-5284): "It's sometimes useful to use only a certain subset of the whole index to train the classifier with".
• Tests. TestBM25NBClassifier.testBasicUsageWithQuery filters with text:not, which matches a document of both sample labels, and the test base force-merges its indexes, so no test has a label without training documents.
• History. Both fallbacks date from the introducing commit (2017); later commits only adapted APIs, and no commit message or comment explains the values.
• Metamorphic check. Two readers with the same live, training-query-matching documents give different answers depending on whether the deleted label's term is still in the dictionary: forceMergeDeletes() turns gone back into fruit. The merge also changes the BM25 statistics, but on the unmerged reader fix A already answers fruit (0.662), so the term, not the statistics, flips the answer.

The strongest "intended" case (cross-model reviews rated it moderate): the class is @lucene.experimental and only approximates naive Bayes with uncalibrated BM25 scores; the fallbacks 1 and log 1 = 0 guard against log 0 = -Infinity, which would make the normalisation NaN when no label has a hit; the private javadoc of assignClassNormalizedList promises "one for each existing class" (:117); and the module's train/test workflow, DatasetSplitter, writes a separate training index, whose term dictionary holds only the labels that reached it (LUCENE-7196 aimed at class coverage, although a label with a single document can still land entirely in the test or cross-validation index, DatasetSplitter.java:153-160).

Why it fails: the guard explains why some value was needed, not why that value is log 0 = log 1, the score of a label with probability 1, which outranks every trained label whose summed logs are negative. A label with any training document never reaches the fallbacks, so they are not a neutral value for weak evidence but the value given only to zero evidence. The training query is documented for training on a subset of one index, and nothing tells the caller to avoid a subset that lacks a label. @lucene.experimental allows API changes, not an answer that contradicts the javadoc. Verdict: not intended, a defect. The deleted-documents variant is the one a maintainer is most likely to push back on (term dictionaries keep deleted terms until a merge).

Two fixes were built and validated; a third is a design alternative. Labels with training documents keep exactly their old raw scores under fixes A and B.

Workaround until a fix ships. Deleted labels: IndexWriter.forceMergeDeletes() (or any merge that rewrites the segment) drops the term, and after reopening "apple" is fruit again; nothing is stored wrong, so no reindex is needed. Training splits: classify on an index that holds only the training documents, for example DatasetSplitter's output.

Either way, add tests like the repro's (both variants and the control), plus the empty-training-set case and getClasses(text, max) for the chosen fix.

+14/-6 lines in BM25NBClassifier.java: the prior fallback becomes Double.NEGATIVE_INFINITY (:230); assignClassNormalizedList skips a label whose prior is -Infinity, which also saves its t likelihood searches (its prior search still runs: that search finds the label has no training document); assignClass returns null when no label is left (:96); getClasses(text, max) clamps max to the list size (:110); the private javadoc says "one for each class that has at least one live document matching the training query".

        double logPrior = calculateLogPrior(term);  // now -Infinity when no hit
        if (logPrior == Double.NEGATIVE_INFINITY) {
          continue;  // no training document has this class: prior 0
        }
        assignedClasses.add(
            new ClassificationResult<>(
                term.bytes(), logPrior + calculateLogLikelihood(tokenizedText, term)));
  // assignClass:
    return assignedClasses.isEmpty() ? null : assignedClasses.get(0);
  // getClasses(text, max):
    return assignedClasses.subList(0, Math.min(max, assignedClasses.size()));

Trade-offs

• Why the two extra changes. Skipping can leave the list empty or shorter than the number of label terms. When the training query matches no labelled document, stock code returns an arbitrary label (every label scores 0: fruit, heldout, vehicle 0.333 each), and the hunt's first skip-only sketch threw IndexOutOfBoundsException at :96. Fix A returns null and an empty list, as SimpleNaiveBayesClassifier.assignClass (:102-113) and KNearestNeighborClassifier.classifyFromTopDocs (:126-138) already do when they have no class; ConfusionMatrixGenerator skips a null result (:109), but if every result is null its average time is time / counter with counter 0, so NaN or Infinity (:149).
• getClasses(text, max) with a max between the number of trained labels and the number of label terms would otherwise throw at :110 where it worked before. The clamp also changes the open #9158 / LUCENE-8110 behaviour (a max larger than the list throws) for this class only, as KNearestNeighborDocumentClassifier already clamps (:112). Mention #9158: the same clamp was committed for this and four other classifiers (af41d02eae6, 2017-12-28) and reverted the same day (152d223b323) after a reviewer asked for a test, so add one.
• null from assignClass is not in its javadoc (Classifier.java:30-37 does not mention null; the getClasses javadoc says it may return null, :43-44, :55-56); it appears only in the empty-training-set case, where the stock answer was arbitrary. The : 1 fallback at :219 becomes unreachable.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 2 of 3 on stock with assertions on and off; with the fix, the whole :lucene:classification:test passes with assertions on and off (13 suites, 46 tests including the new test's 3, TestBM25NBClassifier and TestConfusionMatrixGenerator, 1 skipped, 0 failures each); :lucene:classification:check -x test (google-java-format, ecj lint, forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test 3 of 3 with -ea and -da; the finder's test 2 of 2; the independent reproducer's test 3 of 3; the probe: 0 of 5 inputs to the untrained label in both variants ("apple": fruit 0.632, vehicle 0.368), getClasses("apple", 3) returns the 2 trained labels, and the empty training set gives null and an empty list.

The :230 change of fix A alone, one expression (the independent reproducer's root-cause probe was the same change). An untrained label scores -Infinity and gets probability 0 but stays in the list, so getClasses(text, max) is unchanged: "apple" gives fruit 0.632, vehicle 0.368, heldout 0.000.

-    return topDocs.totalHits.value() > 0 ? Math.log(topDocs.scoreDocs[0].score) : 0;
+    return topDocs.totalHits.value() > 0
+        ? Math.log(topDocs.scoreDocs[0].score)
+        : Double.NEGATIVE_INFINITY;

Trade-offs

When no label has a training document, every score is -Infinity and the normalisation computes exp(-Infinity - (-Infinity)) = NaN: getClasses returns fruit NaN, heldout NaN, vehicle NaN, and assignClass returns fruit with score NaN. Untrained labels still cost 1 + t searches each and stay listed. A maintainer may pair it with an explicit case for "every score is -Infinity" in the normalisation.

What the checks validated

The new test 3 of 3 with the fix shadowed; the whole :lucene:classification:test with assertions on in the scratch worktree (13 suites, 46 tests, 1 skipped, 0 failures); the probe shows the NaN case above.

Use the label's share of the training documents as prior, for example log(count(label AND query) / count(labelled documents AND query)). That is -Infinity for a label without training documents as long as some labelled document matches the query; when none does it is log(0/0) = NaN, so the empty training set needs its own handling (and count(query) alone would also count unlabelled matches).

Trade-offs

Changes every label's score, also on indexes without the bug, and would also address the broader bias towards rare labels that one reviewer reported (not confirmed as a separate bug); it needs an accuracy check, for example with Test20NewsgroupsClassification, and a maintainer's view.

• The hunt unit for lucene/classification built an SMT inventory (Z3 4.12.2) of every division and log in the classifiers' raw scores. For BM25NBClassifier.java:219 and :230 the obligation was that the log of a positive, finite BM25 score is finite; it was proved, and it does not cover the fallback branch.
• Reading those lines, the finder saw that a search with no hit returns probability 1 and log prior 0, and that the label term is MUST in both searches, so only a label with no live training document gets there. A JUnit on the stock jars confirmed it with the training-query and the deleted-documents variant.
• An independent reviewer from another model family (OpenAI GPT-5.6) made the finder narrow the claim ("assigned for every input" holds only while the trained labels' BM25 scores stay below 1). Independent verifiers, an adjudication and an audit followed.