SimpleNaiveBayesDocumentClassifier returns NaN for every class when one field value has no tokens

SimpleNaiveBayesDocumentClassifier scores each class by adding, for every value of every configured text field of the input Document, a log prior plus the value's log-likelihood, which is the sum of log P(w|c) over the value's tokens divided by the number of tokens. A value with no tokens, such as an empty TextField or one that the analyzer reduces to nothing (stop words only, punctuation only), gives 0.0 / 0 = NaN. That NaN is added to every class, the log-sum-exp normalisation turns every probability into NaN, assignClass returns null because NaN > maxscore is never true, and getClasses lists the classes in index term order. On three training documents, title "apple" gives fruit 0.762, and the same title with an empty, stop-word-only or punctuation-only body gives NaN for both classes and no class. It is silent, with assertions on or off, needs no setting, and the division is in all 83 release tags checked from 6.0.0 to 10.5.2 and in main; skipping values with no tokens fixes it.

• Document classifier: new SimpleNaiveBayesDocumentClassifier(reader, query, classFieldName, field2analyzer, textFieldNames...) learns from the labelled documents of an index (the class is a term of classFieldName) and classifies an unseen Document: assignClass(doc) returns the best class with its score, getClasses(doc) all classes sorted by score.
• Per-value score: the input document's values are analyzed with the field's analyzer (SimpleNaiveBayesDocumentClassifier.java:148-172). For every class and every value, the class score gains log prior + log-likelihood * boost (:121-131).
• Log-likelihood: the sum of log P(w|c) over the value's tokens, divided by the number of tokens (:201-223). The original comment explains the division: "this is normalized because if not, long text fields will always be more important than short fields" (commit 71cea88773e, 2015).
• Normalisation: a log-sum-exp turns the class scores into probabilities, "the values transforms to a 0-1 range" (SimpleNaiveBayesClassifier.java:292-318, comment at :294).

1. An empty value, or one whose tokens are all removed by the analyzer, gives a token array of length 0 (an absent field gives an empty list, so the scoring loop never runs for it).
2. calculateLogLikelihood then skips its loop and returns result / (tokenizedText.length) with result = 0.0 and length 0 (:222): 0.0 / 0 = NaN.
3. assignNormClasses adds prior + NaN * boost = NaN to every class (:124-130).
4. The normalisation takes the top score as smax and sums exp(score - smax): one NaN input makes every output NaN.
5. assignClass keeps a result only if c.score() > maxscore, starting from -Double.MAX_VALUE (:80-87); with NaN this is never true, so it returns null.
6. getClasses sorts with Double.compare (ClassificationResult.java:31-33), which treats all NaN values as equal, so the list stays in class-term order and getClasses(doc, n) returns the first n class names (SimpleNaiveBayesDocumentClassifier.java:98-103).

Index: {apple, "sweet fruit", fruit}, {pear, "green fruit", fruit}, {car, "four wheels", vehicle}, EnglishAnalyzer (it stems apple to appl). Title field: 3 postings over 3 documents, so 1.0 unique term per document.

Without the body: fruit 0.7619, vehicle 0.2381. With the empty body: both NaN, assignClass = null. Executed cases (stock main):

• Who is exposed. Applications that classify documents with SimpleNaiveBayesDocumentClassifier ("A simplistic Lucene based NaiveBayes classifier", @lucene.experimental). No other Lucene module uses the class.
• What triggers it. One value of any configured text field in the document being classified that analyzes to no tokens: an empty description, a title made only of stop words ("To be or not to be" under EnglishAnalyzer), a value of punctuation or symbols. Three training documents and stock analyzers show it; one such value among many non-empty ones is enough.
• What breaks. Every class probability is NaN and assignClass returns null, so a caller that reads assignClass(doc).assignedClass() throws a NullPointerException later, and a caller that takes getClasses(doc, 1) gets the first class name in index term order whatever the document says. Nothing fails inside Lucene, with assertions on or off.
• Not affected. Classification after empty values in training documents (they enter only through index statistics; with 50 such training documents in the index a normal document still got finite scores and the right class), later calls on the same instance, the index, and the other classifiers on the same input. A different case, not this bug: if every training value of a configured text field is empty, so the field has no indexed terms, classifying a document with tokens in that field throws a NullPointerException (getTextTermFreqForClass, :233-236); the fact check reproduced it, and it has not been checked for prior reports.
• How likely. Such values are ordinary in real data and trigger it every time, but the module has few users, and no report of it was found since 6.0.0 (2016).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected a class with a finite score; assignClass: null, getClasses: [ClassificationResult[assignedClass=[66 72 75 69 74], score=NaN], ClassificationResult[assignedClass=[76 65 68 69 63 6c 65], score=NaN]]
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.classification.document.TestSimpleNaiveBayesDocumentClassifierEmptyField.testEmptyFieldValue(TestSimpleNaiveBayesDocumentClassifierEmptyField.java:62)

• Interface contract. DocumentClassifier.assignClass: "Assign a class (with score) to the given Document", returning "a ClassificationResult holding assigned class of type T and score" (DocumentClassifier.java:31-40). Only getClasses mentions null, "if the classifier can't make lists" (:48-49, :62-63). The contract is implied, not explicit: no javadoc promises a finite score, and the implementation already returns null when the index has no class terms at all.
• Range. "normalization; the values transforms to a 0-1 range" (SimpleNaiveBayesClassifier.java:156, :294). NaN is not in that range.
• The comment above the division describes a sum, log(P(d|c)) = log(P(w1|c))+...+log(P(wn|c)) (SimpleNaiveBayesDocumentClassifier.java:221), which is 0 for no tokens, not NaN.
• Tests. DocumentClassificationTestBase.checkCorrectDocumentClassification asserts a non-null class (:61) and 0 <= score <= 1 (:67); ClassificationTestBase asserts a non-null result (:93, a line added on purpose by LUCENE-7981, #9030, 2017) and a non-null class (:95, older). No test uses an empty or stop-word-only value, and both fixes keep the whole module suite green.
• History. The division is deliberate length normalisation from the commit that created the class (LUCENE-6631, 71cea88773e, 2015); the zero-length case was never handled. LUCENE-5698 (552c367c5d2, 2019) only inlined it. The issue's description and 21 comments do not mention empty values.
• Legal input. No javadoc says field values must produce tokens; TextField and IndexWriter accept empty values, and stock analyzers remove stop words and punctuation.

The strongest "intended" case (built by an intended-behaviour checker; a cross-model reviewer rated it weak): the class is @lucene.experimental and "simplistic", the division is deliberate length normalisation, and an average over zero tokens is undefined, so a caller who passes empty values gets what they asked for.

Why it fails: no javadoc, test, CHANGES or MIGRATE entry, issue or list thread states a "values must have tokens" precondition, and such values are legal input; one empty value cancels the evidence of every other field and value, and assignClass returns no class where its javadoc describes a result holding a class and a score; every sibling path (field absent, the text SimpleNaiveBayesClassifier, KNearestNeighborDocumentClassifier) returns a finite score and a class on the same input; @lucene.experimental allows API changes, not results outside the documented range. The independent reproduction, the cross-model review and the intended-behaviour check rated the bug real and the adjudication confirmed it with high confidence; the refute-lens cross-model review gave only a procedural "not real" at low confidence (it could not run Java) while agreeing with the root cause. Not settled: whether an empty value should count like an absent one (fix A), keep its prior (fix B) or be rejected with an exception; a maintainer could reasonably choose any of these.

Two fixes were built and validated in a scratch worktree of a9779ef3c193. Ask upstream only for finite scores and a class; that an empty value scores like an absent one is fix A's design choice, not a documented promise. Other defensible choices, not built: reject a value with no tokens with an exception, document a precondition, or guard the normalisation against NaN.

In assignNormClasses (SimpleNaiveBayesDocumentClassifier.java:124), continue when the value's token array is empty; an empty value then adds neither the prior nor a likelihood, exactly like an absent field. The hunt's own fix is the same change.

           for (String[] fieldTokensArray : tokensArrays) {
+            if (fieldTokensArray.length == 0) {
+              continue; // no tokens, no evidence: treat the value as absent
+            }
             fieldScore +=
                 calculateLogPrior(term, docsWithClassSize)

Trade-offs

• An empty, stop-word-only or punctuation-only body gives the no-body result: fruit 0.7619 for title "apple", vehicle 0.5556 for title "car".
• A document whose every value is empty gets uniform scores (0.5 / 0.5, assignClass = the first class), which is what a document with no fields at all already gets today; the text SimpleNaiveBayesClassifier returns the prior (0.6667) for empty input instead.

What the checks validated

• Gradle, JDK 25: the new test fails 3 of 4 on stock with assertions on and off; with the fix, the new test plus TestSimpleNaiveBayesDocumentClassifier, TestSimpleNaiveBayesClassifier and TestKNearestNeighborDocumentClassifier pass (4 suites, 15 tests); the whole :lucene:classification:test passes with assertions on and off (13 suites, 47 tests, 1 skipped, 0 failures each); :lucene:classification:check -x test passes.
• Standalone runner with the fix shadowed: the new test 4 of 4 (-ea, -da), the finder's test 2 of 2, the independent reproducer's test 6 of 6.
• Earlier, on ea57360912ed: the 42 upstream classification tests and the security probe (poisoned documents get the no-body result).

At the division (SimpleNaiveBayesDocumentClassifier.java:222), return 0 when the value has no tokens: the empty sum the comment above it describes.

     // log(P(d|c)) = log(P(w1|c))+...+log(P(wn|c))
+    if (tokenizedText.length == 0) {
+      return 0; // empty sum; dividing by the length would give 0.0 / 0 = NaN
+    }
     return result / (tokenizedText.length);

Trade-offs

• Scores become finite, but the empty value still adds its class prior once more (the per-value prior at :125-126), which pulls the result towards the larger class: title "apple" plus an empty body gives fruit 0.8649 instead of 0.7619, and title "car" plus an empty body gives fruit 0.6154 while "car" alone gives vehicle 0.5556. So an empty value can still change the class.
• The smallest change at the division; the adjudication noted upstream may prefer it if they see the per-value prior as intended (the audit lists both fixes as acceptable). With equal class priors it gives the same scores as fix A (arithmetic: the extra prior cancels in the normalisation; no run compared the two).

What the checks validated

• Gradle, JDK 25: the same four suites (15 tests) and the whole :lucene:classification:test with assertions on and off (13 suites, 47 tests, 1 skipped, 0 failures each); check -x test passes; standalone: the new test 4 of 4, the finder's 2 of 2, the independent reproducer's 6 of 6.

Drop field values that analyze to no tokens before classifying (run the field's analyzer, or at least skip empty strings), or treat NaN scores and a null from assignClass as "unclassified" instead of taking the first entry of getClasses. Lucene itself stores nothing wrong (the classifier only reads the index), so the index needs no repair; an application that stored the returned class (for example through Solr's update processor) may need to reclassify the affected documents.

Trade-offs

Needs one extra analysis pass per value, or a NaN check in the caller.

Method: semi-formal (an SMT inventory plus a bounded check of the real code), then a JUnit test. A hunt unit (classification), 2026-10-26, main ea57360912ed.

• The finder wrote a Z3 4.12.2 model (floating-point arithmetic with an uninterpreted log) of every division and log site in the classifiers' raw scores. Site S4, result / tokenizedText.length with length 0, was satisfiable (a NaN field score for every class); with length >= 1 it was proved finite. The site was read in the code while building the inventory, so this was a guided rather than a blind find.
• 21 of 21 Z3 and cvc5 lemmas characterised normClassificationResults, and a bounded check of the real method over 1,082,400 special-value vectors (0 failures) showed that an output is NaN exactly when an input is NaN or +Inf, or every input is -Inf: one NaN poisons every class.
• A public-API JUnit on the stock jar then witnessed it (fruit = NaN for an empty and a stop-word-only body), with assertions on and off, on main, 10.5.2 and 9.12.3.