MoreLikeThis boosting divides by the lowest term score, so a term that scores 0 makes like() throw

MoreLikeThis with the opt-in setBoost(true) weights each clause of the query it builds by its term's score divided by bestScore, the score of the first term it pops from its queue. Since LUCENE-5795 (4.10.0) that queue pops the lowest score first, so the divisor is the lowest selected score. A term that scores exactly 0, which a custom TFIDFSimilarity such as the textbook idf log(N/df) gives a term found in every document, makes the first boost 0/0 = NaN: like() throws IllegalArgumentException: boost must be a positive float, got NaN on 8.0.0 through 10.6.0 and main, with assertions on and off, and 4.10 to 7.x silently build NaN and Infinity boosts. The default ClassicSimilarity (idf at least 1) never gives 0, so Lucene's own callers are not affected.

All lines are in lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java at 645d4fe14a20.

1. createQueue (:624-677) scores each candidate term as tf * similarity.idf(docFreq, numDocs) (:660-661) and keeps the best min(maxQueryTerms, #terms) in PriorityQueue.usingComparator(limit, FloatComparator.comparing(st -> st.score)) (:628-629). Lucene's PriorityQueue keeps the least element on top, so pop() returns the lowest score first.
2. createQuery (:593-616) pops the terms one by one. With setBoost(true) it stores the first popped score as bestScore (:596, :602-604) and wraps each clause in new BoostQuery(tq, boostFactor * myScore / bestScore) (:606). So every score is divided by the lowest selected score.
3. If that lowest score is 0, the first boost is 0 / 0 = NaN (and every later positive term would get score / 0 = +Infinity).
4. The BoostQuery constructor rejects a boost that is not finite or is negative (BoostQuery.java:40-41; 0 is accepted), so like() throws IllegalArgumentException: boost must be a positive float, got NaN before any search runs.

Every like() overload ends in createQuery: like(int) (:555-563), like(Map) (:569-576) and like(String, Reader...) (:584-590); the first and the last were executed.

A new term replaces the queue's lowest entry only if it scores strictly higher (:669), so a 0-score term stays selected while fewer than maxQueryTerms (default 25) other terms score above 0. With a zero-idf similarity, 1, 23 or 24 positive-score terms plus the zero-score term throw; with 25 or 30 the zero-score term drops out and the query builds.

ClassicSimilarity.idf is log((docCount + 1) / (docFreq + 1)) + 1 (ClassicSimilarity.java:69-70), and MoreLikeThis passes ir.getDocCount(fieldName) (:634), which counts deleted documents just as docFreq does, so idf is at least 1. Even with 99 of 100 documents deleted and not merged, the stock similarity builds finite boosts (executed on main and 8.11.4). This relies on getDocCount: MoreLikeThis falls back to ir.numDocs() (live documents only) when it returns -1 (:635-637), which the standard readers never do (LeafReader.getDocCount is final and returns the field's Terms.getDocCount() or 0, and Terms.getDocCount() documents no -1). Before 8.3.0 MoreLikeThis always passed ir.numDocs(), so on 4.10 to 8.2 this argument does not hold: there the stock idf could go below 0 with many unmerged deletes (a separate, historical issue).

• Who is hit. Direct MoreLikeThis users who turn on setBoost(true) (DEFAULT_BOOST = false, MoreLikeThis.java:190) and pass their own TFIDFSimilarity (constructor :300 or setSimilarity :309) whose idf can be exactly 0: the textbook log(N/df), or a clamped Robertson-Sparck Jones idf max(0, log((N - df + 0.5) / (df + 0.5))) (both executed). Any like text that contains a term found in every document of the field, plus fewer than 25 other scoring terms, fails: ordinary short input, not crafted input.
• Who is not. ClassicSimilarity and SweetSpotSimilarity (which does not override idf). Lucene's own callers use new MoreLikeThis(reader), that is ClassicSimilarity: MoreLikeThisQuery (:61), the classification module's kNN classifiers (which do call setBoost(true)) and Luke.
• Default vs opt-in. Both triggers are opt-in. Nothing else is needed: the repro uses new IndexWriterConfig() and two documents and creates no IndexSearcher (the exception comes from building the query). It ran under the test framework's randomized codec (CheapBastard in the gradle runs) and failed on every seed tried.
• Effect. On 8.0 and later one request fails loudly, with a message that blames a boost the caller never set. No state damage: the reader, searcher and MoreLikeThis keep working. On 4.10 to 7.x (out of support) the boosts are silently NaN and Infinity: on 7.7.3 with assertions off a search with that query returned the right total hit count but a top list with a placeholder (doc 2147483647, score -Infinity) and doc 1 with score NaN, while the real hit doc 0 was missing.
• Workaround. setMaxDocFreq(...) or setMaxDocFreqPct(...) so a term in every document is never selected, setBoost(false), or a similarity whose idf stays above 0.
• How likely. Rare: no report was found in the recorded searches, although the code has been this way since 4.10.0 (2014). The failure is deterministic: the same input fails every time.

The test indexes common rare and common into a ByteBuffersDirectory with new IndexWriterConfig() (under the test framework the codec is randomized; the gradle runs used CheapBastard), so idf(common) = log(2/2) = 0 and idf(rare) = log(2/1) = 0.69, and calls like("text", new StringReader("common rare")) with StandardAnalyzer, setMinTermFreq(1), setMinDocFreq(1) and setBoost(true). The IllegalArgumentException is rethrown as an AssertionError that states expected versus actual. Nothing in the test uses random(); it failed on 10 of 10 derived seeds with assertions on and off. It also fails on the 10.6.0 release jars. On releases before 9.0 the test-framework package differs (org.apache.lucene.util.LuceneTestCase); before 8.0 there is no exception to catch, and the boost is NaN instead. The file has no license header: add the standard ASF header before submitting it as the regression test (that shifts the test line numbers in the output below; the license check was not run).

TestMoreLikeThisBoostZeroScore > testBoostWithZeroIdfTerm FAILED
    java.lang.AssertionError: expected a query with finite boosts, got java.lang.IllegalArgumentException: boost must be a positive float, got NaN
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:9D2BA54E4B4AEA09]:0)
        at org.apache.lucene.queries.mlt.TestMoreLikeThisBoostZeroScore.testBoostWithZeroIdfTerm(TestMoreLikeThisBoostZeroScore.java:44)

        Caused by:
        java.lang.IllegalArgumentException: boost must be a positive float, got NaN
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.BoostQuery.<init>(BoostQuery.java:41)
            at org.apache.lucene.queries.mlt.MoreLikeThis.createQuery(MoreLikeThis.java:606)
            at org.apache.lucene.queries.mlt.MoreLikeThis.like(MoreLikeThis.java:589)
            at org.apache.lucene.queries.mlt.TestMoreLikeThisBoostZeroScore.testBoostWithZeroIdfTerm(TestMoreLikeThisBoostZeroScore.java:42)

1 test completed, 1 failed

# the same with -Ptests.asserts=false; with the recommended fix the test passes

# the query each setting builds for the same input (a printing probe, not the repro)
stock, setBoost(false): text:common text:rare
stock, setBoost(true):  threw java.lang.IllegalArgumentException: boost must be a positive float, got NaN
fix,   setBoost(true):  (text:common)^0.0 (text:rare)^1.0

• Nothing asks for a positive score. MoreLikeThis(IndexReader, TFIDFSimilarity) and setSimilarity take any TFIDFSimilarity with no precondition (MoreLikeThis.java:300-311). TFIDFSimilarity.idf sets no lower bound: "Terms that occur in fewer documents are better indicators of topic, so implementations of this method usually return larger values for rare terms, and smaller values for common terms." (TFIDFSimilarity.java:408-409). Lucene's base similarity test suite checks that scores are "non-negative and finite" (BaseSimilarityTestCase.java:256, checked at :387-389): a test-framework expectation rather than an API contract, and it allows 0. setBoost's javadoc only says: Sets whether to boost terms in query based on "score" or not (MoreLikeThis.java:422).
• 0 is a legal boost and a legal score. BoostQuery accepts 0 (BoostQuery.java:40). In a verifier's test the same zero-idf similarity scores a 4-document index normally through IndexSearcher.search: a TermQuery on the ubiquitous term returns all 4 documents with score 0.0.
• The same input works without boosting, and worked before 4.10. setBoost(false) returns text:common text:rare. 4.9.1 passes the same test: its queue popped the highest score first (lessThan returned fa > fb), so the divisor was the highest score and a zero-score term got boost 0.
• The change of divisor was a side effect. LUCENE-5795 (#6857, "More Like This: ensures selection of best terms is indeed O(n)") flipped the queue to ascending order for its bounded top-N selection and kept "first popped = bestScore". Its 4.10.0 CHANGES entry sits under Optimizations: "MoreLikeThisQuery now only collects the top N terms instead of collecting all terms from the like text when building the query." (lucene/CHANGES.txt:11949-11951). The issue's 9 comments never mention boosts.
• No test covers it. The upstream TestMoreLikeThis uses only the default ClassicSimilarity, so no boosted test sees a score of 0.

The strongest case for "works as designed" (from the verifiers, the cross-model reviews, the adjudication and the cross-model fact check): setBoost is off by default; TFIDFSimilarity is an expert API; every in-tree similarity has idf at least 1; setMaxDocFreq avoids it; and since 8.0 it fails fast instead of returning wrong results, so it is a misconfiguration. MoreLikeThis is a heuristic term selector, and a maintainer may prefer to reject negative or non-finite similarity scores explicitly rather than turn them into a boost (the recommended fix maps negative and NaN scores to 0 and still throws on +Infinity).

Why it fails: nothing documents that boosted MoreLikeThis needs strictly positive scores; the textbook idf gives 0 for a term in every document; BoostQuery itself accepts 0; the identical call works with boosting off and worked in 4.9.1; and the exception reports an internal 0/0, not a bad argument. These points lower the impact; none makes the exception intended. Verdict: not intended, a defect.

One fix was built and validated with gradle on 645d4fe14a20; the alternatives come from the study's verifiers and the adjudication. What a zero-score term should get (boost 0, or no clause) is the maintainers' choice: the defect is the NaN and the exception. LFV-571 (at the boolean clause limit the same loop keeps the lowest-scoring terms) is a sibling: both come from LUCENE-5795's queue flip, and their fixes touch the same lines.

Take bestScore from the first popped term whose score is above 0, and give a term with score 0 the boost 0. The first popped score of an all-positive selection is positive, so it becomes bestScore exactly as today and no existing boost changes (code reading). An all-zero selection gives every clause boost 0. The zero-score clause stays, so by code reading the boosted query matches the same documents as the unboosted one (not executed); scores, and so ranking and top-N results, differ, as with any boosting.

       if (boost) {
-        if (bestScore == -1) {
-          bestScore = (scoreTerm.score);
-        }
         float myScore = (scoreTerm.score);
-        tq = new BoostQuery(tq, boostFactor * myScore / bestScore);
+        // q pops the lowest score first: normalise by the lowest POSITIVE score, so that a term
+        // whose score is 0 (a TFIDFSimilarity idf may be 0) gets boost 0 instead of 0/0 = NaN
+        if (bestScore == -1 && myScore > 0) {
+          bestScore = myScore;
+        }
+        tq = new BoostQuery(tq, myScore > 0 ? boostFactor * myScore / bestScore : 0f);
       }

Trade-offs

A negative or NaN score from a broken custom similarity also becomes boost 0 instead of an error. If the maintainers want such scores rejected, add an explicit check with a message that names the similarity. The fix covers finite, non-negative scores: a +Infinity score still gives an infinite or NaN boost (Infinity / Infinity), so BoostQuery still throws, as today (code reading).

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (removed after the runs): stock, the repro fails with assertions on and off; with the fix, --tests 'org.apache.lucene.queries.mlt.*' (the repro and the upstream TestMoreLikeThis: 12 tests, 0 failures, 1 @AwaitsFix skipped) passes with assertions on and off, the whole :lucene:classification:test (12 classes, 43 tests, 0 failures, 1 skipped; its kNN classifiers run boosted MoreLikeThis with ClassicSimilarity) passes with assertions on, and :lucene:queries:checkGoogleJavaFormat passes. The classification module with assertions off was run by the page's fact check, in a fresh scratch worktree of 645d4fe14a20 with the fix: 12 classes, 43 tests, 0 failures, 1 skipped.
• A printing probe shows the fixed query for the repro's input: (text:common)^0.0 (text:rare)^1.0.
• Earlier checks: the study's fix check (the repro and the upstream TestMoreLikeThis, 11 run, 0 failed), the verifiers' tests and the adjudication's re-run pass with it; the standalone runner passes the repro with it on 645d4fe14a20 (-ea, -da).

Restore the pre-4.10 normaliser: drain the queue, divide every score by the highest one, and give boost 0 when that is 0.

Trade-offs

Changes the boost of every existing boosted query: positive scores would get boosts in (0, boostFactor] instead of starting at boostFactor (a zero score gets 0 under the guard). For example scores 1 and 2 give boosts 1 and 2 today and 0.5 and 1 then. An all-zero selection still needs the guard, and the loop must read the highest score before it builds the first clause.

Skip the clause of a term whose score is 0, since it adds nothing to the score.

Trade-offs

The boosted query then matches fewer documents than the unboosted one (in the repro, the document that contains only common no longer matches), and an all-zero selection gives an empty query.

Method: semi-formal (a hypothesis formed while modelling, then witnessed by a tool).

• An AI finder wrote a reference implementation of MoreLikeThis term selection and query construction from its javadoc and compared it with the real queries jar in bounded-random translation validation (118,800 checks, seed DEADBEEF). Observation O1: all 8,242 boosted checks were normalised by the lowest selected score, not the highest.
• A small selection model (create_query: best = first popped) showed that the divisor can be a zero minimum. A scratch probe with a zero-idf similarity threw on main, and the finder wrote a public-API JUnit, later reduced to the two-document test.
• So the tool pointed at the lowest-score divisor; the zero-score trigger was then reasoned out and confirmed by execution. It is not a blind discovery by a tool.