LMDirichletSimilarity accepts mu = 0, then scores every document 0

LMDirichletSimilarity validates its smoothing parameter and accepts any finite mu >= 0, and Lucene's own test uses mu = 0 as the "minimum value". At mu = 0, score() computes log(1 + freq / (mu * P)) + log(mu / (dl + mu)) = log(+Infinity) + log(0) = +Infinity + -Infinity = NaN, and the clamp score > 0.0d ? score : 0.0d, written for the paper's negative scores, turns that NaN into 0.0. Every hit of every term on the field then scores 0.0 and comes back in doc-id order (the default collector breaks ties by doc id), with no error; explain() shows Infinity = term weight and -Infinity = document norm under a 0.0 total. Every positive mu tried ranks normally, from Integer.MAX_VALUE (the largest value upstream's test uses) down to Float.MIN_VALUE, which gives the finite mu -> 0 limit log(tf / (dl * P)); only absurd values from about 1e17 up also zero the scores, through double rounding, a separate limit. The bug is as old as the class (added in 2011, first released in 4.0.0-ALPHA, 2012) and is in every release up to 10.6.0 and in main. It is opt-in and low-impact: an @lucene.experimental similarity at one boundary value. Two fixes were validated: compute the score as one log, which is finite at mu = 0 for any P > 0 (always so with the default collection model), or reject mu = 0 the way LMJelinekMercerSimilarity rejects lambda = 0.

LMDirichletSimilarity.java:76-82:

double score =
    stats.getBoost()
        * (Math.log(1 + freq / (mu * ((LMStats) stats).getCollectionProbability()))   // :79
            + Math.log(mu / (docLen + mu)));                                            // :80
return score > 0.0d ? score : 0.0d;                                                     // :81

1. mu = 0 is accepted. Both constructors throw only when mu is negative or not finite ("must be a non-negative finite value", :49-52 and :58-61); -1 and NaN are rejected, 0 is not.
2. First log (:79). Every hit has freq > 0 and P > 0, so freq / (0 * P) is +Infinity and log(1 + Infinity) is +Infinity.
3. Second log (:80). 0 / (dl + 0) is 0 and log(0) is -Infinity.
4. Sum. +Infinity + -Infinity is NaN (IEEE 754); the boost keeps it NaN.
5. Clamp (:81). NaN > 0.0d is false, so the method returns 0.0, for every hit of every term on the field. The clamp was written for the paper's negative scores (class javadoc, :30-32); it also swallows the NaN.
6. explain() (:96-107, :117-125) prints the two logs as children, Infinity = term weight, computed as log(1 + freq /(mu * P)) and -Infinity = document norm, computed as log(mu / (dl + mu)), under 0.0 = score(LMDirichletSimilarity, freq=1.0).

For mu > 0 the two logs equal one log, log((freq + mu * P) / ((dl + mu) * P)). For P > 0 its denominator stays positive at mu = 0, where it equals log(freq / (dl * P)): for doc "a", log(1 / 0.375) = 0.98083. The smallest positive mu values already return that value (mu = 1e-10 and Float.MIN_VALUE both give 0.98082924, from 8.0.0, when score() moved to double). Only the order of operations fails at the boundary.

"a b c d" scores 0.0 at every mu: that is the documented clamp. Indexed in reverse ("b c", "a b c d", "a"), mu = 0 returns "a b c d" first and "a" last, both at 0.0, while Float.MIN_VALUE and 2000 return "a" first. A second probe with four documents shows the same for the three-argument constructor and for the disjunction a OR b.

A Z3 model checked the arithmetic for all legal inputs (the single-log identity and limit in real arithmetic, IEEE binary64 at mu = +0.0), and a bounded-exhaustive run of the stock jar found mu = 0 scoring 0 in 963,900 of 963,900 cases, 276,714 of them with a positive limit.

• Who. Opt-in only: applications that choose LMDirichletSimilarity (marked @lucene.experimental) and set mu to exactly 0. Then every query on that field scores every hit 0.0.
• Downstream. Solr's LMDirichletSimilarityFactory reads mu from the schema and calls the constructor with no extra check, and Elasticsearch and OpenSearch pass a configured mu to the constructor too (read in their source, not run). There a configuration value of 0 is enough.
• Silent. No exception, and the similarity logs nothing; the same documents match, only with score 0.0 (tied hits in doc-id order with the default collector). The one hint is explain()'s Infinity and -Infinity children.
• Not affected. The default mu = 2000 and, from 8.0.0 on, every positive mu in a realistic range: tried from Float.MIN_VALUE up to Integer.MAX_VALUE, the largest value upstream's test uses (its comment: // we just limit the test to "reasonable" mu values but don't enforce this anywhere, TestLMDirichletSimilarity.java:37-40). Only absurd values from about 1e17 up zero the scores too: the arguments of the logs round to 1 in double (doc "a": 2.220446E-16 at 1e16, 0.0 at 1e17 and above), in the current and in the single-log form alike; that is a separate precision limit, not this bug. Before 8.0.0 score() ran in float, so a tiny mu such as Float.MIN_VALUE misbehaved as well; mu = 1e-10 works in every release.
• Nothing is written to the index. Scores are computed at search time, so a fixed jar corrects existing indexes.
• How likely: rare. No earlier report was found since the class first shipped in 4.0.0-ALPHA (2012).

Public API only; the test sits in org.apache.lucene.search.similarities next to TestLMDirichletSimilarity but needs no package access. It asserts only "above 0", not a value, because no document promises the mu -> 0 limit. It does not show the ranking effect (the top hit is "a" because it has the lowest doc id); the reversed-order probe in the full explanation does.

Regression test: this same test; it passes with fix A. With fix B the same setup becomes expectThrows(IllegalArgumentException.class, () -> new LMDirichletSimilarity(0f)).

cp $REPRO/TestLMDirichletMuZero.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/similarities/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestLMDirichletMuZero -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, seed DEADBEEF, assertions on (off: the same failure)
TestLMDirichletMuZero > testMuZeroScoresBestMatchAboveZero FAILED
    java.lang.AssertionError: doc 0 scores 0.0, expected > 0
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:7533574B2D94FB58]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.search.similarities.TestLMDirichletMuZero.testMuZeroScoresBestMatchAboveZero(TestLMDirichletMuZero.java:52)
1 test completed, 1 failed

# explain(body:a, doc 0) at mu = 0, same index
0.0 = weight(body:a in 0) [LMDirichletSimilarity], result of:
  0.0 = score(LMDirichletSimilarity, freq=1.0), computed as boost * (term weight + document norm) from:
    0.0 = mu
    Infinity = term weight, computed as log(1 + freq /(mu * P)) from:
      1.0 = freq, number of occurrences of term in the document
      0.375 = P, probability that the current term is generated by the collection
    -Infinity = document norm, computed as log(mu / (dl + mu))
    1.0 = dl, length of field
    0.375 = collection probability

• The input is accepted on purpose. The constructors define the legal range in their own message, "must be a non-negative finite value" (LMDirichletSimilarity.java:49-52, :58-61), and TestLMDirichletSimilarity.java:30-31 runs mu = 0 as the "minimum value".
• The documented zero case is a different one. LMDirichletSimilarity.java:30-32: "The formula as defined the paper assigns a negative score to documents that contain the term, but with fewer occurrences than predicted by the collection language model. The Lucene implementation returns {@code 0} for such documents." Doc "a" has more occurrences than predicted, and still gets 0.
• No discussion found of what mu = 0 scores. The range came from a generic rule, Robert Muir on LUCENE-7997 (#9045, 2017): "for unbounded parameters (such as normalization c, smoothing parameter mu) we treat them the same as BM25's k1 and just ensure non-negative/finite in the range check, and test the range of 0..Integer.MAX_VALUE". No comment in that thread discusses the scores at mu = 0.
• The clamp was written for negative scores. On LUCENE-3357 (#4430, 2011) David Mark Nemeskey chose option "2. if (score < 0) return 0" for Dirichlet's negative scores, and in the same thread Robert Muir wrote "we should fix the Nan/Inf score problems". Writing it as score > 0 is what turns NaN into 0.
• Upstream's stated rule opposes the outcome. Robert Muir on LUCENE-8010 (#9058, 2017): "We should issue IAE if a parameter is out of range rather than silently screw up later".
• Why upstream's own mu = 0 test passes. BaseSimilarityTestCase checks that scores are finite, non-negative, monotone and equal to explain(); a constant 0.0 passes all of them, and CheckHits.verifyExplanation does not compare a "computed as ... from:" explanation with the sum of its children (pattern at CheckHits.java:375; combined = value at :494-496), so the Infinity and -Infinity children are never checked against the 0.0 total.
• The sister class rejects its endpoint. The same LUCENE-7997 commit made LMJelinekMercerSimilarity reject its no-smoothing value: "lambda must be in the range (0 .. 1]" (LMJelinekMercerSimilarity.java:53-54, :61-62).

The strongest case for "intended": a Dirichlet prior needs mu > 0; mu = 0 is "no smoothing", outside the model the class is named after, and the paper's rank formula is undefined there. Upstream has tested mu = 0 since 2017, the clamp is documented, the class is experimental, and no complaint was found in the 15 years since the class was written (2011; first released 2012).

Why it fails: that argues for a different fix (reject mu = 0, as Jelinek-Mercer does), not for accepting the value and then silently scoring every hit 0. The passing test shows only that the test cannot see the problem. Rarity and the experimental tag lower the priority; they do not make the output intended. The cross-model reviews rated the argument "moderate" and agreed on the root cause and that the input is legal. Verdict: not intended, a defect.

Both options remove the silent all-zero ranking, and no document promises either; let upstream choose. Both were built and run with gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF, 2026-10-31).

In score() (LMDirichletSimilarity.java:76-82) replace the two logs by one (+7/-16 lines with explain()). In explain() the term weight and document norm children, Infinity and -Infinity at mu = 0, go; the leaves mu, freq, P and dl stay, and the description becomes computed as boost * log((freq + mu * P) / ((dl + mu) * P)) from:.

@Override
protected double score(BasicStats stats, double freq, double docLen) {
  double p = ((LMStats) stats).getCollectionProbability();
  // log(1 + freq / (mu * P)) + log(mu / (dl + mu)) written as a single log: it stays finite at
  // mu = 0, where the two terms are +Infinity and -Infinity and their sum is NaN
  double score = stats.getBoost() * Math.log((freq + mu * p) / ((docLen + mu) * p));
  return score > 0.0d ? score : 0.0d;
}

Trade-offs

• Same scores for mu > 0, up to rounding. A probe over every encodable document length (norm bytes 1-255), freq 1..min(dl, 200) and a grid of collection probabilities (6,715,472 cases per mu) compared both formulas, re-implemented in Java doubles with boost 1 and cast to float as SimilarityBase does: at the default mu = 2000, 105 of 2,787,361 positive scores differ, all below 1e-8, where the two-log form cancels; at mu = 1 and 0.001, 48 and 68 differ, all below 1e-7; at mu = 1e6, 223 differ (all at scores up to 3.2e-4) and 3 near-zero scores cross the clamp. In this sampled grid the differences sit among near-zero scores; a query boost scales them, so this is not a proof that no ranking changes.
• mu = 0 then scores log(freq / (dl * P)), the mu -> 0 limit every tiny mu already gives (doc "a": 0.98082924). That value is a choice for upstream; no document promises it.
• Finite at mu = 0 only for a finite P > 0. The default collection model always gives 0 < P <= 1 (LMSimilarity.java:151). CollectionModel is a public interface whose computeProbability may return any double; a custom model that returns P = 0 gets +Infinity at mu = 0 with the fix (stock: 0.0), the same +Infinity the stock code already gives it at every mu > 0 (run at Float.MIN_VALUE, 1 and 2000). Requiring P > 0 would be a separate question for upstream.
• explain() text changes for every mu; anything that parses "term weight" or "document norm" sees a difference.
• No API change. It keeps the accepted range and upstream's mu = 0 test case; whether it suits a bugfix release is upstream's call (some scores change slightly for mu > 0, and the explain() text changes). One Math.log per hit instead of two (not benchmarked).
• Variant (not run): keep the two logs and handle only mu == 0 as log(freq / (docLen * p)): bit-identical for mu > 0, at the cost of a branch in score() and in explain().

What the checks validated

Scratch worktree of 645d4fe14a20, gradle, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off;
• org.apache.lucene.search.similarities.* and TestCombinedFieldQuery in core, TestCombinedFieldQuery in sandbox and TestSpanSimilarity in queries, 5 iterations each: 30 classes, 740 test runs, 0 failures;
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest for core pass;
• explain at mu = 0 on the repro index: 0.98082924 = score(LMDirichletSimilarity, freq=1.0), computed as boost * log((freq + mu * P) / ((dl + mu) * P)) from: with children mu 0.0, freq 1.0, P 0.375, dl 1.0; mu = 1e-3 and 2000 give the stock values 0.9802047 and 8.325702E-4;
• earlier, the study's score-only version passed TestLMDirichletSimilarity and TestSimilarityBase (145 tests, 5 iterations) and every verifier's repro.

Both constructors check mu <= 0 and say "must be a positive finite value"; TestLMDirichletSimilarity drops its mu = 0 case (its "minimum value" becomes Float.MIN_VALUE).

if (Float.isFinite(mu) == false || mu <= 0) {
  throw new IllegalArgumentException(
      "illegal mu value: " + mu + ", must be a positive finite value");
}

Trade-offs

• Loud instead of silent, in line with LMJelinekMercerSimilarity (lambda in (0 .. 1]) and with the rule on #9058 ("issue IAE if a parameter is out of range rather than silently screw up later").
• A behaviour change: code or configurations that pass mu = 0 today get an exception when the similarity is built, so it may suit main (11.0) better than a 10.x bugfix release.
• Tiny positive mu stays accepted and keeps its scores; mu = 0 is the only value whose meaning changes.
• The regression test becomes expectThrows on the constructor.

What the checks validated

Same worktree and command: the minimal test now stops at the constructor with IllegalArgumentException: illegal mu value: 0.0, must be a positive finite value; the same suites without the repro: 29 classes, 735 test runs, 0 failures. Format and lint checks were not run for this option.

Method: non-formal first signal (the hunt's tier for a finding whose first evidence is an executed single-case probe), confirmed by semi-formal and formal checks.

• Ideation. In a cross-model ideation round, an independent model from another family (OpenAI GPT-5.6) suggested checking the mu -> 0 limit of the Dirichlet similarity. Its probe on the stock main jar printed mu = 0 score 0.0 against 1.0132525 at the next positive float; it was first misread as clean because the probe exits 0, and caught on a re-run.
• Continuity sweep. A reference-free sweep of the real jar over 62 similarity families at the edges of their accepted parameter ranges (28,272 cases) isolated LMDirichletSimilarity at mu = 0 as the only true discontinuity.
• Z3 model. 9 obligations, 0 mismatches: the two-log form equals the single log for mu > 0, and mu = +0.0 gives NaN before the clamp and +0.0 after it for every legal input.
• Bounded-exhaustive check of the stock jar. mu = 0 scores 0 in 963,900 of 963,900 cases, 276,714 of them with a positive limit; Float.MIN_VALUE equals the closed-form limit in all of them.
• JUnit. A public-API test then failed 2 of 2 on main with assertions on and off; the minimal repro above keeps the substance of one of its checks (the best match must score above 0) on a smaller, 3-document index.