IBSimilarity with DistributionLL returns -0.0f instead of 0 when the normalized term frequency is 0

DistributionLL.score returns -Math.log(lambda / (tfn + lambda)). When tfn is 0, or so small that tfn + lambda rounds to lambda, the quotient is exactly 1.0, Math.log(1.0) is +0.0 and the unary minus makes it -0.0. NormalizationH1(0) is legal and gives tfn = 0, so IBSimilarity(DistributionLL, LambdaDF, NormalizationH1(0)) scores every match -0.0f. Since 8.0.0 (LUCENE-7996) scorers must not return negative scores, and the test framework's AssertingScorer checks that with Float.compare, which rejects -0.0f. Upstream fixed DistributionSPL's negative scores, which iprithv traced on #16137 to the same -0.0 from -Math.log(1.0), with Math.max(0.0, ...) in 10.6.0 (#16139) but left LL as it is; the same one-line clamp fixes it. Effects are cosmetic: -0.0 in scores and explanations, test-framework assertion failures, and since 10.2.0 zero-score hits out of doc-id order. Default parameters with whole-number frequencies never reach it (a sweep found it at 0 of 199,665 such points per default configuration).

• DistributionLL.java:33-35: return -Math.log(lambda / (tfn + lambda));. For finite tfn >= 0 and finite lambda > 0, which is what IBSimilarity passes, the quotient is at most 1, so -0.0 is the only out-of-range value; it appears when tfn == 0 or tfn / lambda is below about 2^-53. (The method does not check its arguments; NaN or a negative tfn are outside this claim.)
• IBSimilarity.java:115-117 multiplies by the positive query boost, and SimilarityBase.java:185-186 casts to float: -0.0 reaches the scorer as -0.0f.
• tfn is 0 when c = 0 (NormalizationH1.java:57, also H2) or mu = 0; it is tiny when c or mu is tiny, or, depending on the normalization, when freq or a SynonymQuery term boost is tiny (in the study's sweep: with default H1 and H2, and Z with LambdaTTF, but never with default H3).
• Composite scorers that sum from +0.0 turn -0.0 back into +0.0, so the value reaches the collector from single-term-like scorers (TermQuery, SynonymQuery, PhraseQuery) or pass-through ones (a BooleanQuery whose optional clause does not match, BoostQuery).
• The same -0.0 in DistributionSPL was traced to -Math.log(1.0) by iprithv on #16137; #16139 clamps SPL (DistributionSPL.java:61-62). LL has no clamp.

DocScoreEncoder.java:32 orders hits with NumericUtils.floatToSortableInt, which puts -0.0f below +0.0f, while TopScoreDocCollector documents ties as "docID ascending" (:24-26) and TopDocs.merge compares with < / > (TopDocs.java:114-116), which treat the two zeros as equal. Example: three documents (f:a; f:a + g:x; f:z) and MUST f:a + SHOULD BoostQuery(ConstantScoreQuery(g:x), 0): doc 0 scores -0.0, doc 1 scores -0.0 + 0.0 = +0.0, and the hits come back [1, 0] instead of [0, 1] (10.1.0 gives [0, 1]). With one document per segment the order is [1, 0] with 1 slice and [0, 1] with 3 slices. #13935 (10.2.0) made HitQueue break ties with Float.compare (it changed only HitQueue.java and TopDocsCollector.java); #14714 (10.3.0) replaced that queue with DocScoreEncoder, which keeps the same order. Neither is claimed as a separate bug.

BaseSimilarityTestCase.java:389, AssertingSimilarity.java:83, 116 and AssertingScorable.java:36 check score >= 0, which accepts -0.0. So TestDistributionLL, whose DistributionTestCase picks c = 0 as the "minimum value" in one of its four random choices, computes -0.0 in those runs and still passes. A plain top-k TermQuery search through newSearcher() also passes, because the bulk-scorer path goes through AssertingScorable, not AssertingScorer.

• Who. Users of IBSimilarity with DistributionLL (public constructors, @lucene.experimental; the default similarity is BM25).
• When. The study's sweep of the default configurations (H1/H2 c = 1, H3 mu = 800, Z z = 0.3) found it at 0 of the 199,665 integer-freq points (freq 1, 2 and 1000) per configuration; the same sweep, 332,775 points per configuration over five freqs, does find -0.0f at freq 2^-31 and 1e-12 with the H1 and H2 defaults and with Z + LambdaTTF. Reachable with c = 0 (H1, H2) or mu = 0, where every document scores 0; with c or mu of about 1e-9; or, depending on the normalization (default H1 or H2, not H3), with a SynonymQuery term boost of about 1e-20 (the allowed range is (0, 1]) or a sloppy freq of about 2^-31.
• What they see. -0.0 in ScoreDoc.score and explanations; AssertionError: -0.0 from Lucene's test framework in their own tests run with -ea when the score goes through the Scorer path; and since 10.2.0 zero-score hits out of doc-id order, with the order depending on how segments are sliced.
• Not affected. Matching, hit counts and totals, and ranking among non-zero scores; the score is -0.0f with assertions on and off (only AssertingScorer's check is a Java assert that needs -ea). Lucene's CI computes the value but does not catch it: TestDistributionLL can draw c = 0 or mu = 0 and then scores -0.0, but its score >= 0 checks accept that, and RandomSimilarity uses default c and mu.

One test class, one method, nothing random (the seed only satisfies LuceneTestCase). Public API only (IBSimilarity, DistributionLL, LambdaDF, NormalizationH1, Similarity.scorer, SimScorer.score, the FieldStats / TermStats records); it sits in the similarities package like Lucene's other similarity tests but uses nothing package-private. With 100 documents and docFreq 10, lambda = 11 / 101; c = 0 makes tfn = 0.

On 9.x and 10.x the records are called CollectionStatistics / TermStatistics (renamed on main by #15929); the release runs used a copy with only that rename.

Regression test: this same test. It fails on stock main and passes with the fix.

cp $REPRO/TestDistributionLLNegativeZero.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/similarities/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.search.similarities.TestDistributionLLNegativeZero -Ptests.seed=DEADBEEF -Ptests.asserts=true
# same failure with -Ptests.asserts=false

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true
TestDistributionLLNegativeZero > testZeroScoreIsPositiveZero FAILED
    java.lang.AssertionError: expected:<0.0> but was:<-0.0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:B791AEA760E3E4E]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.search.similarities.TestDistributionLLNegativeZero.testZeroScoreIsPositiveZero(TestDistributionLLNegativeZero.java:34)
BUILD FAILED

# -Ptests.asserts=false: the same AssertionError; 10.6.0, 10.5.2 (-ea and -da) and 9.12.3: the same
# with the clamp: 1 test, 0 failures, both ways

# end-to-end probes on main (plain IndexSearcher, lucene-core only)
top score = -0.0 (raw bits 0x80000000)                 # one doc "lucene", TermQuery f:lucene
doc 1 score 0.0 (raw bits 0x0)                          # MUST f:a + SHOULD BoostQuery(ConstantScore(g:x), 0)
doc 0 score -0.0 (raw bits 0x80000000)
order = [1, 0], scores equal by == : true               # 10.1.0: order = [0, 1]

• Contract. lucene/MIGRATE.md:1368-1372: "### Scorer must produce positive scores (LUCENE-7996)" / "Scorers are no longer allowed to produce negative scores." The enforcing check is assert Float.compare(score, 0f) >= 0 : score; (AssertingScorer.java:131, added by the same change).
• The sibling was fixed as a bug. #16137 (romseygeek: "Impacts and competitive iterators all assume that scores can't be negative, a contract which this Similarity breaks"); iprithv traced SPL's value to "-0.0, coming from -Math.log(1.0)"; #16139 added // Clamp to avoid negative scores due to floating-point rounding errors / return Math.max(0.0, -Math.log(fraction));, listed under 10.6.0 Bug Fixes, approved by msfroh ("it sounds like this is generally going to round -0.0 to 0.0"). Neither thread mentions LL. #16139 is framed more broadly, as negative scores "due to floating-point rounding errors", so it is a close precedent rather than a ruling on LL.
• Legal trigger. NormalizationH1(float c) rejects only non-finite or negative c; DistributionTestCase picks c = 0 and mu = 0 as the "minimum value"; SimScorer.score asks only that freq be "finite and positive".

The strongest case for "not a bug" (rated moderate by the cross-model refute review): -0.0f == 0.0f in Java; Lucene's similarity checks use score >= 0 and accept it; the SimScorer javadoc does not spell out the sign of zero; IBSimilarity and DistributionLL are @lucene.experimental; and c = 0 makes every score 0, so nobody would run it.

Why it fails: the rule Lucene enforces for scorers is Float.compare(score, 0f) >= 0, and upstream fixed SPL's negative scores, which iprithv traced to the same -0.0 from -Math.log(1.0), as a bug in 10.6.0. c = 0 is validated as legal and tested on purpose, and the tiny-boost route needs no degenerate parameter. The rest argues for Low priority, which is how it is classed, not for intent; the score >= 0 checks explain why the tests miss it.

One fix was validated through gradle in a scratch worktree of 645d4fe14a20; two equivalent rewrites and a test-side complement are listed but were not run.

DistributionLL.java +2/-1:

@@ -31,7 +31,8 @@
 
   @Override
   public final double score(BasicStats stats, double tfn, double lambda) {
-    return -Math.log(lambda / (tfn + lambda));
+    // Clamp to avoid returning -0.0 when tfn + lambda rounds to lambda
+    return Math.max(0.0, -Math.log(lambda / (tfn + lambda)));
   }

Trade-offs

• The same idiom and comment style as DistributionSPL.java:61-62. Math.max(0.0, -0.0) is +0.0 in Java.
• Math.max(0.0, x) returns x for every x > 0 and for NaN, so for the inputs IBSimilarity passes (finite tfn >= 0, finite lambda > 0) only -0.0 changes. The differentials agree: the study's 5,000,000 pairs found the clamped copy different from the real class only where the real class returns -0.0, and a re-run of 10,000,000 pairs (the study's sampling scheme plus typical inputs) found the same. For invalid inputs (a negative tfn) the clamp also turns a genuinely negative result into +0.0.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle with --offline (in the fresh worktree gradlew first fetched its own wrapper jar), OpenJDK 25.0.3, seed DEADBEEF. Stock: the repro fails with assertions on and off. With the fix: the repro passes both ways; the whole org.apache.lucene.search.similarities test package plus TestSynonymQuery pass (assertions on: 28 suites, 129 tests, 0 failures; off: 28 suites, 129 tests, 0 failures); :lucene:core:checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass with the fix and the test file in place.
• Earlier runs: the study's 9 tests pass with it (AssertingScorer accepts every score, tie order [0, 1] for 1 and 3 slices); TestDistributionLL and TestDistributionSPL pass; the cross-model reviewer's 5 tests pass and TestDistributionLL passes 10/10 over 5 iterations.

return 0.0 - Math.log(lambda / (tfn + lambda)); also maps -0.0 to +0.0 (0.0 - 0.0 = +0.0); for any other result 0.0 - y equals -y exactly, so every other value stays the same, including a genuinely negative result from invalid inputs, which it does not clamp.

Trade-offs

Less obvious to a reader than the clamp, and different from the SPL fix. Not run as a fix; the re-run differential (10,000,000 pairs) found it equal to the clamp on every pair.

Mathematically equal, and never negative for finite tfn >= 0.

Trade-offs

More accurate for small tfn, but it changes the last bits of many scores: in 5,000,000 typical inputs it changed the double result of 502,054 (about 10%) but no float score; with the study's extreme sampling it changed 577,673 float scores that were not -0.0. Not run as a fix.

Use Float.compare(score, 0f) >= 0 in BaseSimilarityTestCase and AssertingSimilarity, as AssertingScorer does, so this class of bug is caught.

Trade-offs

Not a fix on its own. Untested: other similarities might then fail too.

Method: non-formal (code reading prompted by an upstream issue). Study round 3, an SMT experiment on similarities and impacts (2026-10-22, main 7cecf79523).

• The experiment ran an SMT signed-zero range check over the similarities. Its write-up says it looked at LL after reading #16137, so the study does not credit the SMT method with the find: it prompted the look but did not find it independently.
• An independent verifier wrote 9 deterministic JUnit tests (6 LL tests fail, 3 SPL controls pass), re-ran the #16137 doTestBoosts scenario with LL (80 of 80 searches trip AssertingScorer, SPL 0 of 80), compared a clamped copy on 5,000,000 random pairs, and swept the default configurations (0 of 199,665 integer-freq points each; -0.0f only at freq 2^-31 and 1e-12, with the H1, H2 and Z defaults).
• 2026-10-29 the repro was reduced to one public-API test; 2026-11-01 it was re-run on 645d4fe14a20 and three releases, and a cross-model reviewer reproduced it with its own tests.