BaseSimilarityTestCase's "scores don't decrease as terms get rarer" check never runs: it scores `term` again instead of `prevTerm`

BaseSimilarityTestCase, the ready-made test suite for Similarity implementations in the published lucene-test-framework, documents that testRandomScoring checks "scores don't decrease as terms get rarer, e.g. score(term=N) >= score(term=N+1)". The code builds the rarer term, prevTerm (docFreq - 1, totalTermFreq - 1), but then scores the original term again, so it compares a score with itself and, for a deterministic Similarity, the check can never fail; prevTerm is used only in the failure message. A Similarity whose weight is boost * log1p(docFreq) (docFreq 5 scores 0.621, docFreq 4 scores 0.558) passes in every run tried, on main and on every release from 8.0.0 to 10.6.0. The defect is test-only: no search result, score or pruning decision changes. With the argument corrected, four of Lucene's own models fail the check (DFR basic model G and Indri by their formulas, the IB distribution SPL often, the Dirichlet normalization H3 by an ulp or two), so a fix also needs a decision: exempt them (validated here) or drop the documented property.

testRandomScoring's javadoc lists, among the checks it does, "scores don't decrease as terms get rarer, e.g. score(term=N) >= score(term=N+1)" (BaseSimilarityTestCase.java:263 at 645d4fe14a20). For each random draw, doTestScoring computes score = similarity.scorer(boost, corpus, term).score(freq, norm) (:380, :385) and later, under the comment "check score(term-1), given the same freq/norm it should be >= score(term) [scores non-decreasing as terms get rarer]" (:476-477), does:

TermStats prevTerm =
    new TermStats(term.term(), term.docFreq() - 1, term.totalTermFreq() - 1);   // :479-480
SimScorer prevTermScorer = similarity.scorer(boost, corpus, term);              // :481, term, not prevTerm
float prevTermScore = prevTermScorer.score(freq, norm);                         // :482
...
if (prevTermScore < score) {                                                    // :500
  fail("score(" + freq + "," + (prevTerm) + ")=" + prevTermScore + " < score(" ...   // :507

Both scorers are built from the same arguments and scored with the same freq and norm, so prevTermScore == score for any deterministic Similarity and the comparison is always false. prevTerm is read only to build the failure string, which is why no unused-variable lint flags it; in the shipped 8.0.0 and 10.6.0 test-framework jars the Similarity.scorer call is likewise passed the term parameter (javap -c).

The repro's Similarity has weight boost * log1p(docFreq) and score weight * log1p(freq) / (1 + norm). At boost 1, freq 1, norm 1: docFreq 5 gives 0.621, docFreq 4 gives 0.558, so a rarer term scores 10% lower. It passes every other check of testRandomScoring (finite, non-negative, non-decreasing in freq, non-increasing in norm, explanation equals score). On main it passes testRandomScoring 1/1 at seed DEADBEEF with -ea and -da, and 20 of 20 iterations from seed 5EED0073. With prevTerm scored, every one of 20 iterations throws the rarer-term check's own message.

SimScorer.score requires monotonicity in freq and in the unsigned norm only (Similarity.java:207-216); no documented Similarity contract requires docFreq monotonicity, and no core code relying on it was found. The property is a promise of the test framework, which also sets its own bar: "if there is some bug in a given Similarity that this test fails to catch then this test needs to be improved!" (BaseSimilarityTestCase.java:48-51).

• Who. Authors of Similarity implementations who extend BaseSimilarityTestCase, a public class of the published lucene-test-framework artifact since 8.0.0, and Lucene's own 23 concrete suites in lucene/core. testRandomScoring runs for every subclass, so the check is skipped by default, on every seed. How many outside projects extend the class was not measured.
• What they lose. A false pass: a Similarity that scores a rarer term lower (for example an IDF with a sign error) passes the check documented to catch it. The check's rarer term has docFreq and totalTermFreq each one lower, with the same freq, norm and field statistics, so models that use totalTermFreq (Indri, G, SPL, H3) react to both.
• What the dead check hides in Lucene itself. With prevTerm scored, four models score some rarer terms lower:

• On ordinary statistics too (1000 documents of average length 100, term df 500 vs 499, freq 1): DFR G-L-H1 at document length 300 scores 0.848171 for the common term and 0.8481204 for the rarer one; Indri (mu 2000) at length 100 scores -4.6032634 vs -4.6042166. BM25 and ClassicSimilarity score the rarer term higher at every length tried. These are properties of the models, not ranking bugs: no Lucene API documents docFreq monotonicity.
• Library behaviour. Unchanged and correct; only the test framework's promise is broken.

The test builds an anonymous BaseSimilarityTestCase whose getSimilarity returns a Similarity with weight boost * log1p(docFreq), then calls the public testRandomScoring() inside expectThrows(AssertionError.class, ...). Single thread. It calls testRandomScoring() on an anonymous suite instance, so BaseSimilarityTestCase's @BeforeClass (which builds a one-document index) does not run for it; that is fine, because testRandomScoring and doTestScoring never read that index (READER and DIR appear only in beforeClass and afterClass, BaseSimilarityTestCase.java:55-92). The statistics are drawn by the framework itself; for this Similarity the stock framework cannot fail on any draw, because the only check that could reject it is the dead one, so the outcome does not depend on the seed (22 of 22 runs tried).

Releases need small API variants of the same test (same body and assertion). Before main the statistics classes are CollectionStatistics / TermStatistics (main renamed them to FieldStats / TermStats in #15929). On 9.12.0-10.6.0 only those two names differ. On 9.1.0-9.11.1 the test also overrides Similarity.computeNorm, which is abstract there (concrete from 9.12.0). On 8.0.0-9.0.0 it also uses the pre-9.1 test-framework packages. The version table used two variants: one for 8.0.0-9.0.0 and one for 9.1.0-10.6.0 (with the computeNorm override).

Regression test: this same test. With the check fixed it passes, and it fails again if the check goes dead. Natural homes: next to the suites in lucene/core/src/test/org/apache/lucene/search/similarities/ (as is), or beside TestBaseExplanationTestCase in the test framework's own tests.

# main 645d4fe14a20, stock jars, seed DEADBEEF, -ea (-da: identical)
1) testRandomScoringRejectsRarerTermScoringLower(org.apache.lucene.search.similarities.TestBaseSimilarityRarerTermCheck)
junit.framework.AssertionFailedError: Expected exception AssertionError but no exception was thrown
	at __randomizedtesting.SeedInfo.seed([DEADBEEF:AC09ECDEFB034D93]:0)
	at org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1432)
	at org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
	at org.apache.lucene.search.similarities.TestBaseSimilarityRarerTermCheck.testRandomScoringRejectsRarerTermScoringLower(TestBaseSimilarityRarerTermCheck.java:49)
Tests run: 1,  Failures: 1

# gradle, the file copied into the core tests, seed DEADBEEF, -Ptests.asserts=true and false
TestBaseSimilarityRarerTermCheck > testRandomScoringRejectsRarerTermScoringLower FAILED
    junit.framework.AssertionFailedError: Expected exception AssertionError but no exception was thrown
<testsuite name="org.apache.lucene.search.similarities.TestBaseSimilarityRarerTermCheck" tests="1" skipped="0" failures="1" errors="0"

# with the recommended fix: 1 test, 0 failures (gradle and standalone, assertions on and off)

• The contract says the check runs. testRandomScoring's javadoc lists "scores don't decrease as terms get rarer, e.g. score(term=N) >= score(term=N+1)" (BaseSimilarityTestCase.java:263); the comment at :476-477 says the rarer term's score "should be >= score(term)"; the failure message prints prevTerm (:507).
• Intent at creation. LUCENE-7997 (42717d5f4bb, by rmuir, 2017-10-24; migrated as #9045) added the check; the patch note on #9045 says it tests a "nearby slightly rarer term to ensure relevance doesn't go backwards in those cases too". LUCENE-8010 (#9058) repeats the property score(term=N) >= score(term=N+1).
• Never disabled on purpose. No TODO, @AwaitsFix, flag or comment says the check is off. Seven later commits edited the block and kept term, among them LUCENE-8007 (ca5f9b34576), LUCENE-8116 (8fd7ead940f, by jpountz, which moved it to today's scorer API), LUCENE-9537 (9cc5c9b7983, which added the Indri non-negativity exemption inside this very block at :485) and #15929 (fa0e704d2a0, the TermStats rename).
• Lucene tests its test helpers. TestBaseExplanationTestCase in the test framework's own tests "Tests that the BaseExplanationTestCase helper code ... are checking what they are supposed to", with the same expectThrows(AssertionError.class, ...) shape as the repro.

Strongest case for "not a bug": docFreq monotonicity is not part of the Similarity API (Similarity.java:207-216), and Lucene's own BasicModelG and Indri violate it on ordinary statistics, so perhaps nobody wants the property; the defect is test-only and changes no result; and the repro returns one fixed Similarity although getSimilarity's javadoc asks for "a new similarity with all parameters randomized within valid ranges" (BaseSimilarityTestCase.java:94), as the cross-model reviewer noted.

Why it fails: whatever the property's merit, the published javadoc and the code disagree, and the class's own standard calls an uncaught Similarity bug a test defect. The authors clearly meant the check to run (#9045's note, the comment, the message). That Lucene's own models fail the live check is a reason to decide explicitly (exempt, narrow or drop), not to keep a check that silently passes everything. The reviewer rebuilt the test with a fresh randomized Similarity per call and got the same result: it fails on main, every rarity-sensitive case fails with assertions off over 5 iterations, and it passes with the fix.

One fix (A) was validated with gradle in a scratch worktree of 645d4fe14a20. Its exemption list was found by sampling: the one-word fix and three earlier variants were run first and failed (see the option "How the exemption list was found"). The other options were not run.

In doTestScoring, pass prevTerm to similarity.scorer, and skip only the final comparison for IndriDirichletSimilarity, a DFRSimilarity with BasicModelG or NormalizationH3, and an IBSimilarity with DistributionSPL or NormalizationH3. The exempted models still run the rest of the block (the rarer term's score must be finite, match its explanation, and be non-negative except for Indri). 1 file, +27/-2, exact comparison. It follows the existing pattern: the same block already exempts Indri from the non-negativity check by instanceof.

-        SimScorer prevTermScorer = similarity.scorer(boost, corpus, term);
+        SimScorer prevTermScorer = similarity.scorer(boost, corpus, prevTerm);
         ...
-        if (prevTermScore < score) {
+        if (prevTermScore < score && rarerTermsNeverScoreLower(similarity)) {

+  private static boolean rarerTermsNeverScoreLower(Similarity similarity) {
+    if (similarity instanceof IndriDirichletSimilarity) {
+      return false;
+    }
+    if (similarity instanceof DFRSimilarity dfr) {
+      return dfr.getBasicModel() instanceof BasicModelG == false
+          && dfr.getNormalization() instanceof NormalizationH3 == false;
+    }
+    if (similarity instanceof IBSimilarity ib) {
+      return ib.getDistribution() instanceof DistributionSPL == false
+          && ib.getNormalization() instanceof NormalizationH3 == false;
+    }
+    return true;
+  }

Trade-offs

• The test framework names four concrete classes (it already imports IndriDirichletSimilarity), and an outside Similarity that legitimately violates the property has no way to opt out (see the hook option).
• The list was found by sampling, so a rare violating combination in another model cannot be excluded; in a randomized CI it would show up as an occasional failure.
• The H3 exemption covers 1-2-ulp drops that look like float rounding rather than an exact-math violation, and it switches the comparison off for every DFR and IB similarity that uses H3. SPL's larger drops were not analysed.
• The diff uses pattern-matching instanceof (Java 16+): fine on main and 10.x (Java 21); a backport to branch_9x (Java 11) needs explicit casts.
• Models from three families (DFR, IB, Indri) fail the property, which is also an argument for dropping or narrowing the promise (option C).

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle --no-daemon, OpenJDK 25.0.3: checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain of the test framework pass.
• :lucene:core:test --tests 'org.apache.lucene.search.similarities.*' (26 classes, 118 tests): 0 failures for seeds DEADBEEF, 12345678, CAFEBABE, 0BADF00D with assertions on and DEADBEEF with assertions off.
• Every suite's testRandomScoring with -Ptests.iters=100 for 6 root seeds: 13,800 test runs, 0 failures.
• The minimal repro passes with assertions on and off (gradle, and standalone with the fixed class shadowed in); a diagnostic copy that also checks the message passes 20 of 20 iterations, so the rejection comes from the rarer-term check itself.

Trade-offs

Shows why the one-word fix alone is not enough and why a fixed ulp tolerance does not help: SPL's drops reach 0.33%, and H3's rounding drops reach 2 ulps.

Delete the comparison (or the whole rarer-term block) and the javadoc bullet "scores don't decrease as terms get rarer"; the repro test is then dropped too.

Trade-offs

The smallest change, with no risk of new test failures, and what the code has effectively done since 2017 for deterministic similarities; but it gives up a check that the remaining models pass and that would catch an inverted IDF in an outside Similarity. A reasonable choice if maintainers do not want an exemption list found by sampling.

A protected method on BaseSimilarityTestCase, for example "does this Similarity guarantee that rarer terms never score lower?", default true, that the affected suites override.

Trade-offs

Lets outside authors opt out too and keeps model names out of the framework; but doTestScoring is private static, and H3 is drawn at random inside the shared final getSimilarity of BasicModelTestCase and DistributionTestCase, so the hook needs the drawn Similarity as an argument: a larger diff (SPL, G and Indri are fixed per suite and could be plain per-suite overrides).

• Context. The 2026-10-22 study's SMT experiment on similarities and impacts encoded the scoring contracts that dynamic pruning relies on (monotone in freq and norm, finite, non-negative) for SMT solvers, using the statistics domains of BaseSimilarityTestCase#testRandomScoring. Reading that method to derive the domains, the AI tool saw that the rarer-term block scores term, not prevTerm.
• Confirmation. A meta-test that expects testRandomScoring to reject an anti-IDF Similarity failed on main for 3 seeds; with the one-word fix it passed, and the core suites showed which models violate the property.
• Independent verification. A separate verifier wrote its own tests (reflection into doTestScoring, a conformance suite, javap of the shipped bytecode, an 8-seed sweep of the 23 core suites with the stock and the fixed framework) and confirmed it as a test-only bug.
• Tier. Non-formal: the solver work was the context; the bug itself was found by reading code.