BayesianScoreQuery rounds the pruning threshold up to 1e-7, so top-k search can lose the best hits

BayesianScoreQuery (new in 10.5.0, experimental) turns its inner query's score into a probability with a sigmoid, sigmoid(alpha * (score - beta) + logit(baseRate)). To support top-k pruning it inverts the sigmoid: the minimum competitive probability the collector asks for becomes a minimum inner score for the inner scorer. Before inverting, BayesianScoreScorer#setMinCompetitiveScore clamps the request to at least 1e-7, while the scores themselves are not clamped. So when the minimum the collector asks for is below 1e-7, the inner scorer is told to skip documents that can still win.

With the default IndexSearcher.search(query, n), pruning starts once more than max(1,000, n) hits are counted; in the repros the search then in effect stops, returns the first matches in doc-id order and reports totalHits as "1001+", with no error. In the repro search(q, 1) returns doc 3900 (1.16228565E-8) instead of doc 5000 (1.8876845E-8): the requested minimum 1.1622857E-8 is forwarded as BM25 threshold 2.6974144, above every score in the index. It needs a BayesianScoreQuery that does the top-level scoring (alone, next to filters, or as the only active signal of a LogOddsFusionQuery; where the fusion scorer itself runs, fused queries are not affected) and a calibration that puts the collector's minimum below 1e-7, which parameters estimated by BayesianScoreEstimator reached on a 100,000-document index. It has been there since the class was added (#15827) and affects 10.5.0 to 10.6.0 and main. Dropping the lower clamp fixes it; LFV-450, a separate bug in the same four lines, needs its own fix.

1. Pruning starts after max(1,000, n) hits. IndexSearcher.search(Query, int) (IndexSearcher.java:623-625) calls searchAfter, which uses TopScoreDocCollectorManager(cappedNumHits, after, TOTAL_HITS_THRESHOLD) with TOTAL_HITS_THRESHOLD = 1000 (:100, :598-599); the manager keeps Math.max(totalHitsThreshold, numHits) (TopScoreDocCollectorManager.java:69), scoring in ScoreMode.TOP_SCORES (TopScoreDocCollector.java:59). Once a collector has counted more than that, it calls scorer.setMinCompetitiveScore(Math.nextUp(topScore)) with the k-th best score so far (TopScoreDocCollector.java:116-120, :154-162); with several slices it can also take a higher minimum found by another slice (:99-100, :137-151). In the repro n = 1.
2. The scores are not clamped. score() returns sigmoid(alpha * (innerScore - beta) + logitBaseRate) (:263-266), with logitBaseRate = log(baseRate / (1 - baseRate)) or 0 (:75-79); getMaxScore (:274-278) and explain() (:199-201) use the same transform. Probabilities below 1e-7 are returned and ranked.
3. The threshold is clamped. BayesianScoreScorer#setMinCompetitiveScore (:281-292), for 0 < minScore < 1 (:286):

float clamped = Math.max(1e-7f, Math.min(1f - 1e-7f, minScore));   // :287
float logitMin = (float) Math.log(clamped / (1f - clamped));        // :288
float innerMin = (logitMin - logitBaseRate) / alpha + beta;         // :289
in.setMinCompetitiveScore(Math.max(0f, innerMin));                   // :290

1. What the inner scorer gets. For any request below 1e-7 it gets the inner score whose probability is 1e-7: beta + (logit(1e-7) - logit(baseRate)) / alpha, about beta - 2.30/alpha with base rate 1e-6 and beta - 16.12/alpha without one. It may then skip every document whose probability lies between the request and 1e-7. The upper clamp only lowers a threshold, so it is safe.
2. Not needed to avoid log(0). The guard at :286 already excludes 0 and 1; for every positive float the logit is finite (about -103 at the smallest one). Overflow in the rest of the inversion, with extreme legal parameters, is a separate matter.

The collector scores 1,001 documents, then asks for minimum score 1.1622857E-8. The stock code forwards BM25 threshold 2.6974144 (= 5 - 2.30), so the TermQuery scorer skips the remaining 100 matches and search(q, 1) returns doc 3900 with totalHits "1001+". Without the lower clamp it forwards 0.5452175, scores all 1,101 matches, then forwards 1.03018 and returns doc 5000. The skipping is done by the inner TermScorer, which acts on the threshold through its impacts because BayesianScoreQuery forwards setTopLevelScoringClause (:238-241, TermScorer.java:61-74, :127-131).

• Who is hit. A BayesianScoreQuery at top level, or as the only scoring clause of a conjunction (also reproduced with FILTER clauses), on 10.5.0 or later.
• Fused queries. Where the fusion scorer runs they are not affected: LogOddsFusionScorer.setMinCompetitiveScore only updates its own block propagator and never forwards the threshold to its sub-scorers (LogOddsFusionScorer.java:213-218). But LogOddsFusionQuery hands a lone active signal's scorer straight to the collector (LogOddsFusionQuery.java:230-233), for example on a segment where the other signals have no match, and rewrites to the clause itself when only one clause is left (:359-360, :393-394); there the bug fires too. Executed on main with assertions on and off: on the repro's index, fusing body:a with a signal on an absent term returns doc 3900 instead of doc 5000, and so does a two-segment index where the second signal matches only in the second segment; both pass with the fix, and a control with both signals active passes on stock. Open PR #16410's description says "Single-active-signal leaves and MatchNoDocsQuery rewrites could also bypass or change the declared fusion model". Fusion with a vector signal that has no hit on some segments is plausible but was not run.
• No special setting. Plain IndexSearcher.search(query, n), the default codec, a term or a disjunction inner query, single-threaded or sliced search, assertions on or off. Exhaustive collection is not affected.
• What it takes. More than max(1,000, n) matches counted by a collector; a minimum competitive score below 1e-7 from the collector, which sends Math.nextUp of the k-th best score so far, so a k-th best probability below 1e-7: with base rate 1e-6 (the smallest BayesianScoreEstimator returns), an inner score more than 2.3/alpha below beta; without a base rate, more than 16.1/alpha below beta; and better hits later in doc-id order that the inner scorer can skip. The estimator sets beta to the median and alpha to 1/std of pseudo-query scores (BayesianScoreEstimator.java:41, :147, :162) and clamps the base rate to [1e-6, 0.5] (:55-56, :170).
• Measured reach. With parameters from BayesianScoreEstimator.estimate on a 100,000-document default-codec index of mostly rare tokens (alpha 1.2305909, beta 4.930353, base rate 1.7e-5), a single-term query on a term in one third of the documents: search(q, 10) returned docs 4..112 in doc-id order and none of the ten true top hits. In the same verifier's scratch sweep (its code was not kept), a 200,000-document index with the term in 10% of the documents calibrated to 1.29e-7 and was not affected, so the window is real but bounded.
• Silent. No exception; totalHits "1001+" is the normal lower bound of a pruned search.
• Likelihood: rare. New and @lucene.experimental (BayesianScoreQuery.java:41), and it needs the collector's minimum below probability 1e-7. Its javadoc presents the probabilities as "suitable for combination with other probability signals via LogOddsFusionQuery" (:27-30); fusion shields the signals only where the fusion scorer runs, so fused use lowers the exposure but does not remove it. Nobody has reported it.

The test sits in org.apache.lucene.search, as Lucene's own search tests do, but uses no package-private member. It pins TestUtil.getDefaultCodec() (LuceneTestCase otherwise randomizes the codec) and uses ByteBuffersDirectory, the IndexWriterConfig default StandardAnalyzer and the default BM25 similarity. The base rate 1e-6 is the smallest one BayesianScoreEstimator returns. The assertion is the TOP_SCORES-equals-COMPLETE rule of CheckHits.checkTopScores. It is also the suggested regression test.

As a regression test it needs no edits: :lucene:core:check -x test passes with it in place, with either fix.

TestBayesianScoreTopScores > testTopHitIsNotPruned FAILED
    java.lang.AssertionError: exhaustive doc=5000 score=1.8876845E-8 shardIndex=-1, search(q, 1) doc=3900 score=1.16228565E-8 shardIndex=-1 expected:<5000> but was:<3900>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:A8DACEEFAE12CDB7]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.search.TestBayesianScoreTopScores.testTopHitIsNotPruned(TestBayesianScoreTopScores.java:49)
:lucene:core:test (FAILURE): 1 test, 1  failure

# the same message with -Ptests.asserts=false, and on the 10.6.0 release jars with -ea and -da

# probe on the same index and query, assertions off
# stock main 645d4fe14a20
[match] count(q)=1101
[doc] 3900 inner BM25=0.5452175 calibrated=1.16228565E-8
[doc] 5000 inner BM25=1.03018 calibrated=1.8876845E-8
[complete] top=doc=5000 score=1.8876845E-8 shardIndex=-1 totalHits=1101 hits
[search(q, 1)] top=doc=3900 score=1.16228565E-8 shardIndex=-1 totalHits=1001+ hits
[search(q, 1)] inner documents scored=1001
[search(q, 1)] call 0: requested minScore=1.1622857E-8 -> forwarded inner threshold=2.6974144
# with the lower clamp removed
[search(q, 1)] top=doc=5000 score=1.8876845E-8 shardIndex=-1 totalHits=1101+ hits
[search(q, 1)] inner documents scored=1101
[search(q, 1)] call 0: requested minScore=1.1622857E-8 -> forwarded inner threshold=0.5452175
[search(q, 1)] call 1: requested minScore=1.8876847E-8 -> forwarded inner threshold=1.03018

• The contract. "Tell the scorer that its iterator may safely ignore all documents whose score is less than the given {@code minScore}." (Scorable.java:45-51).
• The class contradicts itself. Scores, max scores and explanations are unclamped (:263-278, :199-201); only the threshold is floored. The method's own comment states the goal: "Invert the sigmoid to get the minimum inner score needed" (:282).
• Upstream tests. testMaxScoreCorrectness, testBaseRateMaxScoreCorrectness and testEstimatedMaxScoreCorrectness (TestBayesianScoreQuery.java:227-231, :330-334, :435-442) call CheckHits.checkTopScores, which compares a TOP_SCORES collector with a COMPLETE one (CheckHits.java:690-705). Their scores are far above 1e-7, so they never reach the clamp.
• Sibling pruning code rounds the other way. WANDScorer.scaleMinScore rounds a min competitive score down "in order to make sure that we do not miss any matches" (WANDScorer.java:107-120).
• History. #15827: "Since sigmoid is monotone, the original BM25 ranking is preserved exactly", with "WAND max score correctness" among its tests. #15948: the base rate shifts scores "without changing ranking order within a single query". No review comment on #15827, #15948, #16110 or #16410 discusses this clamp or pruning accuracy; no javadoc, CHANGES or MIGRATE text declares a floor.

The strongest case (an intended-behaviour reviewer of another model family and the intended-behaviour verifier): the class is @lucene.experimental, the symmetric 1e-7 clamp has been there since #15827, and LogOddsFusionScorer uses the same floor (CLAMP_MIN = 1e-7f, LogOddsFusionScorer.java:48-49). On that reading probabilities below 1e-7 are a saturated tail whose order does not matter, and the clamp is a numerical guard that also makes pruning cheaper.

Why it fails: the fusion floor applies to scores and max scores alike and never to a pruning threshold; this query returns the unclamped scores, so callers see and rank them. The lower clamp is not needed to avoid log(0) (the logit of any positive float is finite). The effect is not a reorder of near-ties: the search stops early and returns the first matches in doc-id order (none of the ten true top hits in the estimator case). "Experimental" allows the API to change, not wrong top hits. The cross-model lenses rated the case weak; the independent reproducer and the devil's advocate argued it through and rejected it.

Both options were run in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3), removed afterwards. "Closest suites" means TestBayesianScoreQuery (26 tests) and TestLogOddsFusionQuery (40 tests), the only tests that use these classes, plus the repro. Open PR #16410 by the class author rewrites the same method (dividing by the boost) and keeps Math.max(1e-7f, ...); either fix applies there the same way. Neither option fixes LFV-450, a separate unsafe-inversion bug in the same four lines: its minimal test still fails on 645d4fe14a20 with either one. LFV-450's recommended fix also passes this bug's minimal test, so that fix covers both.

Keep only the upper bound, which can only lower the threshold. The minScore > 0f && minScore < 1f guard already keeps the logarithm finite.

       if (minScore > 0f && minScore < 1f) {
-        float clamped = Math.max(1e-7f, Math.min(1f - 1e-7f, minScore));
+        float clamped = Math.min(1f - 1e-7f, minScore);
         float logitMin = (float) Math.log(clamped / (1f - clamped));
         float innerMin = (logitMin - logitBaseRate) / alpha + beta;
         in.setMinCompetitiveScore(Math.max(0f, innerMin));

Trade-offs

• Smallest change; removes exactly the direction that over-prunes.
• Keeps the existing float inversion, whose rounding can move the threshold by a few ulps; that is not part of this bug and is not claimed (in the probe the forwarded thresholds equal the BM25 scores, 0.5452175 and 1.03018).
• Pruning gets weaker only where it was wrong, below probability 1e-7.

What the checks validated

• Stock 645d4fe14a20: the repro fails through gradle with assertions on and off.
• With the fix: the repro passes with assertions on and off; the closest suites pass with seeds DEADBEEF and 5EED1234 (67 tests each, 0 failures); :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs) passes with the repro test in place.
• Five of the verifier test classes fail on stock (the sixth, the independent reproduction's control, passes by design); all six pass with the fix; the safety-property probe's bug check fires in 0 of 48 cases. The full Lucene test suite was not run.

Compute the logit as Math.log(minScore) - Math.log1p(-(double) minScore) and the inner threshold in double, take Math.nextDown twice of its float value and forward it only when it is positive.

      if (minScore > 0f && minScore < 1f) {
        double logitMin = Math.log(minScore) - Math.log1p(-(double) minScore);
        double innerMin = (logitMin - logitBaseRate) / alpha + beta;
        float threshold = Math.nextDown(Math.nextDown((float) innerMin));
        if (threshold > 0f) {
          in.setMinCompetitiveScore(threshold);
        }
      }

Trade-offs

• Rounds the threshold down, in the spirit of WANDScorer.scaleMinScore; that mitigates some rounding of the inversion but is not a proven conservative inverse: LFV-450's minimal test still fails with it.
• The two-ulp margin is a heuristic, not proven safe for every parameter combination; more code to review; slightly weaker pruning.

What the checks validated

• On 645d4fe14a20: the repro passes, the closest suites pass (67 tests, seed DEADBEEF) and :lucene:core:check -x test passes.
• Earlier, on the study's main ea57360912ed: every verifier repro, TestBayesianScoreQuery 26/26 and TestLogOddsFusionQuery 40/40.

#16410 ("Correct Bayesian score fusion semantics and pruning", by the class author, labelled Stale, last activity 2026-08-24) changes this method to minProbability = minScore / boost and keeps Math.max(1e-7f, Math.min(1f - 1e-7f, minProbability)) (read from its diff, not executed).

Trade-offs

Avoids two PRs editing the same lines; depends on that PR moving forward.

Method: semi-formal (the finder's label): bounded-exhaustive translation validation of the real main jar, with the oracle "TOP_SCORES top-k equals COMPLETE top-k" (the rule of CheckHits.checkTopScores), not a proof.

• The harness ran the real query classes over an exhaustive tiny corpus and seeded random corpora under 17 similarities.
• It flagged BayesianScoreQuery with alpha 0.5, beta 30 and base rate 1e-6 on the exhaustive corpus and on two 3,000-document corpora, for example under BM25 top-1 doc 4 at 4.06e-13 instead of doc 2354 at 4.25e-13.
• On the 3,000-document corpora (23,676 queries over the 26 text query classes) it was the only class with differences not explained by IndriDirichletSimilarity's negative scores.
• The finder then found the clamp by reading the code and wrote a deterministic public-API JUnit test on the stock jar.