Relevance-sorted searches never set the minimum competitive score at the 1,000-hit threshold, so they can score every match

IndexSearcher#search(query, n) counts hits exactly up to 1,000 (or up to n when n is larger) and then passes the n-th best score to the scorer through Scorable#setMinCompetitiveScore, so the scorer can skip documents that cannot make the top n. A search sorted by relevance through TopFieldCollector (Sort.RELEVANCE, or Sort(SortField.FIELD_SCORE, tiebreak), which deep paging uses for an order that stays stable across index changes) is meant to do the same: LUCENE-8507 added it in 8.0.0, and 8.0.0, 8.5.2, 8.6.0 and 8.6.3 do it. Since 8.7.0 it no longer happens on the hit that crosses the threshold. TopFieldCollector.countHit (TopFieldCollector.java:77-90 on main a9779ef3c193) first sets totalHitsRelation to GREATER_THAN_OR_EQUAL_TO, and the updateMinCompetitiveScore call in thresholdCheck that should follow is guarded by totalHitsRelation == EQUAL_TO (:104-107), so it can never take effect; the searchAfter branch (:276-282) is dead the same way. Skipping starts only when a later hit beats the queue bottom, and later segments start with no minimum either. With the best documents first in doc-id order, or n hits tied at the top score already in the queue at the crossing, the search scores every match: 10,011 hits instead of 1,024 in the repro, 183,669 instead of 1,060 on a 200k-document index sorted by a static rank. Results, their order and the TotalHits lower bound stay correct; no wall-clock time was measured. Introduced by LUCENE-9449 (99220677fe4, apache/lucene-solr#1725). The fix calls updateMinCompetitiveScore right after the flip and removes the two dead branches; it does not help when every score ties, an older gap.

• Total hits threshold: IndexSearcher counts hits exactly up to TOTAL_HITS_THRESHOLD = 1000 (IndexSearcher.java:100), or up to n when n is larger (both collectors use Math.max(1000, n): TopScoreDocCollectorManager.java:69, TopFieldCollector.java:337); past it the count may become a lower bound (TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO, printed as 1024+ hits).
• Minimum competitive score: once the threshold is passed and the top-n queue is full, the collector calls Scorable#setMinCompetitiveScore(bottom score); scorers in ScoreMode.TOP_SCORES may then skip documents that score lower (ScoreMode.java:28-32).
• Relevance-first sort: a Sort whose first field is descending relevance: Sort.RELEVANCE, new Sort(), or Sort(SortField.FIELD_SCORE, tiebreak). TopFieldCollector then uses ScoreMode.TOP_SCORES and may set the minimum score (canSetMinScore, TopFieldCollector.java:342-347). IndexSearcher#search(Query, int, Sort) and searchAfter(..., Sort) build it with the 1,000 threshold (IndexSearcher.java:677-679, :719-735).

1. Every hit runs countHit(), then thresholdCheck(doc) (true for a non-competitive hit) or collectCompetitiveHit(doc) (TopFieldCollector.java:203-213).
2. On hit 1,001, countHit sets totalHitsRelation = GREATER_THAN_OR_EQUAL_TO for every non-exhaustive score mode, TOP_SCORES included (:83-89).
3. thresholdCheck then reaches } else if (totalHitsRelation == TotalHits.Relation.EQUAL_TO) { // we can start setting the min competitive score if the threshold is reached for the first time here. updateMinCompetitiveScore(scorer); } (:104-107). The relation is no longer EQUAL_TO; and before the threshold updateMinCompetitiveScore does nothing, because it needs totalHits > totalHitsThreshold (:381). So this call can never have an effect. The "Already collected on a previous page" branch of PagingFieldCollector has the same guard (:276-282).
4. Per hit, only collectCompetitiveHit and collectAnyHit still call updateMinCompetitiveScore (:114-132): skipping waits for a hit that strictly beats the queue bottom. (setScorer also calls it, :139, but only without a shared accumulator, and TopFieldCollectorManager.java:90 creates one for every finite threshold.)
5. Later segments read only the shared MaxScoreAccumulator in setScorer (:135-143; TopFieldCollectorManager.java:90 creates it for every finite threshold). It is filled only by a successful updateMinCompetitiveScore (:388-390), so they start with no minimum too.
6. TopScoreDocCollector makes the call on that same hit: if (hitCountSoFar == totalHitsThreshold + 1) { ... updateMinCompetitiveScore(scorer); } (TopScoreDocCollector.java:116-120).

10,011 documents in one segment; docs 0-1,010 "a b c" score 0.8867903, docs 1,011-10,010 "a b" score 4.6311867E-5; a OR b OR c, top 10. TopScoreDocCollector calls setMinCompetitiveScore(0.88679034) at hit 1,001 (doc 1,000) and collects 1,024 hits. The relevance-sorted TopFieldCollector never calls it and collects all 10,011. Same top 10 (docs 0-9). With the fix it calls setMinCompetitiveScore(0.8867903) at hit 1,001 and collects 1,024. Sort(FIELD_SCORE, id) and searchAfter page 2 behave the same, with one segment or six.

• Who is exposed. Default settings and stable entry points: IndexSearcher#search(q, n, Sort.RELEVANCE) (or new Sort()), search(q, n, sort, doDocScores), searchAfter(after, q, n, sort) with Sort(FIELD_SCORE, tiebreak), and any TopFieldCollectorManager whose first sort field is descending relevance. A tie-break field after the score keeps a deep-paging order stable across index changes (doc ids break ties only within one reader), and there is no TopScoreDocCollector equivalent for it.
• Not exposed. search(q, n) / TopScoreDocCollector; sorts whose first field is not descending relevance; exhaustive counts (totalHitsThreshold = Integer.MAX_VALUE).
• When the loss is large. When no hit after the threshold strictly beats the n-th best. That happens when the best documents come first in doc-id order (for example an index sorted by a static rank with IndexWriterConfig#setIndexSort) or when the top scores tie and the queue already holds n of them at the crossing (short fields, single terms: an equal-score later hit loses the tie on doc id).

• Effect. Extra scoring work only. Top hits, their order and the TotalHits lower bound stay correct, and the collector keeps only its fixed-size top-n queue (memory was not measured). The worst case is scoring and comparing every match, more than an exhaustive count of the same query, which need not score. All numbers are hits collected; no wall-clock time was measured.
• How likely. Plausible: every relevance-first sorted search past the threshold runs the dead code, but the big loss needs one of the two data shapes above. No earlier report was found in 236 recorded searches, although the code has been there since 8.7.0 (2020).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected min competitive score >= 2, got 0.0
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.search.TestRelevanceSortMinCompetitiveScore.testMinCompetitiveScoreSetOnceThresholdIsExceeded(TestRelevanceSortMinCompetitiveScore.java:63)

• Documented intent. lucene/CHANGES.txt:6850-6851 (LUCENE-8507, 8.0.0): "TopFieldCollector can now update the minimum competitive score if the primary sort is by relevancy and the total hit count is not required." Its commit 947f82679af put the call on the non-competitive branch, already behind today's totalHitsRelation == EQUAL_TO guard ("we just reached totalHitsThreshold, we can start setting the min competitive score now"; 8.0.0 TopFieldCollector.java:143-146). The guard then meant "minimum score not set yet": in 8.0.0 only updateMinCompetitiveScore (:331) and the index-sort early termination (:138) set GREATER_THAN_OR_EQUAL_TO. LUCENE-8992 (8.4.0, branch_8x commit 1c23a3c14e7; the master commit 93d135e79d7 first shipped in 9.0.0) gave the two PagingFieldCollector branches the same guard. A related promise, from an entry about field comparators that skip documents: CHANGES.txt:4864-4869 (LUCENE-9280), a finite totalHitsThreshold "instructs Lucene to skip non-competitive documents whenever possible".
• The code says so. The comment on the dead branch: "we can start setting the min competitive score if the threshold is reached for the first time here" (TopFieldCollector.java:105-106).
• The sibling does it. TopScoreDocCollector.java:116-120 calls updateMinCompetitiveScore on the non-competitive hit with hitCountSoFar == totalHitsThreshold + 1.
• It stopped, by every sign as a side effect. 8.0.0, 8.5.2, 8.6.0 and 8.6.3 skip; 8.7.0 and later do not. The 8.7.0 source with only the new flip suppressed for relevance-first collectors skips again. LUCENE-9449 changed the flip from filteringLeafComparator != null && ... (never true for a relevance comparator) to scoreMode.isExhaustive() == false && ..., after a review suggestion on #1725 to set GREATER_THAN_OR_EQUAL_TO "every time a TOP_SCORES collection reaches the total hits threshold"; no comment on #1725, #1856 or #10489 mentions relevance sorts. The relation change itself was deliberate; nothing in the record shows that losing the minimum-score start was.
• Tests. No upstream test asserts the current behaviour. TestTopFieldCollector#testSetMinCompetitiveScore crosses the threshold on a competitive hit; its comment "Make sure we do not call setMinCompetitiveScore for non-competitive hits" (:303) comes from the LUCENE-8507 commit and checks that the setter is not called again. It passes with the fix.

The strongest case for "works as designed": ScoreMode.TOP_SCORES only "optionally" allows skipping (ScoreMode.java:28-32); TopFieldCollector is @lucene.experimental (TopFieldCollector.java:35); Sort.RELEVANCE "returns the same results as calling IndexSearcher#search() without a sort criteria, only with slightly more overhead" (Sort.java:36-41); and the 8.7.0 CHANGES entry for LUCENE-9449 says "TopFieldCollector sets TotalHits.relation to GREATER_THAN_OR_EQUAL_TO, as soon as the requested total hits threshold is reached, even though in some cases no skipping optimization is applied and all hits are collected" (8.7.0 release lucene/CHANGES.txt:19-23; not in main's CHANGES).

Why it fails: the skipping is a documented feature built for exactly this path and lost as a side effect of LUCENE-9449; the entry point IndexSearcher#search(Query, int, Sort) is stable API; scoring every match (4x-170x the hits of search(q, n) in the probes where it happens) is not "slight", and Sort(FIELD_SCORE, tiebreak) has no alternative. The 8.7.0 note explains why the reported relation can be a lower bound when every hit was collected; it does not announce that relevance-first sorts stop setting the minimum score, and the code comment still says they set it. Verdict: not intended, as far as the record shows (the relation change was deliberate; losing the minimum-score start at the threshold appears to be its side effect); a performance regression with correct results.

One fix was built and validated through gradle; the finder's one-line variant was validated with the standalone runner; two alternatives were not built.

In countHit, call updateMinCompetitiveScore(scorer) after totalHitsRelation = GREATER_THAN_OR_EQUAL_TO; remove the else if (totalHitsRelation == EQUAL_TO) branch of thresholdCheck (TopFieldCollector.java:104-107) and the EQUAL_TO block in the PagingFieldCollector "Already collected on a previous page" branch (:278-282). 1 file, +3/-9.

if (scoreMode.isExhaustive() == false
    && totalHitsRelation == TotalHits.Relation.EQUAL_TO
    && totalHits > totalHitsThreshold) {
  // for the first time hitsThreshold is reached, notify comparator about this
  comparator.setHitsThresholdReached();
  totalHitsRelation = TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO;
  // and start setting the min competitive score (only has an effect when the primary sort is
  // by relevance and the queue is full)
  updateMinCompetitiveScore(scorer);
}

Trade-offs

• updateMinCompetitiveScore returns at once unless the sort is relevance-first with a finite threshold, the queue is full and the threshold is passed (TopFieldCollector.java:381), so field sorts are untouched; the flip runs at most once per collector, so this is at most one extra call per collector (none for a collector whose relation the shared accumulator flipped first). The call passes the queue bottom from before the current hit is compared; that loses nothing, because a competitive crossing hit calls updateMinCompetitiveScore again with the new bottom (:119). The removed blocks can never have an effect in any score mode.
• For searchAfter, if the page's queue is not full at the crossing, the existing call in collectAnyHit (:128-131) sets the score when it fills (but see the deep-page limit below). The score is also accumulated, so later segments start with it.
• It passes the bottom score itself, as the competitive-hit path does, not Math.nextUp as TopScoreDocCollector does. That is needed when a tie-break follows the score; with Sort.RELEVANCE alone and tied top scores it still scores more than search(q, n) (1,187 vs 1,001 hits in one probe). That gap predates this bug (the 8.6.3 source also passes the bottom score). When every score is equal, A does not help at all (20,000 docs with body "a", TermQuery body:a): search(q, 10) collects 1,001, Sort.RELEVANCE and Sort(FIELD_SCORE, id) 20,000 with or without A (fact-check probe).
• In a concurrent search, a collector whose relation the shared accumulator already flipped skips countHit's block; its own bottom score then waits for its next competitive hit, and the shared score it has can be low. In a fact-check probe (two segments, one per slice, 40 searches each way) stock Sort.RELEVANCE collected 10,011 every time; with A 1,024-4,096 in 39 of 40 (3,072 in 28) and 10,011 once, when that edge hit; search(q, 10) collected 1,024-2,184. The counts depend on thread timing.
• A deep page whose queue fills with tied low scores is not helped either: on the repro's index, the page after hit 1,005 collects 10,011 with or without A, against 1,051 for searchAfter(q, 10). The page queue fills at hit 1,015 with the low score that every later doc ties: the same nextUp gap.
• No wall-clock benchmark was run. Open PR #16857 rewrites the same leaf collector.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle: the new test fails 2 of 2 on the stock tree and passes 2 of 2 with the fix, with assertions on and off.
• With the fix: 11 sort and collector suites (TestTopFieldCollector, TestTopFieldCollectorEarlyTermination, TestSortOptimization, TestSearchAfter, TestTopDocsCollector, TestSortRandom, TestTopDocsMerge, TestSort, TestElevationComparator, TestSortRescorer, the new test), 108 tests, 0 failures, assertions off; the whole org.apache.lucene.search package, 215 suites, 2,333 tests, 0 failures, assertions on; :lucene:core:check -x test passes.
• Standalone runner: the new test passes 20 of 20 over 10 seeds; the plain repro counts 1,024 for both searches.

The finder's +3-line diff: the same added call, nothing removed.

Trade-offs

Same behaviour, and the smallest patch to review; leaves two branches that can never fire and a comment that is no longer true.

What the checks validated

With the standalone runner: the finder's repro 1,024 vs 1,024 and 103 tests in 9 sort and collector classes pass; the independent reproducer's collector and searcher tests pass, and the devil's advocate's static-rank probe gives the score-sorted counts.

Skip the countHit flip when canSetMinScore is true, so the EQUAL_TO-guarded call fires again.

Trade-offs

Restores the skipping on 8.7.0 (executed as a shadow of the 8.7.0 source), but reverts the relation change LUCENE-9449 made on purpose and touches the documented 8.7.0 relation behaviour. More change for the same result.

Replace both totalHitsRelation == EQUAL_TO guards with totalHits == totalHitsThreshold + 1.

Trade-offs

Works for the non-competitive crossing hit and also covers the concurrent edge above (seen once in 40 concurrent searches with A), but must be repeated at each branch. Not built.

Method: non-formal (code reading and testing, hunt unit on sorted search, score modes and threshold invariance, 2026-10-27, main a9779ef3c193).

• The unit's primary method was a metamorphic threshold-invariance check on the real jars (about 305M checks): a sorted search with a small totalHitsThreshold must return the same hits and sort values as the exhaustive one. It compares results only, so it cannot see a performance-only defect.
• While debugging the unit's other candidate (in DrillSidewaysQuery), the finder saw TopFieldCollectorManager(Sort.RELEVANCE, 10, 10) count all 100 matches where TopScoreDocCollectorManager(10, 10) stopped at 11.
• Reading countHit and thresholdCheck and git log led to LUCENE-9449; a public-API main() repro failed on main, 10.5.2, 9.12.3 and 8.7.0 and passed on 8.6.3 and with a one-line shadowed fix.