SortRescorer can give kept hits other hits' scores when topN is smaller than the number of hits

SortRescorer.rescore(searcher, firstPassTopDocs, topN) re-sorts first-pass hits by a Sort and keeps the best topN; the search package docs give exactly this use: "the best 1,000 hits by BM25 score may be sorted by descending popularity in order to compute the final top-100 hits". Since 8.0.0 the method puts the first-pass scores back on the kept hits by position: it sorts the kept hits by doc id and gives the i-th of them the score of the i-th first-pass hit by doc id. That is sure to pair each hit with itself only when no hit was dropped, which is what the assert just before the copy assumes; otherwise a kept hit keeps its own score only if no hit with a smaller doc id was dropped. With topN smaller than the number of hits, the documents, their order and their FieldDoc sort values are right, but kept hits typically carry other hits' scores (how many depends on the sort; none with a FIELD_DOC sort): keeping 3 of 6 hits, docs 3, 5, 1 come back with the scores of docs 1, 2, 0, and in a probe of the documented 1,000 -> 100 pipeline 99 of 100 kept hits had a wrong score. With assertions enabled every call with topN < hits throws AssertionError at SortRescorer.java:86. Expression#getRescorer returns an ExpressionRescorer, which extends SortRescorer without overriding rescore, and behaves the same. 4.8.0 to 7.7.3 returned each hit's own score; the copy and the assert came with LUCENE-8412 (55bfadbce115, released in 8.0.0). The fix pairs the scores by doc id instead of by position.

• Rescorer: takes the top hits of a cheap first search and re-orders them with something more expensive. Rescorer.rescore(searcher, firstPassTopDocs, topN) returns "How many re-scored hits to return" (Rescorer.java:40).
• SortRescorer(sort): a rescorer that re-sorts the first-pass hits by a Sort (a field such as popularity, the score itself, or an expression). Expression#getRescorer(bindings) returns an ExpressionRescorer, a SortRescorer subclass that only adds a more detailed explain.
• Doc id: a document's number in the index. The first-pass hits come best first; SortRescorer works on them in doc-id order.

1. rescore clones the first-pass hits and sorts the clone by doc id (SortRescorer.java:41-44).
2. It creates a field-sorted collector that keeps the best min(topN, hits) entries (TopFieldCollectorManager(sort, topN, null, Integer.MAX_VALUE), :48-49) and feeds it every hit, in doc-id order, with that hit's own score (:57-82, the score at :77). So the kept set, its order and the sort values (also FIELD_SCORE and expressions over _score) are right.
3. The collector's topDocs() (:84) carries no scores (NaN): a field-sorted collector does not track them since LUCENE-8412 (executed: [doc 3 fields[0]=60 score=NaN] ...).
4. To put scores back (:85-91, comment "set scores from the original score docs") it sorts the kept hits by doc id and runs rescoredDocsClone[i].score = hits[i].score (:90), where hits holds all first-pass hits by doc id. The k-th smallest kept doc id gets the score of the k-th smallest first-pass doc id: the same document exactly when no hit with a smaller doc id than that kept hit was dropped, which topN >= hits guarantees. The assert just before, hits.length == rescoredDocs.scoreDocs.length (:86), states that assumption.
5. With -ea the assert throws on every call with topN < hits. With -da (production) the copy runs silently. The clone shares its ScoreDoc objects with the returned array, so the returned hits get the wrong values.
6. ExpressionRescorer (ExpressionRescorer.java:37-47) never overrode rescore, so Expression#getRescorer (Expression.java:112-114) has the same defect.
7. Not triggered with topN >= hits. A FIELD_DOC sort keeps the smallest doc ids, so the positions line up by accident under -da (-ea still throws).

Main ea57360912ed, assertions off, one segment, BM25. Docs 0-5: 8 words each with "lucene" 3, 1, 5, 2, 4, 6 times; popularity 5, 40, 10, 60, 20, 50. First pass body:lucene: doc 0 0.0529, doc 1 0.0337, doc 2 0.0598, doc 3 0.0463, doc 4 0.0570, doc 5 0.0618.

With assertions on, every topN < hits row throws AssertionError at SortRescorer.java:86.

• Who is exposed. Callers of new SortRescorer(sort).rescore(searcher, hits, topN) or expression.getRescorer(bindings).rescore(...) with topN < hits.scoreDocs.length: the use the search package docs give (1,000 hits re-sorted to a top 100, package-info.java:371-373) and the typical use the Rescorer javadoc describes ("collecting the top few hundred hits perhaps").
• What goes wrong. Only ScoreDoc.score. Each kept hit gets the first-pass score of another hit from the same result, sometimes of a hit that was not kept. Code that displays scores, applies a minimum score, merges result lists by score or feeds the score into a later pass gets those values. With a FIELD_SCORE sort the same FieldDoc holds its own score in fields[0] and another hit's in .score.
• Not affected. Which documents are returned, their order and their sort values. Calls with topN >= hits.
• How often. With assertions on, every call with topN < hits throws. With assertions off, the number of kept hits with a wrong score ranges from 0 to topN and depends on how the sort reorders doc ids: 3 of 3 in the 6-document example, 99 of 100 in one probe corpus of the documented pipeline, 0 with a FIELD_DOC sort. A randomized check on 30 random indexes confirmed the doc-id-rank pairing in all 2,072 cases it checked; in 1,737 of the 1,832 cases with topN < hits at least one kept hit had a wrong score, and in the other 95 the pairing happened to give every kept hit its own score.
• With assertions enabled (-ea, as in Lucene's own test runs) the call throws AssertionError, so an application test that covers this path with assertions on fails.
• Settings. Opt-in API, nothing else non-default: new IndexWriterConfig() (so BM25) and a plain IndexSearcher; the diagram and security probes also ran Lucene's default codec, the JUnit tests the test framework's randomized default codec, which does not change BM25 scores; 2 hits and topN = 1 suffice. Rescorer and ExpressionRescorer are @lucene.experimental; SortRescorer itself carries no experimental tag.
• Downstream (code search of default branches): Solr and OpenSearch do not reference SortRescorer or ExpressionRescorer; Elasticsearch has no SortRescorer and wraps Expression#getRescorer in its expression module, with only a unit test found calling it. A 2021 java-user thread recommends SortRescorer for re-sorting results. Private code is not visible.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError
       org.apache.lucene.search.SortRescorer.rescore(SortRescorer.java:86)
       org.apache.lucene.search.TestSortRescorerTopNScore.testTopHitKeepsItsOwnScore(TestSortRescorerTopNScore.java:44)

• The contracts. Search package docs (package-info.java:371-373): "SortRescorer, to rescore using a Sort. For instance, the best 1,000 hits by BM25 score may be sorted by descending popularity in order to compute the final top-100 hits." Rescorer (Rescorer.java:21-25, :40): "collecting the top few hundred hits perhaps, and then use this class to mix in a more costly second pass scoring"; "@param topN How many re-scored hits to return". The code's comment (SortRescorer.java:85): "set scores from the original score docs". ScoreDoc.score (ScoreDoc.java:25): "The score of this document for the query." ExpressionRescorer (ExpressionRescorer.java:30-33): "Functionally this is the same as SortRescorer ... except for the explain method".
• History. 4.8.0 (LUCENE-5545) to 7.7.3 created the collector with trackDocScores=true, so every kept hit recorded its own score; all 14 releases 7.0.0-7.7.3 pass (7.4.1 and 7.5.1 have CHANGES sections but were never released). LUCENE-8412 (55bfadbce115, 8.0.0) removed that option and added the comment, the assert and the positional copy. Its CHANGES entry says scores "need to be set on top hits via TopFieldCollector#populateScores instead" and its MIGRATE entry explains an efficiency change; neither says rescored scores became undefined, and the issue discussion (#9459) never mentions SortRescorer or topN.
• Untested case. TestSortRescorer calls rescore with topN 10 for 3 hits (TestSortRescorer.java:96, :134) or topN == hits (testRandom, :179-186); TestExpressionRescorer with topN 10 for 3 hits (:106). None asserts a score after rescoring.
• No 'by design' record. No issue, PR, review comment, list thread or JIRA entry calls the current behaviour intended.

The strongest case for "intended" (built by a devil's-advocate verifier and by an OpenAI GPT-5.6 reviewer): the class javadoc promises only that it "re-sorts according to a provided Sort"; since LUCENE-8412 a field-sorted collector computes no scores, so scores of re-sorted hits are unspecified; the assert at :86 states a precondition, topN >= hits; Rescorer is @lucene.experimental; and for ExpressionRescorer the javadoc says it assigns "second pass scores", so the first-pass score is not the obvious expected value.

Why it fails: the code sets the scores on purpose (the comment at :85) and returns each hit's own score whenever topN >= hits; without the copy the scores would be NaN, and what it returns is neither NaN nor "unspecified" but another document's plausible score. The assert encodes an internal assumption that the documented use contradicts, not a documented precondition: no javadoc, CHANGES or MIGRATE text restricts topN, and the package docs give topN < hits as the example. LUCENE-8412 was an efficiency refactor that announced no SortRescorer change, and 4.8.0-7.7.3 returned own scores. @lucene.experimental allows API changes, not wrong values. For ExpressionRescorer, another hit's first-pass score is wrong under every reading. That few callers read scores after a re-sort argues for low impact, not for intended behaviour. The independent reproducer, the devil's advocate and the adjudication rejected the intended-behaviour argument, and the cross-model review records rate it weak. Verdict: a defect.

The kept hits are a subset of the first-pass hits and both arrays are already sorted by doc id, so a merge walk restores each hit's own score. One option was built and validated in a scratch worktree at ea57360912ed (removed afterwards); the others are not built.

Replace the positional loop at SortRescorer.java:89-91 by a walk over both doc-id-sorted arrays, and change the assert to what is actually true: rescoredDocs.scoreDocs.length == Math.min(topN, hits.length). +9/-4 lines in SortRescorer.java; ExpressionRescorer inherits the fix.

assert rescoredDocs.scoreDocs.length == Math.min(topN, hits.length);
ScoreDoc[] rescoredDocsClone = rescoredDocs.scoreDocs.clone();
Arrays.sort(rescoredDocsClone, docIdComparator);
int h = 0;
for (ScoreDoc rescored : rescoredDocsClone) {
  while (hits[h].doc != rescored.doc) {
    h++;
  }
  rescored.score = hits[h].score;
}

Trade-offs

• Cost. The positional loop becomes a walk of at most hits.length steps (O(hits + topN)), next to the existing O(hits log hits) doc-id sort; no new allocation.
• Safety. Every kept doc is a first-pass hit, so the walk cannot run past the end. This relies on the documented input contract that the first-pass hits come from the same searcher (Rescorer.java:38-39); caller-built TopDocs that break it were not examined.
• Rebase. Open draft #16847 (long doc ids) changes the doc-id types in this file but not these lines.

What the checks validated

• Stock: both new test classes fail through gradle with assertions on and off, and over 5 iterations of a second seed.
• Patched, assertions on and off: every core rescorer suite (TestSortRescorer, TestSortRescorerTopNScores, TestQueryRescorer, TestDoubleValuesSourceRescorer, TestLateInteractionRescorer, TestRescoreTopNQuery, TestRescoreTopNQueryPrefetch, TestRescoreTopNQueryFloat16Prefetch: 27 tests, 0 failures, 2 skipped as @Nightly) and the whole expressions module (14 classes, 0 failures).
• Patched, seed 5EED1234, 10 iterations of TestSortRescorer* (with its random-index test) and TestExpressionRescorer*: 0 failures.
• checkGoogleJavaFormat, ecjLint and forbiddenApis pass on core and expressions (in the logged run Gradle reused an earlier successful run of the core tasks on the same inputs, UP-TO-DATE; the expressions main-source lint was not requested, as no expressions source changed).
• Earlier hunt checks of the same change: the bounded-exhaustive harness at up to 3 documents (203,720 checks, 0 mismatches), a randomized check on 30 random indexes (0 wrong scores) and the diagram's 6-document probe (own scores).

Arrays.binarySearch(hits, rescored, docIdComparator) for each kept hit, or a doc-to-score map built before collection.

Trade-offs

Same result and does not depend on walking in step, but O(N log M) or an extra map; option A is simpler and cheaper.

Call rescore(searcher, hits, hits.scoreDocs.length) and keep the first topN entries; then the positional copy lines up.

Trade-offs

Executed in the verifier's 1,000 -> 100 probe: the same 100 documents in the same order (compared with assertions off), each with its own score. Costs a collector queue the size of the first pass (1,000 instead of 100).

Throw IllegalArgumentException when topN < hits, or document the restriction.

Trade-offs

Contradicts the search package docs' own example and the typical use the Rescorer javadoc describes.

Method: code reading, bounded by bounded-exhaustive translation validation, then a public-API JUnit.

• A hunt unit (2026-10-26) swept the siblings of earlier rescorer findings. Its plan named the lead from code reading: with topN < hits, the copy at SortRescorer.java:90 pairs the i-th smallest doc id of all hits with the i-th smallest doc id of the top N.
• The finder, an AI tool, compared the stock jar with an independent reference (the first min(topN, hits) hits in sort order, each with its own score and sort values) on every index of up to 5 documents in every segment layout, plus zero-leaf and empty readers, every value assignment, every first-pass hit subset, topN 1 to hits + 1 and 11 sorts (including FIELD_SCORE, FIELD_DOC, a values-source sort field and 3 expressions): 41,261,000 checks with assertions off and 32,369,216 with them on, of which 20,630,500 in each mode check scores, the doc set, order and sort values (the rest check totalHits, a separate observation).
• Every wrong score and every AssertionError had topN < hits; there were none with topN >= hits, and 0 doc-set, order or sort-value mismatches. Two injected mutants of SortRescorer (the collector size, the collected score) were detected, so the check is not vacuous.
• A public-API JUnit on the stock jars followed; then the cross-model reviews, verifiers, an adjudication, the release sweep, the prior-report search and the security screen.