DoubleValuesSourceRescorer and LateInteractionRescorer throw NoSuchElementException instead of returning no hits when the reader has no segments

DoubleValuesSourceRescorer.rescore looks up the reader's first segment (leaves().getFirst()) before it looks at the first-pass hits. On a reader with no segments (a brand-new index, an NRT reader before the first document, an index after deleteAll() or after every document was deleted and merged away, an empty MultiReader) that list is empty, and the JDK throws java.util.NoSuchElementException with no message instead of returning the 0 hits the first pass already produced. LateInteractionRescorer inherits the method. QueryRescorer and SortRescorer return 0 hits for the same input. It never returns wrong hits, and a reader with one segment is never affected; the failure is one unchecked exception, in every release since the classes were added in 10.3.0 and on main.

1. DoubleValuesSourceRescorer.rescore (DoubleValuesSourceRescorer.java:45-86) rewrites the values source (:47), then clones the first-pass hits and sorts them by doc id (:49-50).
2. Before its per-hit loop it runs List<LeafReaderContext> leaves = searcher.getIndexReader().leaves(); (:52), LeafReaderContext ctx = leaves.getFirst(); (:53) and int leafEnd = ctx.docBase + ctx.reader().maxDoc(); (:55). Nothing checks whether there are hits or segments.
3. With no segments, leaves is empty and the JDK's List.getFirst() throws NoSuchElementException with no message. It is not a Lucene assertion, so -ea and -da behave the same.
4. The per-hit loop (:58-77) would not have run at all for 0 hits. The method's deliberate check, an IllegalStateException for hit doc ids past the last segment (:60-66), is for hits from another searcher.
5. LateInteractionRescorer extends DoubleValuesSourceRescorer (LateInteractionRescorer.java:32) and does not override rescore(), so it fails at the same line.
6. QueryRescorer (QueryRescorer.java:55, :67, :73) and SortRescorer (SortRescorer.java:46, :60, :66) get the same leaves list but index it only inside while (hitUpto < hits.length), so with 0 hits they never touch a segment and return 0 hits.

• The minimal repro: new IndexSearcher(new MultiReader()), a MatchAllDocsQuery first pass with 0 hits, rescore(searcher, firstPass, 10): NoSuchElementException at DoubleValuesSourceRescorer.java:53, with -ea and -da, on main and on the 10.6.0 jars.
• Six ordinary ways to a reader with no segments, all failing at the same line on main, 10.5.2 and 10.3.0: a committed empty index; an NRT reader before the first document; deleteAll(), then an NRT reader; every document deleted, then forceMerge(1) (the fully deleted segment is dropped); a KnnFloatVectorQuery first pass on an empty index followed by LateInteractionRescorer; an empty MultiReader.
• Not affected: a 1-segment index whose first pass has 0 hits (0 hits back) or 1 hit (rescored normally); an empty sub-reader inside a non-empty MultiReader.
• Lucene's own test reached it: the original TestDoubleValuesSourceRescorer.testBasic indexed nextInt(100) documents, and 6 of 1000 iterations on the 10.3.0 jars threw this exception. #14793 later raised the count to nextInt(50, 100) to fix a different failure, which also removed the 0-document draw.

• Who is hit. Callers of DoubleValuesSourceRescorer subclasses or LateInteractionRescorer, the multi-vector reranker whose javadoc describes running a cheap first pass and then reranking its top hits (LateInteractionRescorer.java:25-28), on any searcher whose whole reader has no segments: a freshly created index, an NRT reader before the first document arrives, after deleteAll(), after every document was deleted and merged away, an empty MultiReader, and by the same token a shard whose index is still empty (reasoned; not run as a shard setup).
• Default vs opt-in. Default IndexWriterConfig, codec and IndexSearcher; the trigger is the opt-in rescorer API. DoubleValuesSourceRescorer itself has no @lucene.experimental tag; its base class Rescorer and its subclass LateInteractionRescorer do.
• Scale. One rescore call fails, immediately, with an unchecked exception that has no message. No wrong hits are returned and Lucene writes nothing; for an empty first pass produced by that same searcher, the correct answer was empty. With one segment the same call works.
• Likelihood. Deterministic whenever an application that always rescores meets an empty reader. Lucene's own randomized test could reach it: the original testBasic indexed nextInt(100) documents, and in a 1000-iteration replay on the 10.3.0 jars (by this hunt, not an upstream CI record) 6 iterations threw this exception; #14793 later changed the document count. Solr, the one downstream project found that uses DoubleValuesSourceRescorer, skips rescoring when the first pass is empty, so it is not affected; code search also found one small application using LateInteractionRescorer (not examined), and other downstream use is unknown.
• Workaround. Skip rescore() when the first pass returned no hits.

The test searches new IndexSearcher(new MultiReader()) with MatchAllDocsQuery.INSTANCE (0 hits) and rescores the result with an anonymous DoubleValuesSourceRescorer over DoubleValuesSource.constant(1), topN 10. It turns the NoSuchElementException into an AssertionError so the failure states expected vs actual; the cause keeps the getFirst() frame. LateInteractionRescorer inherits the same rescore() and fails the same way. The method could equally go into the existing TestDoubleValuesSourceRescorer. On 10.3.x replace MatchAllDocsQuery.INSTANCE with new MatchAllDocsQuery() (the constant first appears in 10.4.0).

TestDoubleValuesSourceRescorerNoSegments > testRescoreOnReaderWithoutSegments FAILED
    java.lang.AssertionError: expected 0 rescored hits (as QueryRescorer returns), got java.util.NoSuchElementException
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:14C1189231F4CECF]:0)
        at org.apache.lucene.search.TestDoubleValuesSourceRescorerNoSegments.testRescoreOnReaderWithoutSegments(TestDoubleValuesSourceRescorerNoSegments.java:23)

        Caused by:
        java.util.NoSuchElementException
            at java.base/java.util.List.getFirst(List.java:823)
            at org.apache.lucene.search.DoubleValuesSourceRescorer.rescore(DoubleValuesSourceRescorer.java:53)
            at org.apache.lucene.search.TestDoubleValuesSourceRescorerNoSegments.testRescoreOnReaderWithoutSegments(TestDoubleValuesSourceRescorerNoSegments.java:21)

1 test completed, 1 failed

# the same with -Ptests.asserts=false and on the 10.6.0 release jars; with the recommended fix the test passes

• The contract. Rescorer.java:37-39: "@param searcher {@link IndexSearcher} used to produce the first pass topDocs" and "@param firstPassTopDocs Hits from the first pass search. It's very important that these hits were produced by the provided searcher; otherwise the doc IDs will not match!" No javadoc on Rescorer, DoubleValuesSourceRescorer or LateInteractionRescorer asks for a non-empty reader or non-empty hits.
• The siblings. QueryRescorer and SortRescorer return 0 hits for an empty committed index on 9.12.3, 10.2.2, 10.3.0, 10.5.2 and main, and for new MultiReader() on main, 10.5.2 and 10.3.0.
• Zero-segment readers are a supported searcher input. TestIndexSearcher.testGetSlicesNoLeavesNoExecutor and testGetSlicesNoLeavesWithExecutor build an IndexSearcher over new MultiReader() and check that it has 0 slices (TestIndexSearcher.java:224-233), and the test framework's QueryUtils.wrapUnderlyingReader deliberately mixes empty MultiReaders into the searchers it checks queries against (QueryUtils.java:165-180).
• Lucene guards this case elsewhere before taking a first segment: MultiDocValues.getNumericValues returns null for 0 leaves (MultiDocValues.java:183-184), JoinUtil returns MatchNoDocsQuery("JoinUtil.createJoinQuery with no segments") (JoinUtil.java:533-534), StringValueFacetCounts checks leafCount == 0 (StringValueFacetCounts.java:265). A verifier's grep for direct first-segment calls (leaves().get(0), getFirst()) found DoubleValuesSourceRescorer.java:53 to be the only unguarded one in production code outside benchmarks; first-segment access written in other forms was not searched.
• Precedent. lucene/CHANGES.txt:338 (11.0.0, Bug Fixes): "GITHUB#16926: RescoreTopNQuery no longer throws when its inner query matches no documents. (Greg Miller)". For the same API family, "LUCENE-5682: NPE in QueryRescorer when Scorer is null" (CHANGES.txt:12275) was a bug fix too.
• History. No review comment on #14776 or #14729 discusses empty readers or empty hits, and every upstream test of the two classes indexes documents first. The original testBasic could index 0 documents and hit this exception; #14793 (ChrisHegarty) raised the count to fix a different failure, the test's own ArrayIndexOutOfBoundsException at scoreDocs[0], and does not mention this one.

The strongest case for "works as designed": rescoring an empty result is pointless, so callers should skip it; Rescorer and LateInteractionRescorer are @lucene.experimental; the method validates its input on purpose (the IllegalStateException at :60-66), and in the #14776 review the author wanted a mismatched searcher to surface as an error, there in explain(); and no test ever specified the empty case.

Why it fails: the documented precondition is met and nothing asks callers to skip empty results; @lucene.experimental is about API stability, not about accepting crashes, and DoubleValuesSourceRescorer itself is not tagged; the deliberate check targets foreign hits, while this throw is an unguarded JDK call with no message, made before any hit is looked at; the sibling rescorers return 0 hits; and the maintainers have just fixed an analogous empty-input crash in RescoreTopNQuery as a bug. Verdict: not intended; a small robustness bug.

Option A (the finder's fix) was applied in two scratch worktrees of 645d4fe14a20 and run with gradle (the worktrees were removed after the runs); options A and C were also shadowed into the stock main jars, and the independent fact check ran C through gradle. Option B was not built. All fix checks are targeted (the repro plus the rescorer suites); neither fix was run against the whole core test suite or on a 10.x branch, and C went through gradle with assertions on only. Until a fix ships, callers can skip the rescore when the first pass is empty.

+3 lines in DoubleValuesSourceRescorer.rescore, after the clone and sort (:49-50) and before the segment lookup (:52-53). It could also sit before the values-source rewrite at :47, to skip that work, but then a custom source's rewrite (and any check or side effect in it) would no longer run on an empty first pass.

if (hits.length == 0) {
  return new TopDocs(firstPassTopDocs.totalHits, hits);
}
List<LeafReaderContext> leaves = searcher.getIndexReader().leaves();
LeafReaderContext ctx = leaves.getFirst();

Trade-offs

• Smallest change; returns what QueryRescorer and SortRescorer return and keeps the first pass's totalHits, as the normal path does (:85). Fixes LateInteractionRescorer, which inherits the method.
• Foreign hits on an empty reader (documented misuse) still throw NoSuchElementException instead of the method's IllegalStateException; that case is not part of this bug.

What the checks validated

• Gradle, stock 645d4fe14a20, seed DEADBEEF: the repro fails with assertions on and off (1 test, 1 failure each).
• Gradle with the fix, assertions on: the repro, TestDoubleValuesSourceRescorer (3), TestLateInteractionRescorer (2), TestQueryRescorer (8) and TestSortRescorer (3), 17 tests, 0 failures; :lucene:core:checkGoogleJavaFormat passes. Assertions off: the same 17 tests, 0 failures. An earlier worktree gave the same 17 of 17 with assertions on.
• Shadowed into the stock main jars with assertions off: the repro and the two upstream rescorer suites, 6 tests, 0 failures (the stock control fails 1 of 6).
• Earlier hunt checks: the finder's fix check 3/3 plus the upstream rescorer suites 5/5; a verifier's equivalent fix turned its 7 tests green.

3 lines changed: start with no segment (ctx = null, currLeaf = -1, leafEnd = 0) and let the existing while (hit.doc >= leafEnd) loop advance to segment 0 on the first hit.

List<LeafReaderContext> leaves = searcher.getIndexReader().leaves();
LeafReaderContext ctx = null; // was: leaves.getFirst()
int currLeaf = -1;            // was: 0
int leafEnd = 0;              // was: ctx.docBase + ctx.reader().maxDoc()

Trade-offs

• With no hits nothing is looked up. With hits on an empty reader, currLeaf == leaves.size() - 1 holds (both -1), so the method's own IllegalStateException fires instead of NoSuchElementException.
• Changes the loop's starting state, so a reviewer must re-check the segment walk; its cost was not benchmarked.

What the checks validated

Shadowed into the stock main jars: the repro and the two upstream rescorer suites, 6 tests, 0 failures, with assertions on and off. The independent fact check also ran it through gradle on 645d4fe14a20, with assertions on: the repro and the four rescorer suites, 17 tests, 0 failures, and the format check passes.

Return the empty result when there are no hits, else throw the IllegalStateException.

Trade-offs

Same behaviour as C, with more lines.

Method: non-formal (a code-reading lead, then executed checks; the finder tagged it semi-formal; this page classes it by the first signal).

• The hunt's plan for the rescoring unit listed DoubleValuesSourceRescorer.java:53 as a lead: "leaf navigation starts at the first leaf before any hit is examined".
• An AI finder then ran a bounded-exhaustive translation-validation harness on the stock jar: QueryRescorer and DoubleValuesSourceRescorer over every index of up to 5 documents (including an empty MultiReader and an empty DirectoryReader), every subset of first-pass hits in random order, first-pass scores {0, 1}, values {missing, 0, 1} and topN from 0 to the hit count + 1: 1,801,236 checks per rescorer against exact rescoring results. QueryRescorer had no mismatch; DoubleValuesSourceRescorer had one mismatch class, this exception on readers with no segments (4 cases).
• The finder turned it into a public-API JUnit; independent verifiers found five more ordinary ways to reach a reader with no segments, and the case was later reduced to the one-method test. So it is a code-reading find confirmed by execution, not a blind discovery by a tool.