SortRescorer and Expression#getRescorer pre-allocate their sort queue for topN instead of for the first-pass hits, so topN = Integer.MAX_VALUE throws and a large topN costs memory in proportion to topN

SortRescorer.rescore(searcher, firstPassTopDocs, topN) re-sorts the hits it is given, so it can never return more than firstPassTopDocs.scoreDocs.length of them. It still builds its TopFieldCollector with numHits = topN (SortRescorer.java:48-49 on main 645d4fe14a20), and that collector pre-allocates arrays of length numHits (TopFieldCollectorManager.java:44-45). So topN = Integer.MAX_VALUE, a common way to say "re-sort all of them" (it works with the sibling rescorers and IndexSearcher), throws on 5 hits: OutOfMemoryError: Requested array size exceeds VM limit on main, IllegalArgumentException: maxSize must be >= 0 and < 2147483631 in 7.5.0 to 10.6.0 (NegativeArraySizeException in 4.9.1 to 7.4.0). A topN that fits the array limit but not the heap throws OutOfMemoryError: Java heap space, and below that every call allocates about 12 bytes per topN slot for one long sort field, whatever the hit count: 120 MB for 5 hits at topN = 10^7. QueryRescorer and DoubleValuesSourceRescorer return min(topN, hits) for any topN >= 0, and IndexSearcher caps numHits at maxDoc. Expression#getRescorer returns a subclass that inherits rescore(). Every release since 4.8.0 (2014) has it. In these topN >= hits.length calls the returned hits, order and sort values are correct whenever the call succeeds (a smaller topN hits the separate LFV-424). Fix (one new line, one changed line): size the collector with Math.min(topN, Math.max(1, hits.length)).

• Rescorer: Rescorer#rescore(IndexSearcher searcher, TopDocs firstPassTopDocs, int topN), documented as "@param topN How many re-scored hits to return" (Rescorer.java:40), with no bound. Rescorer is @lucene.experimental (:30).
• SortRescorer: "A Rescorer that re-sorts according to a provided Sort" (SortRescorer.java:27). Expression#getRescorer(bindings) returns an ExpressionRescorer (Expression.java:112-114), which extends SortRescorer, never overrides rescore() and is documented as "Functionally ... the same as SortRescorer" (ExpressionRescorer.java:30-37).
• Pre-allocation: "The instances returned by this method pre-allocate a full array of length numHits" (TopFieldCollectorManager.java:44-45). For a long sort field that is new long[numHits] (LongComparator.java:38), plus the queue's new Object[maxSize + 1] (PriorityQueue.java:129-134).

1. rescore clones the first-pass hits and sorts them by doc id (SortRescorer.java:42-44), so hits.length is known.
2. It then builds new TopFieldCollectorManager(sort, topN, null, Integer.MAX_VALUE).newCollector() (SortRescorer.java:48-49): the size is the caller's topN.
3. newCollector() calls FieldValueHitQueue.create(sort.getSort(), numHits) (TopFieldCollectorManager.java:112-114), which builds one comparator of that size per sort field (FieldValueHitQueue.java:139-151) and then the queue (:153), whose guard rejects maxSize >= ArrayUtil.MAX_ARRAY_LENGTH with IllegalArgumentException (PriorityQueue.java:121-124, "Throw exception to prevent confusing OOME").
4. The loop then feeds only the hits.length hits (SortRescorer.java:60-82); :86 even asserts hits.length == rescoredDocs.scoreDocs.length. Every slot beyond hits.length is never used.
5. On main, #14873 (unreleased) builds the comparators before the queue's guard runs, so new long[Integer.MAX_VALUE] fails first with OutOfMemoryError; that ordering is the separate bug LFV-993 (different fix site). Releases reach the guard and throw IllegalArgumentException.

5 hits, prices 10, 9, 8, 7, 6, one long sort field, main 645d4fe14a20, -Xmx1g. Bytes allocated by one SortRescorer.rescore call: 1,840 at topN = 5; 1,904 at 10; 13,784 at 1,000; 12,001,784 at 10^6; 120,001,784 at 10^7: about 12 bytes per topN slot (in this probe exactly 1,784 + 12 x topN for even topN, 4 bytes less for odd topN such as 5, from the 8-byte alignment of the queue's Object[topN + 1]; the constant depends on the index, and the figures are for one JVM with compressed references). topN = 2,147,483,630 throws OutOfMemoryError: Java heap space; Integer.MAX_VALUE throws OutOfMemoryError: Requested array size exceeds VM limit. With the fix: 1,840 bytes at every topN, the 5 hits always returned. QueryRescorer (12,856 bytes) and DoubleValuesSourceRescorer (3,744 bytes) allocate the same at topN = 10^7 and at Integer.MAX_VALUE and return the 5 hits, as does IndexSearcher.search(q, Integer.MAX_VALUE, sort).

• Needs: SortRescorer or Expression#getRescorer with a topN far above the number of first-pass hits. Any Sort, any index size, no setting.
• Not affected: QueryRescorer, DoubleValuesSourceRescorer and IndexSearcher.search(q, n, sort) (which caps n at maxDoc, IndexSearcher.java:721, :729).
• What stays right: in these topN >= hits.length calls, whenever the call succeeds, the hits, their order and their sort values are correct.
• Not claimed: topN <= 0 (at topN = 0 SortRescorer throws "numHits must be > 0", like IndexSearcher.search(q, 0); a negative topN throws IllegalArgumentException in all three rescorers).
• Same method, different bug: with topN below the hit count, returned hits get another hit's score (LFV-424).

• Who is hit. Code that re-sorts collected hits with SortRescorer or an expression and asks for all of them. That use is real: on java-user (2021-11-30, "How to change sorting after getting search results") Michael Sokolov pointed a user who collects all hits and re-sorts them to the Rescorers, and the user picked SortRescorer (credit: Michael Sokolov).
• Default or opt-in. No setting is needed; any Sort, any index size. The API itself is opt-in and @lucene.experimental.
• Effect. At Integer.MAX_VALUE: OutOfMemoryError on main (an Error, which code catching RuntimeException for bad arguments does not see, and which makes a JVM run with -XX:+ExitOnOutOfMemoryError exit), IllegalArgumentException naming an internal maxSize in 7.5.0 to 10.6.0, NegativeArraySizeException in 4.9.1 to 7.4.0. A topN that fits the array limit: OutOfMemoryError: Java heap space. Below that: no error, about 12 bytes per topN slot on every call. The heap OutOfMemoryError depends on the heap (runs used -Xmx1g).
• Gradle detail. Lucene's test task runs with -XX:+HeapDumpOnOutOfMemoryError, so on main the repro also writes a heap dump of about 50 MB into lucene/core/build/tests-cwd/.
• How likely: rare. The package docs describe topN below the hit count ("the best 1,000 hits by BM25 score may be sorted by descending popularity in order to compute the final top-100 hits"); the failure is loud at the extremes, so a developer meets it in testing; no matching public report was found.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected 2 re-sorted hits, but rescore threw java.lang.OutOfMemoryError: Requested array size exceeds VM limit
       org.apache.lucene.search.TestSortRescorerMaxTopN.testTopNIntegerMaxValue(TestSortRescorerMaxTopN.java:46)
     Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
       org.apache.lucene.search.comparators.LongComparator.<init>(LongComparator.java:38)
       org.apache.lucene.search.SortField.getComparator(SortField.java:503)
       org.apache.lucene.search.FieldValueHitQueue.create(FieldValueHitQueue.java:146)
       org.apache.lucene.search.TopFieldCollectorManager.newCollector(TopFieldCollectorManager.java:114)
       org.apache.lucene.search.SortRescorer.rescore(SortRescorer.java:49)
       [...]

• Contract. Rescorer.java:40: "@param topN How many re-scored hits to return", no bound and no cost note. The pre-allocation NOTE sits on TopFieldCollectorManager (:44-45), an implementation detail the caller of SortRescorer never sees.
• Siblings. QueryRescorer.java:120-123 and DoubleValuesSourceRescorer.java:79-82: if (topN < hits.length) select topN, otherwise return every hit. The same call returns the hits for two rescorers and throws for the third. ExpressionRescorer is documented as "Functionally ... the same as SortRescorer".
• Tests. TestSortRescorer.java:96, :134 and TestExpressionRescorer.java:106 call rescore(searcher, hits, 10) on 3 hits; the assert at SortRescorer.java:86 (hits.length == rescoredDocs.scoreDocs.length) holds only when topN is at least the hit count.
• Precedent. LUCENE-2119: "Don't throw NegativeArraySizeException if you pass Integer.MAX_VALUE as nDocs to IndexSearcher search methods"; LUCENE-4938: "Don't use an unnecessarily large priority queue in IndexSearcher methods that take top-N" (lucene/CHANGES.txt:18439-18441, :13901-13902). IndexSearcher caps at max(1, maxDoc) (IndexSearcher.java:721, :729) and TestIndexSearcher.testHugeN ("should not throw exception") calls search/searchAfter with Integer.MAX_VALUE, with and without a Sort. As Michael McCandless wrote on SOLR-5661: "Lucene's IndexSearcher tries to prevent this, when you ask for Integer.MAX_VALUE as the topN hits to the search method, it drops that to the maxDoc for the reader" (the precedent and its framing are theirs). testHugeN as precedent was first pointed out by an independent reviewer from another model family (OpenAI GPT-5.6), in its review of a sibling candidate of the same hunt (RescoreTopNQuery and the LatLonPoint nearest queues).
• History. The line has passed topN since the class was written (LUCENE-5545, 2014); the 8 later commits that changed it are API migrations that keep topN (e.g. 38ca8d3e42c LUCENE-10002, bc18828fccc #13977), and the issue thread never discusses sizing.

The intended-behaviour checker, the reviewers from another model family (OpenAI GPT-5.6) and the adjudication built the strongest case: the pre-allocation is documented, the caller chooses topN, Rescorer is @lucene.experimental, the documented use keeps topN below the hit count, and the LUCENE-2119 description (2009) even called Integer.MAX_VALUE "a nonsense value to pass in, since our PQ impl allocates the array up front". Direct users of TopFieldCollectorManager and TopScoreDocCollectorManager choose N and get an N-sized collector by design. So the answer could be "pass scoreDocs.length". The cross-model fact check rates this case moderate. On the related open #9588 (LUCENE-8542), Adrien Grand wrote that Lucene computes "the top-k matches of a query for some reasonable value of k" (credit them if this framing is used).

Why it fails: the NOTE is on the collector manager, not on SortRescorer or Rescorer; the same LUCENE-2119 text goes on "This trips users up fairly often because they assume our PQ doesn't statically pre-allocate (a reasonable assumption...)", and that issue was resolved by capping, not by rejecting the input (Michael McCandless, who on #9588 also called full-topN pre-allocation "really unexpected behavior"); the sibling rescorers in the same API cap; upstream tests use topN above the hit count; @lucene.experimental permits changing the API, which a cap does not need; and a topN that passes the queue's guard still fails on the heap, while a smaller one silently costs 120 MB for 5 hits. Sizing by N fits a collector that cannot know how many hits will arrive; SortRescorer already holds the finite hit list, so the exact bound, hits.length, is in hand. Verdict: not intended; expect the "pass scoreDocs.length" reply and answer it with the siblings and the IndexSearcher precedent.

One fix was built and validated; two alternatives were not built. LFV-993's fix (the allocation order in FieldValueHitQueue on main) is separate: it would turn main's OutOfMemoryError back into the release IllegalArgumentException, but this call would still fail and still allocate by topN.

In SortRescorer.rescore, before the collector is built (1 file, +3/-1). The finder and the independent reproducer each wrote the same expression; a Z3 check of the hunt shows the size stays in [1, MAX_ARRAY_LENGTH) and that min(size, hits) == min(topN, hits) for every topN >= 1.

// The collector pre-allocates numHits slots, and at most hits.length hits can come back:
int numHits = Math.min(topN, Math.max(1, hits.length));
TopFieldCollector collector =
    new TopFieldCollectorManager(sort, numHits, null, Integer.MAX_VALUE).newCollector();

Trade-offs

• Same results for every topN >= 1: the collector never holds more than hits.length entries anyway. max(1, ...) keeps an empty first pass legal (TopFieldCollectorManager rejects numHits <= 0); topN <= 0 is still rejected, as today.
• Fixes Expression#getRescorer (inherited). Matches QueryRescorer, DoubleValuesSourceRescorer and IndexSearcher's cap.
• Leaves LFV-424 (topN below the hit count) and LFV-993 (main's allocation order) as they are.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, JDK 25.0.3. Stock + the repro test: 1 test, 1 failed (OutOfMemoryError), assertions on and off.
• Fix + the repro test: TestSortRescorerTopN, TestSortRescorer, TestQueryRescorer and TestDoubleValuesSourceRescorer 15 tests, TestExpressionRescorer 4 tests, 0 failed, with -Ptests.asserts=true and false.
• Fix: the whole org.apache.lucene.search.* test package of lucene/core: 215 suites, 2,340 tests, 49 skipped, 0 failed; :lucene:core:check -x test (google-java-format, ecjLint, forbiddenApis): passes. The full Lucene suite was not run.
• The finder's 3-test JUnit (SortRescorer at Integer.MAX_VALUE and at topN = 2,147,483,630, Expression#getRescorer at Integer.MAX_VALUE) with the fixed class: 3 of 3 pass. With the fix one call allocates 1,840 bytes at every topN for 5 hits.

Math.min(topN, Math.max(1, searcher.getIndexReader().maxDoc())). Not built.

Trade-offs

Consistent with IndexSearcher.java:721, but still allocates in proportion to the index size for a handful of hits; hits.length is the exact bound and is already in hand.

A javadoc NOTE on SortRescorer/Rescorer, or an IllegalArgumentException for topN > hits.length. Not built.

Trade-offs

Rejecting breaks the upstream tests that call rescore(..., 10) on 3 hits and makes SortRescorer stricter than its siblings; a NOTE alone leaves the Integer.MAX_VALUE idiom broken.

Method: non-formal first signal, then a differential harness (the hunt's hunt unit for rescorers and other top-N queues pre-sized by an unbounded topN, 2026-10-28, main a9779ef3c193).

• Lead. A planning probe of a numeric-limits round read the code and ran SortRescorer and Expression#getRescorer at topN = Integer.MAX_VALUE, 10^8 and 10^7 on 5 hits: OutOfMemoryError, a heap OutOfMemoryError after 762 MiB under a 1 GiB heap, and 114 MiB allocated, while the QueryRescorer and DoubleValuesSourceRescorer controls returned the 5 hits.
• Harness. A bounded-exhaustive differential harness on the stock jar: 39 index layouts of 1 to 3 segments x 11 topN values, with oracles for the prefix property, the exact hit count, a reference sort model and an O(hits) memory bound. 234 SortRescorer/ExpressionRescorer failures (every layout at the two largest topN values, plus the memory oracle), 0 for the controls, 0 with the fix.
• Z3. Bit-vector obligations show the cap Math.min(topN, Math.max(1, hits.length)) stays in [1, MAX_ARRAY_LENGTH) and keeps min(size, hits) == min(topN, hits); two mutants are caught.
• JUnit. Reduced to a public-API JUnit; this doc then reduced it to the one-method test above.