On main (11.0, unreleased), a field-sorted collector asked for Integer.MAX_VALUE hits throws OutOfMemoryError instead of IllegalArgumentException

On main, new TopFieldCollectorManager(sort, Integer.MAX_VALUE, null, 1).newCollector() with a built-in sort (long, string, relevance and index order executed) throws OutOfMemoryError: Requested array size exceeds VM limit. Lucene 9.10.0 to 10.6.0 refuse the same call with IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647, releases from 7.5.0 refuse the equivalent TopFieldCollector.create call the same way, and main's score-sorted TopScoreDocCollectorManager still throws an IllegalArgumentException (with its own message). Since #14873 (39549329509, 2025-07-03), FieldValueHitQueue.create builds the sort comparators, each pre-allocating numHits slots, before it calls the constructor that reaches PriorityQueue's size check, a guard SOLR-5661 added in 4.7.0 "to prevent confusing OOME" and Lucene fixed again for exactly Integer.MAX_VALUE in 2018 (7.5.0). The call never succeeded on any version, so only the error type changes; IndexSearcher.search(query, n, sort) caps n at maxDoc and is not affected, while direct users and SortRescorer are. No release has it; the 11.0 release branch branch_11x has the same code. A small up-front size check in create() restores the old error; it passes the minimal test with assertions on and off, the closest core suites and core's static checks.

• TopFieldCollectorManager: builds the collectors for a search sorted by Sort fields. Its javadoc says the collectors "pre-allocate a full array of length numHits" (TopFieldCollectorManager.java:44-45, :96-97); its constructor rejects only numHits <= 0 (:62-64).
• FieldValueHitQueue: the field-sorted hit queue, a PriorityQueue (@lucene.experimental), built by the static create(fields, size).
• The guard: PriorityQueue's constructor throws IllegalArgumentException for maxSize < 0 || maxSize >= ArrayUtil.MAX_ARRAY_LENGTH, with the comment "Throw exception to prevent confusing OOME" (PriorityQueue.java:121-125). MAX_ARRAY_LENGTH is 2147483631 here, so on main and since 7.5.0 the guard covers the 17 values 2147483631 to 2147483647.

1. newCollector() calls FieldValueHitQueue.create(sort.getSort(), numHits) (TopFieldCollectorManager.java:112-114), the only production caller of create.
2. create() (FieldValueHitQueue.java:132-159) first loops over the sort fields and calls field.getComparator(size, ...) for each (:142-151, the call at :146). Each built-in comparator allocates size slots in its constructor: LongComparator.java:38 values = new long[numHits];, and likewise the int, float, double, relevance, string and doc comparators. A CUSTOM sort gets its comparator from user code (SortField.java:511-514); one that does not pre-allocate reaches the guard and gets the normal IllegalArgumentException.
3. Only then does it call new FieldValueHitQueue<>(fields, size, ...) (:153-158), whose constructor calls super(size, lessThan) (:114-115) and so reaches the guard. For Integer.MAX_VALUE the first comparator's new long[2147483647] has already thrown OutOfMemoryError: Requested array size exceeds VM limit, so the guard never runs.
4. Up to 10.6.0 the queue's constructor called super(size) first and built the comparators after it (10.5.2 and 10.6.0 FieldValueHitQueue.java:122-123). #14873 moved the comparator loop into create(), ahead of the constructor, because the new LessThan object needs the comparators.

numHits = 2147483647 (Integer.MAX_VALUE): main throws OutOfMemoryError: Requested array size exceeds VM limit from LongComparator.<init>:38 via FieldValueHitQueue.create:146; 10.5.2 and 10.6.0 throw IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647. numHits = 2147483631 with a 1 GB heap: main OutOfMemoryError: Java heap space, 10.5.2 the IllegalArgumentException. numHits = 2147483630, one below the window: main, 10.5.2 and 10.6.0 throw OutOfMemoryError: Java heap space (4.7.0, whose older guard was wider, throws its IllegalArgumentException). The score-sorted TopScoreDocCollectorManager on main throws IllegalArgumentException: initialCapacity must be > 0 and < 2147483630 for the window, from TernaryLongHeap's copy of the guard.

• Who is hit. Code on a main / 11.0 build that passes a numHits of 2147483631 or more (in practice Integer.MAX_VALUE meaning "all") to a field-sorted collector without capping it: direct TopFieldCollectorManager users, and callers that forward an uncapped size, such as SortRescorer.rescore(searcher, topDocs, topN) (SortRescorer.java:49) and ExpressionRescorer, which extends it. A manager built with such a size also fails inside IndexSearcher.search(query, manager), which calls newCollector() without capping (IndexSearcher.java:750-752). Any built-in SortField type, including Sort.RELEVANCE and Sort.INDEXORDER (a CUSTOM sort depends on its comparator source), default settings.
• Not hit. IndexSearcher.search(query, n, sort) and searchAfter cap n at max(1, maxDoc) before building the manager (IndexSearcher.java:721, :729, :733). No release: none throws this OutOfMemoryError from a comparator (sampled from 4.7.0 to 10.6.0).
• What changes. Only the error: the call never succeeded on any version. Code that catches IllegalArgumentException to report a bad size no longer catches it. With -XX:+ExitOnOutOfMemoryError the JVM terminates on main ("Terminating due to java.lang.OutOfMemoryError: Requested array size exceeds VM limit", exit 3) where 10.5.2 throws a catchable exception and keeps running. Lucene's gradle test JVMs run with -XX:+HeapDumpOnOutOfMemoryError (TestsAndRandomizationPlugin.java:451 in build-tools/build-infra), so the recorded test run wrote a heap dump.
• Scale and likelihood. No index or data plays a part; the first such call fails at once, so a developer meets it in testing. Likelihood rare, priority Low.

TestTopFieldCollectorManagerMaxNumHits builds new Sort(new SortField("price", SortField.Type.LONG)) and asserts expectThrows(IllegalArgumentException.class, () -> new TopFieldCollectorManager(sort, Integer.MAX_VALUE, null, 1).newCollector()). It sits in org.apache.lucene.search like TestTopFieldCollector but uses only public API; no randomness of its own. Integer.MAX_VALUE gives the heap-independent "Requested array size exceeds VM limit"; at 2147483631, the bottom of the window, main gives the heap-dependent Java heap space (1 GB heap); the values in between were not run. It compiles unchanged against 10.5.2 and 10.6.0, where it passes with assertions on and off.

As the regression test: this same test, for example as a method in TestTopFieldCollector. It passes with the recommended fix.

cp TestTopFieldCollectorManagerMaxNumHits.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.search.TestTopFieldCollectorManagerMaxNumHits -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Dumping heap to java_pid4670.hprof ...

TestTopFieldCollectorManagerMaxNumHits > testNumHitsIntegerMaxValue FAILED
    junit.framework.AssertionFailedError: Unexpected exception type, expected IllegalArgumentException but got java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:FCF900E87FF408D6]:0)
        at app//org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1437)
        at app//org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
        at app//org.apache.lucene.search.TestTopFieldCollectorManagerMaxNumHits.testNumHitsIntegerMaxValue(TestTopFieldCollectorManagerMaxNumHits.java:26)

        Caused by:
        java.lang.OutOfMemoryError: Requested array size exceeds VM limit
            at org.apache.lucene.search.comparators.LongComparator.<init>(LongComparator.java:38)
            at org.apache.lucene.search.SortField.getComparator(SortField.java:503)
            at org.apache.lucene.search.FieldValueHitQueue.create(FieldValueHitQueue.java:146)
            at org.apache.lucene.search.TopFieldCollectorManager.newCollector(TopFieldCollectorManager.java:114)
            at org.apache.lucene.search.TestTopFieldCollectorManagerMaxNumHits.lambda$testNumHitsIntegerMaxValue$0(TestTopFieldCollectorManagerMaxNumHits.java:28)

• Stated purpose. PriorityQueue.java:121-125: if ((maxSize < 0) || (maxSize >= ArrayUtil.MAX_ARRAY_LENGTH)) { // Throw exception to prevent confusing OOME:. CHANGES.txt:12883-12884, 4.7.0 "Bug fixes": "SOLR-5661: PriorityQueue now refuses to allocate itself if the incoming maxSize is too large (Raintung Li via Mike McCandless)".
• Fixed again for exactly this value. From 4.9.1 to 7.4.0 the guard let Integer.MAX_VALUE overflow past it (LUCENE-5844, b293b947cb3, moved heapSize = maxSize + 1 ahead of the check), so the call threw NegativeArraySizeException. Michael McCandless fixed that in 2018 (4c646dab3fb, "Fix PriorityQueue constructor to throw IllegalArgumentException if you pass Integer.MAX_VALUE as maxSize (thanks rsaavedraf)", released in 7.5.0), writing today's check and message and adding TestPriorityQueue.testMaxIntSize.
• Re-applied in newer heaps. LongHeap.java:57-58 and TernaryLongHeap.java:57-58 carry the same check and comment; main's TopScoreDocCollectorManager rejects these sizes with an IllegalArgumentException from that copy ("initialCapacity must be > 0 and < 2147483630"), so the two top-N collector families now disagree.
• A refactor, not a decision. #14873 ("Change PriorityQueue to take a LessThan interface implementation in its constructor rather than overriding the lessThan abstract method", CHANGES.txt:75-76, 11.0.0 API Changes) moved the comparator loop for a structural reason; its MIGRATE note and review thread discuss the API and performance, not size validation.
• Tests. TestPriorityQueue.testMaxIntSize (TestPriorityQueue.java:378-382) pins the IllegalArgumentException for the bare queue (added by the 2018 fix); no core search test gives a field-sorted collector a numHits near Integer.MAX_VALUE (those arguments are totalHitsThreshold), so nothing caught the reorder.

The strongest case for "intended" (put forward by an independent reviewer from another model family, OpenAI GPT-5.6, and argued in full by the intended-behaviour check): no javadoc promises the IllegalArgumentException; FieldValueHitQueue is @lucene.experimental and PriorityQueue @lucene.internal; both javadocs document the numHits pre-allocation, and create() says only "Must be greater than zero"; IndexSearcher caps n, and, as Michael McCandless pointed out on SOLR-5661, callers should bound the size they pass (Adrien Grand described top-k collection as designed for "some reasonable value of k" on LUCENE-8542); and on main and in 10.x the guard covers only 17 values: at 2147483630, main, 10.5.2 and 10.6.0 throw a heap OutOfMemoryError (1 GB heap).

Why it fails: the guard is a deliberate bug fix with a stated purpose, Lucene fixed it again in 2018 for exactly Integer.MAX_VALUE, re-applies it in newer heaps, main's other top-N collector still has it, and the refactor did not mean to drop it; core's own SortRescorer forwards an uncapped topN into this path. The honest residue: the fix restores a clearer error, not a capability, and the contract rests on intent and history, not on a javadoc.

Option A was validated through gradle in a scratch worktree of 645d4fe14a20. B and C come from the adjudication's and the reviewers' notes; neither was run.

In FieldValueHitQueue.create, right after the empty-sort check, throw the PriorityQueue message when size < 0 || size >= ArrayUtil.MAX_ARRAY_LENGTH, and import ArrayUtil.

diff --git a/lucene/core/src/java/org/apache/lucene/search/FieldValueHitQueue.java b/lucene/core/src/java/org/apache/lucene/search/FieldValueHitQueue.java
index 104b438a721..ff0fbcc1948 100644
--- a/lucene/core/src/java/org/apache/lucene/search/FieldValueHitQueue.java
+++ b/lucene/core/src/java/org/apache/lucene/search/FieldValueHitQueue.java
@@ -18,6 +18,7 @@ package org.apache.lucene.search;
 
 import java.io.IOException;
 import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.util.ArrayUtil;
 import org.apache.lucene.util.PriorityQueue;
 
 /**
@@ -135,6 +136,12 @@ public class FieldValueHitQueue<T extends FieldValueHitQueue.Entry> extends Prio
     if (fields.length == 0) {
       throw new IllegalArgumentException("Sort must contain at least one field");
     }
+    // the comparators below pre-allocate arrays of this size: reject it before they do, as
+    // PriorityQueue would (SOLR-5661), instead of failing with a confusing OutOfMemoryError
+    if (size < 0 || size >= ArrayUtil.MAX_ARRAY_LENGTH) {
+      throw new IllegalArgumentException(
+          "maxSize must be >= 0 and < " + ArrayUtil.MAX_ARRAY_LENGTH + "; got: " + size);
+    }
 
     int numComparators = fields.length;
     FieldComparator<?>[] comparators = new FieldComparator<?>[numComparators];

Trade-offs

• Restores the 10.x behaviour exactly, with the same message, at the one place every field-sorted collector goes through, including external callers of the public create().
• Cost: one comparison per newCollector(), not per hit.
• It duplicates the guard's bound and message; a shared helper could avoid that.
• It turns the main-only OutOfMemoryError of uncapped callers (the SortRescorer and grouping findings LFV-997 and LFV-964) back into this exception without fixing their own sizing.
• The diff is against 645d4fe14a20; git apply rejects it on upstream main 2f424fd428da and on branch_11x (#16857 added an import next to its context). The same 7 lines regenerated against 2f424fd428da pass git apply --check there and on branch_11x, but were not compiled or tested on those revisions.
• Like PriorityQueue, it accepts size == 0, while create()'s javadoc says the size "Must be greater than zero" (FieldValueHitQueue.java:130); TopFieldCollectorManager already rejects numHits <= 0, so only direct create() callers can pass 0.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF, 2026-11-01: without the fix the test fails with assertions on and off; with it, it passes with assertions on and off.
• Closest core suites with the fix: TestTopFieldCollector (13 tests), TestTopFieldCollectorEarlyTermination (5), TestSort (30), TestSortRandom (1), TestSortOptimization (31), TestSortRescorer (3), TestPriorityQueue (18) and the new test: 8 classes, 102 tests, 0 failures, 0 skipped.
• ./gradlew :lucene:core:check -x test with the fix and the test as a new file: passes (google-java-format, ecj lint of main and test code, forbidden APIs). The full Lucene test suite was not run.
• Earlier: the finder's 4-test JUnit passes with the fix (assertions on and off), and so do TestTopFieldCollector + TestSort (43 of 43) on a9779ef3c193; the minimal test passes with the fix shadowed into the stock jars.

Next to the existing numHits <= 0 check (TopFieldCollectorManager.java:62-64).

if (numHits >= ArrayUtil.MAX_ARRAY_LENGTH) {
  throw new IllegalArgumentException(
      "numHits must be < " + ArrayUtil.MAX_ARRAY_LENGTH + "; got: " + numHits);
}

Trade-offs

• Fails earlier, at construction, and covers every Lucene caller today: newCollector() is the only production caller of create().
• Misses direct callers of the public FieldValueHitQueue.create.

What the checks validated

Not run.

Restore the old order inside create().

Trade-offs

The LessThan handed to the queue's constructor needs the comparators, so this needs a late-bound comparator holder: more change for the same effect.

What the checks validated

Not run.

• First signal: an executed probe on the stock jars (2026-10-28) ran SortRescorer and the expressions rescorer with topN = Integer.MAX_VALUE and saw OutOfMemoryError: Requested array size exceeds VM limit on main but the PriorityQueue IllegalArgumentException on 10.5.2 and the 10.6 release candidate.
• Root cause: the finder, explaining that main-vs-release difference, read FieldValueHitQueue.create and its git history and found the comparator loop that #14873 moved ahead of the queue constructor.
• Confirmation: a public-API JUnit (long, relevance and string sorts plus a score-sorted control) failed 3 of 4 on main and passed on 10.5.2 and the release candidate. No model or formal tool was involved, so the method is non-formal.
• Later: independent verifiers, cross-model reviews (reading only), an adjudication, the registration check, a credibility audit, the one-test minimal repro (2026-10-29) and this doc's gradle run (2026-11-01).