SearchGroup.merge pre-sizes its result by topN, so merging shard groups with topN = Integer.MAX_VALUE throws OutOfMemoryError (and NullPointerException for offset >= 1)

SearchGroup.merge(topGroups, offset, topN, groupSort) is the public, @lucene.experimental helper that merges the first-pass top groups of several shards. It creates its result list as new ArrayList<>(topN) (SearchGroup.java:294), sized by the request rather than by the groups the shards hold, so topN = Integer.MAX_VALUE (or MAX - 1), a common way to ask for every group, throws OutOfMemoryError: Requested array size exceeds VM limit at any offset, even when only 2 groups exist; smaller large values reserve about 4 bytes per requested group (400 MB at 10^8) and throw Java heap space when the heap is smaller than that. Up to 6.1.0 the offset-0 call returned the groups: the pre-size came with LUCENE-7356 in 6.2.0. Independently, the queue bound offset + topN is an int sum (:281), so with offset >= 1 and offset + topN >= 2^31 it wraps negative and, as soon as one shard holds a group, the pruning loop dereferences null (:274), as it has since the method was added in 3.3.0. The javadoc sets no bound and nothing checks the arguments.

Reach is narrow: Lucene's own FirstPassGroupingCollectorManager and GroupingSearch fail earlier for these values, in the first-pass collector (LFV-964), so only code that calls SearchGroup.merge directly is hit. The failure is loud: no wrong groups. A small fix in GroupMerger.merge (saturate the sum at :281 and size the list at :294 by the groups the shards hold, counted in a short loop) removes both failures and passes the grouping module's tests (57 with the repro).

• :281: final int maxQueueSize = offset + topN; an int sum that wraps for offset + topN >= 2^31.
• :271-275: after each insertion updateNextGroup prunes while (queue.size() > topN) (here topN is maxQueueSize) with queue.pollLast() and then group.inQueue = false, with no null check. With a negative bound it empties the TreeSet, pollLast() returns null and :274 throws NullPointerException: Cannot assign field "inQueue" because "group" is null.
• :294: new ArrayList<>(topN): the result list is sized by the caller's topN. For Integer.MAX_VALUE and MAX - 1 the tested JVM (OpenJDK 25.0.3) refuses the array before allocating (Requested array size exceeds VM limit), with a 64 MiB heap as with the default heap; for smaller large values it allocates topN references, about 4 bytes each with compressed references.
• :298-320: the pull loop stops when the queue is empty or at topN results, so it never needs more slots than there are groups.

Assertions on and off give the same lines. So the pre-size fails at any offset, and only a wrapping sum fails earlier, while the queue is seeded.

One merge of three groups allocates 1,360 bytes for topN = 10, 4,001,320 for 10^6 and 400,001,320 for 10^8; with the fix 1,368 bytes for all three (measured on main a9779ef3c193, identical grouping code, OpenJDK 25.0.3 with compressed references). Below the VM limit the call needs that much free heap: topN = 10^8 throws OutOfMemoryError: Java heap space with -Xmx64m and returns the groups with -Xmx1g; 10^9 throws with -Xmx1g.

• Who is hit. Code that merges first-pass top groups itself with SearchGroup.merge and passes topN = Integer.MAX_VALUE (or MAX - 1) for "all remaining groups": every such call throws, at any offset, however few groups exist. With offset >= 1, any offset + topN >= 2^31 throws NullPointerException as soon as at least one shard holds a group (with only empty shards the pruning loop never runs, so there is no NullPointerException: the call returns no groups, or fails at the pre-size when topN itself is near Integer.MAX_VALUE; probe). Two executed examples: each shard collected by a bounded FirstPassGroupingCollectorManager(..., 0, 10) and then merged with merge(shardGroups, 0, Integer.MAX_VALUE, sort); and each shard collected completely (topNGroups = the shard's maxDoc) and merged with "all". Both throw; the same merge with a finite topN returns the groups.
• Who is not. FirstPassGroupingCollectorManager and GroupingSearch callers with these values fail earlier, in the first-pass collector constructor: NegativeArraySizeException for (0, MAX), IllegalArgumentException "topNGroups must be >= 1 (got -2147483648)" for (1, MAX), OutOfMemoryError for (0, MAX - 1). That is LFV-964, with its own fix site. Callers with ordinary page sizes are unaffected.
• Large but legal topN. Values such as 10^8 used as "all" reserve about 4 bytes per requested group per merge (400 MB at 10^8): they return the right groups when the heap has room, and throw OutOfMemoryError: Java heap space when it does not (10^8 with a 64 MiB heap, probe).
• Default vs opt-in. No setting or index is involved; SearchGroup is @lucene.experimental; the trigger is the topN value of a direct call.
• Severity and scale. An exception for that call; no wrong or missing groups, nothing persisted. Every release from 6.2.0 (offset 0) or 3.3.0 (offset >= 1) through 10.6.0, and main. Rare: the documented searches (Prior reports, below) found no public report.

Put the file in lucene/grouping/src/test/org/apache/lucene/search/grouping/ and run it with gradle. The OutOfMemoryError is caught once and rethrown as an AssertionError whose message states expected against actual; the error stays as the cause. On the tested JVM (OpenJDK 25.0.3) the array is refused before anything is allocated, with a 64 MiB heap as with the default heap.

• Formatting: google-java-format wraps two over-long lines (the shards declaration and the AssertionError); nothing else changes, and the wrapped file passes checkGoogleJavaFormat.
• On unfixed code the test JVM also writes a heap dump of about 57 MB into lucene/grouping/build/tests-cwd/, because Lucene's build runs tests with -XX:+HeapDumpOnOutOfMemoryError.
• The test covers the offset-0 leg. The offset >= 1 leg is the one-line call SearchGroup.merge(shards, 1, Integer.MAX_VALUE, sort) on the same input, which throws NullPointerException at SearchGroup.java:274 (probe); the same fix removes it.

cp $REPRO/TestSearchGroupMergeAllGroups.java $LUCENE/lucene/grouping/src/test/org/apache/lucene/search/grouping/
cd $LUCENE
./gradlew :lucene:grouping:test --tests org.apache.lucene.search.grouping.TestSearchGroupMergeAllGroups -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false (same failure)

# the file above, main 645d4fe14a20, seed DEADBEEF, assertions on (JUnit console runner):
TestSearchGroupMergeAllGroups:testMergeWithTopNMaxValue
  => java.lang.AssertionError: expected the 2 merged groups for topN = Integer.MAX_VALUE, but got java.lang.OutOfMemoryError: Requested array size exceeds VM limit
     __randomizedtesting.SeedInfo.seed([DEADBEEF:E1AAF9DB920CFC30]:0)
     org.apache.lucene.search.grouping.TestSearchGroupMergeAllGroups.testMergeWithTopNMaxValue(TestSearchGroupMergeAllGroups.java:41)
   Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
     java.base/java.util.ArrayList.<init>(ArrayList.java:157)
     org.apache.lucene.search.grouping.SearchGroup$GroupMerger.merge(SearchGroup.java:294)
     org.apache.lucene.search.grouping.SearchGroup.merge(SearchGroup.java:334)
     org.apache.lucene.search.grouping.TestSearchGroupMergeAllGroups.testMergeWithTopNMaxValue(TestSearchGroupMergeAllGroups.java:38)

# the same failure with assertions off, with gradle ("> Task :lucene:grouping:test FAILED", 1 test, 1 failure)
# and on the 10.6.0 release jars, there from SearchGroup.java:295

# the offset >= 1 leg, assertions off, from the hunt's two-method JUnit (two small indexes, first pass per shard, then the merge);
# the repro's own input gives the same exception at the same line (probe):
SearchGroup.merge(offset=1, topN=MAX_VALUE) threw java.lang.NullPointerException: Cannot assign field "inQueue" because "group" is null
  at org.apache.lucene.search.grouping.SearchGroup$GroupMerger.updateNextGroup(SearchGroup.java:274)
  at org.apache.lucene.search.grouping.SearchGroup$GroupMerger.merge(SearchGroup.java:289)
  at org.apache.lucene.search.grouping.SearchGroup.merge(SearchGroup.java:334)

• No documented bound. The javadoc (SearchGroup.java:326-331) only asks for a matching groupSort and fillFields=true; nothing checks offset or topN (:332-335). The package docs tell sharded users to do this merge themselves (package-info.java:76-77).
• A regression from a micro-optimisation. LUCENE-7356 "SearchGroup tweaks (initialCapacity, size==0 vs. isEmpty)" (#8410, by Christine Poerschke, one "+1" from Adrien Grand) changed new ArrayList<>() to new ArrayList<>(topN). The issue text is just "initialCapacity" and "size()==0 vs. isEmpty()"; nobody discussed large topN. Every release from 3.3.0 (the first with the method) to 6.1.0 returns the groups for topN = Integer.MAX_VALUE (47 releases executed).
• Same module, same pattern, fixed and tested. SOLR-3956 (1ad382f3046, Chris M. Hostetter, 2013) removed an offset + limit overflow from grouped-facet paging; its test still loops limit over {2, 10, 100, Integer.MAX_VALUE} (TestTermGroupFacetCollectorManager.java:106-139). Today GroupedFacetResult.getFacetEntries sizes by Math.min(limit, facetEntries.size() - offset) (GroupedFacetResult.java:91), and core TopDocs.merge by Math.min(size, availHitCount - start) (TopDocs.java:313).
• An internal NPE is no answer. Whatever range is intended, a NullPointerException from polling an empty queue after a signed overflow names an internal field, not an argument. An IllegalArgumentException would be an acceptable alternative.

The strongest case for "works as designed": topN is a concrete count on an @lucene.experimental expert helper; the grouping first pass deliberately allocates by topNGroups (FirstPassGroupingCollector.java:116, :121); Michael McCandless's LUCENE-2119 description calls Integer.MAX_VALUE "a nonsense value to pass in, since our PQ impl allocates the array up front"; and Lucene's own manager never reaches the merge with such values.

Why it fails: SearchGroup.merge has no pre-allocated priority queue; it prunes a TreeSet lazily and worked with Integer.MAX_VALUE for offset 0 from 3.3.0 to 6.1.0. LUCENE-2119 itself was resolved Fixed (IndexSearcher caps n at maxDoc) because "This trips users up fairly often". The module's own SOLR-3956 test treats Integer.MAX_VALUE as "no limit". The reach argument limits who is hit, not whether it is a bug, and the doc states that limit plainly.

Three options; A was built and validated, B and C are alternatives from the hunt's checks.

Compute the queue bound in long and clamp it to Integer.MAX_VALUE, and create the result list with min(topN, total groups in the shard lists) slots. Both reproduced failures go away: the bound cannot wrap, and the list never reserves more slots than the shards hold groups, so LUCENE-7356's capacity hint stays useful for ordinary topN.

// GroupMerger.merge, at the queue bound (was: final int maxQueueSize = offset + topN;)
final int maxQueueSize = (int) Math.min((long) offset + topN, Integer.MAX_VALUE);
long shardGroupCount = 0;
for (Collection<SearchGroup<T>> shard : shards) {
  shardGroupCount += shard.size();
}
// ... and at the result list (was: new ArrayList<>(topN))
final List<SearchGroup<T>> newTopGroups =
    new ArrayList<>((int) Math.min(topN, shardGroupCount));

Trade-offs

One size() call per shard collection; the capacity is an upper bound (a group returned by several shards is counted once per shard), so it is only large when the shards really hold that many groups. Results unchanged is argued from the code and the module's tests, not tested exhaustively. Full diff:

diff --git a/lucene/grouping/src/java/org/apache/lucene/search/grouping/SearchGroup.java b/lucene/grouping/src/java/org/apache/lucene/search/grouping/SearchGroup.java
index 9f434b57d03..83e5167baf4 100644
--- a/lucene/grouping/src/java/org/apache/lucene/search/grouping/SearchGroup.java
+++ b/lucene/grouping/src/java/org/apache/lucene/search/grouping/SearchGroup.java
@@ -278,7 +278,12 @@ public class SearchGroup<T> {
     public Collection<SearchGroup<T>> merge(
         List<Collection<SearchGroup<T>>> shards, int offset, int topN) {
 
-      final int maxQueueSize = offset + topN;
+      // saturate: offset + topN may exceed Integer.MAX_VALUE (e.g. topN = Integer.MAX_VALUE)
+      final int maxQueueSize = (int) Math.min((long) offset + topN, Integer.MAX_VALUE);
+      long shardGroupCount = 0;
+      for (Collection<SearchGroup<T>> shard : shards) {
+        shardGroupCount += shard.size();
+      }
 
       // System.out.println("merge");
       // Init queue:
@@ -291,7 +296,9 @@ public class SearchGroup<T> {
       }
 
       // Pull merged topN groups:
-      final List<SearchGroup<T>> newTopGroups = new ArrayList<>(topN);
+      // size by the groups the shards hold, not by the caller's topN
+      final List<SearchGroup<T>> newTopGroups =
+          new ArrayList<>((int) Math.min(topN, shardGroupCount));
 
       int count = 0;
 

What the checks validated

• gradle in a scratch worktree of main 645d4fe14a20: stock, the repro fails (OOM at :294); with the fix applied by git apply, ./gradlew :lucene:grouping:test -Ptests.seed=DEADBEEF passes with assertions on and with assertions off: 15 classes, 57 tests (the repro included), 0 failures, 0 skipped; :lucene:grouping:checkGoogleJavaFormat passes.
• standalone runner with the fixed class ahead of the stock jars: the repro passes with assertions on and off, the hunt's two-method test (offset 0 and offset 1) passes 2 of 2, and all six probe calls return the groups.
• the hunt's checks of the same code: 29 upstream grouping tests pass; one merge of three groups allocates a constant 1,368 bytes for topN 10, 10^6 and 10^8.

Go back to new ArrayList<>() at :294 and saturate :281. The smallest diff.

Trade-offs

Gives up LUCENE-7356's capacity hint. Not built as one change; the version-history runs ran each half alone: reverting only :294 fixes offset 0 but keeps the NPE, saturating only :281 turns the NPE into the OOM, so both lines are needed.

Validate offset >= 0, topN >= 1 and that offset + topN does not overflow.

Trade-offs

A clear error instead of an internal NPE, but it turns topN = Integer.MAX_VALUE, which worked for offset 0 up to 6.1.0 and which core TopDocs.merge accepts for hits, into an error. Not built; the adjudication and the audit call it an acceptable alternative.

Hunt unit "grouping offset + limit overflow" (2026-10-28), a semi-formal hunt:

• The finder wrote a Z3 bit-vector model of every int size site in the grouping module. Site S6 is SearchGroup.java:281 (the sum) and :294 (the pre-size). Query Q6 is satisfiable at offset 1, topN Integer.MAX_VALUE (Java int sum -2^31); Q6b is satisfiable for the offset-0 pre-size above the VM array limit.
• A shard-merge probe on the stock main jar, assertions off, witnessed both: OutOfMemoryError at :294 for topN MAX - 2 to MAX, and NullPointerException at :274 for offsets 1 and 2 with topN MAX, while finite topN returned the groups.
• The unit was planned from a "top-n size parameters" area and an earlier, never-executed read-only lead on the same int-sum pattern (LFV-262, in core TopDocs.merge).