GroupingSearch throws for groupLimit or groupDocsLimit = Integer.MAX_VALUE instead of returning all groups

GroupingSearch, the grouping module's convenience class, throws when the caller passes Integer.MAX_VALUE ("all") as groupLimit or as groupDocsLimit, although the same search with the limit set to reader.maxDoc() works and IndexSearcher.search(query, Integer.MAX_VALUE) works. With offset 0 the search throws NegativeArraySizeException: -2147483648, OutOfMemoryError: Requested array size exceeds VM limit or a queue-size IllegalArgumentException; with offset 1 it throws IllegalArgumentExceptions that name a value the caller never passed: "topNGroups must be >= 1 (got -2147483648)", "numHits must be > 0" or "maxDocsPerGroup must be >= 1 (got -2147483648)". Every grouping mode fails (term, function, term with caching enabled, the Supplier constructor, doc block), with assertions on and off; a large finite limit such as 10,000,000 does not throw but allocates memory in proportion to the limit. The cause is one missing cap: GroupingSearch hands the caller's offsets and limits unchanged to collector managers that add offset + limit in int and pre-size their queues and comparators by the limit, while IndexSearcher caps the count at maxDoc since LUCENE-2119 (2009). No silently wrong result was observed in 21,040 differential searches. Every release tested since 4.0.0 (2012) fails, as does main; capping the limits at maxDoc in GroupingSearch fixes it, and the grouping module's tests pass with that change.

GroupingSearch.search(searcher, query, groupOffset, groupLimit)      GroupingSearch.java:128-138  values passed on unchanged
 term / function / cached: groupByFieldOrFunction                    :141
   new FirstPassGroupingCollectorManager(.., groupOffset, groupLimit, ..)   :146-148
     FirstPassGroupingCollectorManager.java:92-97  checks only offset >= 0, topNGroups >= 1
     FirstPassGroupingCollectorManager.java:110    groupOffset + topNGroups     <- int: 1 + MAX_VALUE = -2^31
     FirstPassGroupingCollector.java:97-98         "topNGroups must be >= 1 (got " + topNGroups + ")"
     FirstPassGroupingCollector.java:116           getComparator(topNGroups + 1) <- MAX_VALUE + 1 = -2^31
                                                   -> new float[-2147483648]: NegativeArraySizeException
   new TopGroupsCollectorManager(.., groupDocsOffset, groupDocsLimit, ..)   :197-205
     TopGroupsCollectorManager.java:107            withinGroupOffset + maxDocsPerGroup   <- int sum
     TopGroupsCollector.java:127-147               per-group top-N collectors sized by it
 doc block: groupByDocBlock                                          :221
   new BlockGroupingCollectorManager(.., groupDocsOffset + groupDocsLimit)  :226-235  <- int sum at :235
     BlockGroupingCollectorManager.java:143        groupOffset + topNGroups     <- int sum
     BlockGroupingCollector.java:217, :237         queue and comparators sized by topNGroups

IndexSearcher.searchAfter (core):  :588 limit = Math.max(1, reader.maxDoc());  :597 cappedNumHits = Math.min(numHits, limit)

• There are at most maxDoc groups and at most maxDoc documents per group, so a limit above maxDoc never changes the result; it only sizes the collectors. GroupingSearch sizes them for the caller and has no cap.
• Every failing value passes the API's own checks (offset >= 0, limit >= 1).

Function grouping, term grouping with caching enabled (setCaching; it fails in the first pass, before any cached replay) and the Supplier<GroupSelector> constructor fail the same way (with groupLimit = MAX_VALUE, assertions on and off). Below the wrap, memory grows with the limit: on a 20-document index (5 groups of 4, field sort), one term-grouping search allocated 600,119,480 bytes with groupDocsLimit = 10,000,000 and 187,228,760 bytes with groupLimit = 10,000,000, against 56,488 bytes for IndexSearcher.search(q, 10,000,000, sort); with the cap, 120,680 and 120,264 bytes at every limit.

• Who is exposed. Applications that use GroupingSearch and pass Integer.MAX_VALUE as groupLimit or groupDocsLimit to mean "all", or forward an uncapped user-chosen page size or offset. Term, function, cached and doc-block grouping all fail; the default groupDocsLimit is 1, so the per-group path needs an explicit setGroupDocsLimit.
• What happens. That search throws an unchecked exception instead of returning the groups; with offset 1 or more the message names a value the caller never passed. No silent wrong result was observed in 21,040 differential searches, and the index does not change.
• Scale. None for Integer.MAX_VALUE: a 1-document index is enough. Large finite limits cost memory in proportion to the limit (about 600 MB for groupDocsLimit = 10,000,000 on a 20-document index).
• Versions. Every release tested from 4.0.0 (2012) to 10.6.0 fails, and main; releases not run are inferred affected (the same int sums in every tag read, no capping commit).
• How likely. Rare: it fails on the first call, so developers who try this value switch to a finite value; Lucene's own DatasetSplitter passes setGroupDocsLimit(originalIndex.maxDoc()), and no public report was found.
• Workaround. For a valid offset and limit (offset >= 0, limit >= 1), cap both before the call, with the rule of the fix: maxDoc = Math.max(1, reader.maxDoc()), offset = Math.min(offset, maxDoc), limit = Math.max(1, Math.min(limit, maxDoc - offset)), for the group page and for setGroupDocsOffset / setGroupDocsLimit. Leave other values as they are (or validate them first), as the fix does; applied to limit 0 or a negative limit, the formula would turn it into 1 instead of letting the API reject it. Capping only the limit (Math.min(limit, reader.maxDoc())) is not enough: a large offset still wraps offset + limit (e.g. offset Integer.MAX_VALUE - 1 with limit 20), and an empty index gives limit 0, which the API rejects ("topNGroups must be >= 1 (got 0)").

Copy the file into lucene/grouping/src/test/org/apache/lucene/search/grouping/ and run the command. The test body is deterministic (ByteBuffersDirectory, a plain IndexWriterConfig), though it runs under Lucene's randomized test framework; pass -Ptests.asserts explicitly, because by default the build picks it from the seed. With the file added, :lucene:grouping:check -x test passes.

The exception is wrapped in an AssertionError that states expected and actual. The other symptoms (offset 1, groupDocsLimit, doc-block grouping) come from the same missing cap. The test is the suggested regression test as it stands.

./gradlew :lucene:grouping:test --tests "org.apache.lucene.search.grouping.TestGroupingSearchLimitMaxValue" -Ptests.seed=DEADBEEF -Ptests.asserts=true

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestGroupingSearchLimitMaxValue > testGroupLimitMaxValue FAILED
    java.lang.AssertionError: expected the 1 group of the index, got java.lang.NegativeArraySizeException: -2147483648
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:9E3B10358EB89447]:0)
        at org.apache.lucene.search.grouping.TestGroupingSearchLimitMaxValue.testGroupLimitMaxValue(TestGroupingSearchLimitMaxValue.java:48)

        Caused by:
        java.lang.NegativeArraySizeException: -2147483648
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.FieldComparator$RelevanceComparator.<init>(FieldComparator.java:149)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.SortField.getComparator(SortField.java:485)
            at org.apache.lucene.search.grouping.FirstPassGroupingCollector.<init>(FirstPassGroupingCollector.java:116)
            at org.apache.lucene.search.grouping.FirstPassGroupingCollectorManager.newCollector(FirstPassGroupingCollectorManager.java:108)
            at org.apache.lucene.search.grouping.FirstPassGroupingCollectorManager.newCollector(FirstPassGroupingCollectorManager.java:52)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.MultiCollectorManager.newCollector(MultiCollectorManager.java:53)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:752)
            at org.apache.lucene.search.grouping.GroupingSearch.groupByFieldOrFunction(GroupingSearch.java:175)
            at org.apache.lucene.search.grouping.GroupingSearch.search(GroupingSearch.java:131)
            at org.apache.lucene.search.grouping.TestGroupingSearchLimitMaxValue.testGroupLimitMaxValue(TestGroupingSearchLimitMaxValue.java:45)
:lucene:grouping:test (FAILURE): 1 test, 1  failure

# with the recommended fix, assertions on and off:
:lucene:grouping:test (SUCCESS): 1 test
BUILD SUCCESSFUL

• No documented bound. groupLimit: "The number of groups to return from the specified group offset" (GroupingSearch.java:123); groupDocsLimit: "The number of documents to return inside a group" (:321). No javadoc, package doc, CHANGES or MIGRATE entry limits them, and every failing value passes the API's own checks (FirstPassGroupingCollectorManager.java:92-97, BlockGroupingCollectorManager.java:112-128).
• The wrapped values have no rationale. "topNGroups must be >= 1 (got -2147483648)" and "maxDocsPerGroup must be >= 1 (got -2147483648)" report sums that Lucene computed (FirstPassGroupingCollectorManager.java:110, BlockGroupingCollectorManager.java:143, GroupingSearch.java:235).
• Core precedent. lucene/CHANGES.txt, Bug fixes of 2.9.3 / 3.0.2: "LUCENE-2119: Don't throw NegativeArraySizeException if you pass Integer.MAX_VALUE as nDocs to IndexSearcher search methods." The fix is the cap at IndexSearcher.java:588/:597; FacetsCollectorManager and DrillSideways cap the same way.
• Integer.MAX_VALUE per group was made to work elsewhere. In LUCENE-4832 (#5897, 4.3.0), Michael McCandless asked in review "Could we just fix it so passing in Integer.MAX_VALUE for maxDocsPerGroup works?" for the join module's ToParentBlockJoinCollector, and the committed fix capped it with the group size.
• The module's grouped-facet test (the grouped-facet API, not GroupingSearch) passes Integer.MAX_VALUE as a limit and guards the offset + limit sum itself (TestTermGroupFacetCollectorManager.java:106-118); no upstream test asserts the exception.

The strongest case for "intended": the LUCENE-2119 issue text (#3195) calls Integer.MAX_VALUE "a nonsense value to pass in, since our PQ impl allocates the array up front"; core documents that TopScoreDocCollectorManager instances "pre-allocate a full array of length numHits"; no Lucene doc says that Integer.MAX_VALUE means "all" for these limits, and no index can hold that many documents (IndexWriter.MAX_DOCS is smaller); GroupingSearch is @lucene.experimental; the defaults are safe; Solr caps in its own layer; and the doc-block offset-0 case already gives an honest queue-size message.

Why it fails: LUCENE-2119 itself was resolved as a bug by capping in the convenience layer, and GroupingSearch is the convenience layer for grouping, which sizes the collectors on the caller's behalf; the pre-allocation note is on the core collector managers, not on GroupingSearch's limits; and no design reading explains an exception that names -2147483648. The defence fits the memory cost of large finite limits best, so present that part as secondary.

Option A fixes every GroupingSearch caller. Code that builds the collector managers directly still gets the wrapped sums; option B covers it and can be added to A. No CHANGES entry is drafted.

In GroupingSearch.search, cap groupOffset and groupLimit; for the documents per group, pass capped groupDocsOffset and groupDocsLimit to TopGroupsCollectorManager and BlockGroupingCollectorManager. Invalid values (negative offset, limit < 1) pass through unchanged, so they behave exactly as today: the group-page checks reject them, and invalid per-group values give the same result with and without the patch. +29/-4 lines in GroupingSearch.java. The code below is the complete patch; git apply takes it on 645d4fe14a20.

diff --git a/lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupingSearch.java b/lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupingSearch.java
index 88b21b50686..2edabe58ce7 100644
--- a/lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupingSearch.java
+++ b/lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupingSearch.java
@@ -127,6 +127,15 @@ public class GroupingSearch {
   @SuppressWarnings("unchecked")
   public <T> TopGroups<T> search(
       IndexSearcher searcher, Query query, int groupOffset, int groupLimit) throws IOException {
+    // A page of groups cannot extend past maxDoc (there are at most maxDoc groups): cap the
+    // caller's values, as IndexSearcher#search caps numHits, so that "offset + limit" cannot
+    // overflow and the collectors are not sized beyond the index. Invalid values are passed
+    // through unchanged so that the collectors still reject them.
+    final int maxDoc = Math.max(1, searcher.getIndexReader().maxDoc());
+    if (groupOffset >= 0 && groupLimit >= 1) {
+      groupOffset = Math.min(groupOffset, maxDoc);
+      groupLimit = Math.max(1, Math.min(groupLimit, maxDoc - groupOffset));
+    }
     if (grouperFactory != null) {
       return groupByFieldOrFunction(searcher, query, groupOffset, groupLimit);
     } else if (groupEndDocs != null) {
@@ -137,6 +146,22 @@ public class GroupingSearch {
     }
   }
 
+  /** groupDocsOffset capped at maxDoc (same rule as the group page in #search). */
+  private int cappedDocsOffset(IndexSearcher searcher) {
+    final int maxDoc = Math.max(1, searcher.getIndexReader().maxDoc());
+    return groupDocsOffset >= 0 && groupDocsLimit >= 1
+        ? Math.min(groupDocsOffset, maxDoc)
+        : groupDocsOffset;
+  }
+
+  /** groupDocsLimit capped so that the capped offset plus the limit is at most maxDoc + 1. */
+  private int cappedDocsLimit(IndexSearcher searcher) {
+    final int maxDoc = Math.max(1, searcher.getIndexReader().maxDoc());
+    return groupDocsOffset >= 0 && groupDocsLimit >= 1
+        ? Math.max(1, Math.min(groupDocsLimit, maxDoc - cappedDocsOffset(searcher)))
+        : groupDocsLimit;
+  }
+
   @SuppressWarnings({"unchecked", "rawtypes"})
   protected <T> TopGroups<T> groupByFieldOrFunction(
       IndexSearcher searcher, Query query, int groupOffset, int groupLimit) throws IOException {
@@ -200,8 +225,8 @@ public class GroupingSearch {
             topSearchGroups,
             groupSort,
             sortWithinGroup,
-            groupDocsOffset,
-            groupDocsLimit,
+            cappedDocsOffset(searcher),
+            cappedDocsLimit(searcher),
             includeMaxScore);
 
     TopGroups<T> secondResult;
@@ -231,8 +256,8 @@ public class GroupingSearch {
             groupSort.needsScores() || sortWithinGroup.needsScores(),
             groupEndDocs,
             sortWithinGroup,
-            groupDocsOffset,
-            groupDocsOffset + groupDocsLimit);
+            cappedDocsOffset(searcher),
+            cappedDocsOffset(searcher) + cappedDocsLimit(searcher));
 
     return searcher.search(query, bcm);
   }

Trade-offs

A few cheap maxDoc() reads per search (the patch reads it up to 4 times for term and function grouping and 5 times for doc block, because the helpers recompute it and cappedDocsLimit calls cappedDocsOffset); no result changed in the differential. Direct users of FirstPassGroupingCollectorManager, TopGroupsCollectorManager and BlockGroupingCollectorManager are not covered (option B). Computing the capped values once is a possible clean-up.

What the checks validated

• Gradle in a scratch worktree of main 645d4fe14a20, seed DEADBEEF, assertions on and off: the repro alone fails stock and passes with the fix.
• The repro with TestGroupingSearch, the upstream test class of GroupingSearch: stock 1 failure (the repro), with the fix 4 of 4 pass, each way.
• All of :lucene:grouping:test with the fix, assertions on: 15 classes (14 upstream), 57 tests, 0 failures.
• :lucene:grouping:check -x test (google-java-format, ecjLint, forbiddenApis) passes with the fix and the repro added; the hunt's diff needed its 4-line comment re-wrapped (three of its lines were 101 columns).
• Re-run in a second fresh scratch worktree of 645d4fe14a20 (2026-10-30): stock, the repro fails with assertions on and off; with the fix, the repro and TestGroupingSearch pass 4 of 4 each way and all 57 tests of :lucene:grouping:test pass. It did not repeat check -x test.
• The fact check's own fresh scratch worktree of 645d4fe14a20 (2026-10-30), with the patch extracted from this doc and applied with git apply: stock, the repro command fails with assertions on and off; with the patch, it passes both ways, the repro and TestGroupingSearch pass 4 of 4 each way, all 57 tests of :lucene:grouping:test pass and :lucene:grouping:check -x test passes.
• Earlier, in the repro harness: the finder's 8-case JUnit passes 8 of 8, 29 upstream grouping tests pass, a 21,040-search differential agrees 21,040 of 21,040, and a z3 bit-vector model proves, for the int arithmetic and an abstract model of the group window (not the collectors' code), that the capped window selects the same groups and no sum can wrap for any maxDoc up to IndexWriter.MAX_DOCS; the allocation no longer grows with the limit.

Make the offset + limit sums in FirstPassGroupingCollectorManager.newCollector, TopGroupsCollectorManager.newCollector, BlockGroupingCollectorManager.newCollector and GroupingSearch.groupByDocBlock overflow-safe: reject offset > Integer.MAX_VALUE - limit in the constructors with a clear message, or saturate the sum.

Trade-offs

Removes the misleading -2147483648 messages for every caller, but on its own does not make "all" work: the first-pass collector still computes topNGroups + 1, and the queues are still pre-sized by the limit. Best as a complement to A.

State in the javadoc of search and setGroupDocsLimit that the limits pre-allocate and must not exceed maxDoc, and reject larger values with a clear message.

Trade-offs

Honest, but callers still need their own cap, and it differs from IndexSearcher.search, which accepts Integer.MAX_VALUE.

• First signal (non-formal). Code reading of the PriorityQueue / TopDocs.merge area (2026-10-26) noted the grouping module's int offset + limit sums as a lead without running them; a planning probe on 2026-10-28 ran them on main, 10.5.2 and branch_10_6 and showed NegativeArraySizeException: -2147483648 and "topNGroups must be >= 1 (got -2147483648)".
• Then (semi-formal). A public-API JUnit (term and doc-block grouping x groupLimit or groupDocsLimit = Integer.MAX_VALUE x offset 0 or 1, each compared with the same search at limit maxDoc): 8 of 8 fail. A bounded-exhaustive differential harness against an exact long-arithmetic reference: 6 seeded indexes of 2-15 documents in 1-3 segments, 5 grouping modes, field and relevance sorts, sequential and executor searchers, a boundary grid of offsets and limits: 21,040 searches, 7,872 agree, 13,168 throw, 0 silent mismatches; 21,040 of 21,040 agree with the cap. A z3 bit-vector model of the 8 int size sites: a wrap witness at every site, the wrap region is exactly offset + limit >= 2^31, and, in an abstract model of the group window, the cap is proved result-preserving and overflow-free; two fix mutants refuted.
• The repro was later minimized to the one test above.