TopGroups.merge throws NegativeArraySizeException when docOffset + docTopN overflows int

TopGroups.merge combines the second-pass grouping results of several shards, then skips docOffset documents per group and keeps docTopN. It adds the two ints without an overflow check before calling TopDocs.merge, so docOffset 1 with docTopN Integer.MAX_VALUE ("all after the first") throws NegativeArraySizeException: -2147483648 as soon as one group's shard results hold a document, for relevance and field order. docOffset 0, or docTopN Integer.MAX_VALUE - 1, works. It has been there since 3.3 (2011). Lucene's own GroupingSearch and TopGroupsCollectorManager fail earlier on such values (LFV-964), so the demonstrated reachable path is hand-written shard-merge code that bounds each shard's window itself.

• Two-pass grouping across shards: each shard runs the first pass, SearchGroup.merge picks the top groups, each shard runs the second pass for those groups, and TopGroups.merge combines the per-group documents. The grouping package leaves this to the caller: "Sharding is not directly supported, though is not too difficult, if you can merge the top groups and top documents per group yourself" (package-info.java:76-77).
• docOffset, docTopN: "which document to start from within each group (for pagination)" and "how many top documents to keep within each group" (TopGroups.java:128-129). No bound is documented, and merge checks neither.

1. For each group, merge calls TopDocs.merge(docOffset + docTopN, shardTopDocs) for relevance order (TopGroups.java:241) or TopDocs.merge(docSort, docOffset + docTopN, ...) for a field sort (:247). The sum is an int: 1 + 2147483647 is -2147483648.
2. TopDocs.merge(topN, ...) is merge(0, topN, ...) (TopDocs.java:203-204, :216-217). When the group's shard results hold at least one ScoreDoc, mergeAux sizes its result with new ScoreDoc[Math.min(size, availHitCount - start)] (TopDocs.java:313), and the negative size throws NegativeArraySizeException. A group whose shards return no ScoreDoc takes the empty branch (:310-311), but one group with a returned ScoreDoc fails the whole call.
3. The design itself handles "no limit": merge keeps the top hits from position 0, reads the group's best score from scoreDocs[0] (TopGroups.java:244-245) and drops the first docOffset itself (:250-260); core TopDocs.merge sizes by the available hits, not by topN. That is why docOffset 0 with Integer.MAX_VALUE works. Only the addition breaks.

Executed on main 645d4fe14a20, two shards with one document each (scores 2 and 1), relevance and field order alike: (0, 10) and (0, 2147483647) give 2 documents; (1, 10) and (1, 2147483646) give 1; (1, 2147483647) and (2, 2147483646) throw -2147483648; (2, 2147483647) throws -2147483647.

• Who is hit (the demonstrated reachable path). Hand-written shard merging, the recipe the package docs point to (package-info.java:76-77), that bounds each shard's second-pass window itself and passes docOffset >= 1 with docTopN > Integer.MAX_VALUE - docOffset (typically Integer.MAX_VALUE) to the merge. An independent reproducer built exactly that (two indexes, the first pass, SearchGroup.merge, TopGroupsCollectorManager per shard with the window at maxDoc, then TopGroups.merge), and it fails on main and 10.5.2.
• Who is not hit yet. TopGroupsCollectorManager.newCollector adds withinGroupOffset + maxDocsPerGroup itself (TopGroupsCollectorManager.java:107) before reduce calls merge (:116-122), so GroupingSearch and the manager fail first: IllegalArgumentException: numHits must be > 0 for (1, MAX), OutOfMemoryError for (0, MAX) with a field sort (that is LFV-964). If that sum is fixed and reduce still passes the caller's values on, this line becomes the first failure. Lucene's own sharded test passes the same window to each shard and to the merge (TestGrouping.java:1527, :1552-1557). Solr passes a finite docsPerGroup for group.limit < 0 since SOLR-4164; for group.limit >= 0 it passes the request's limit and offset to merge unchanged (read only; whether its shard requests fail earlier was not checked). A code search found no call in Elasticsearch or OpenSearch.
• Default or opt-in. No setting needed, but the call is an expert one: a static helper of an @lucene.experimental class (TopGroups.java:38).
• Effect. An exception instead of the merged groups whenever a group returns a document; no wrong or partial result. Workaround: docTopN = Integer.MAX_VALUE - docOffset, or a finite cap.
• Scale. None: 2 documents. Likelihood: rare; no report since 3.3 (2011).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected group g's second document, but merge(docOffset=1, docTopN=MAX_VALUE) threw java.lang.NegativeArraySizeException: -2147483648
       org.apache.lucene.search.grouping.TestTopGroupsMergeOffsetOverflow.testDocOffsetOneWithUnboundedDocTopN(TestTopGroupsMergeOffsetOverflow.java:54)
     Caused by: java.lang.NegativeArraySizeException: -2147483648
       org.apache.lucene.search.TopDocs.mergeAux(TopDocs.java:313)
       org.apache.lucene.search.TopDocs.merge(TopDocs.java:217)
       org.apache.lucene.search.TopDocs.merge(TopDocs.java:204)
       org.apache.lucene.search.grouping.TopGroups.merge(TopGroups.java:241)
       org.apache.lucene.search.grouping.TestTopGroupsMergeOffsetOverflow.testDocOffsetOneWithUnboundedDocTopN(TestTopGroupsMergeOffsetOverflow.java:46)

• Contract. TopGroups.java:128-129 give the two parameters' meaning and no bound; the documented preconditions are the same sorts and the same top groups on every shard (:113-115, :124-125). merge validates neither value.
• The method does not need a bound. It does not allocate by docTopN; core TopDocs.merge accepts topN = Integer.MAX_VALUE and sizes by the available hits (TopDocs.java:313).
• The exception is accidental: a JVM allocation failure inside core, not a defined rejection like the IllegalArgumentExceptions merge throws for mismatched shards (TopGroups.java:164-166, :202-210).
• Precedent. SOLR-3956 (type Bug, fixed 2013, commit 1ad382f3046): for grouped facets the committer chose Integer.MAX_VALUE as "unlimited" and changed getFacetEntries(offset, limit) "to not have int overflow if Integer.MAX_VALUE < offset+limit" (Chris M. Hostetter, JIRA comment of 2013-03-27). Main still tests getFacetEntries(1, Integer.MAX_VALUE) (TestTermGroupFacetCollectorManager.java:106, :135).
• Tests never tried it. Upstream tests merge with finite windows (TestTopGroups.java:174-181, TestGrouping.java:1552-1557).

The strongest case (the refute review, the intended-behaviour check): an unwritten precondition that the merge window equals the per-shard second-pass window, which is itself docOffset + docTopN (TopGroupsCollectorManager.java:107, TestGrouping.java:1527), so any pair that overflows in merge has already failed in the shard; TopGroups is @lucene.experimental; sharding is "not directly supported". The refute reviewer rated it moderate and, on that reading, judged the inputs outside the contract ((1, MAX) "is not a supported pagination request"), while noting the javadoc states no bound, "so this is not a conclusive contractual refutation".

Why it is weakened, though not conclusively disproved: nothing documents that precondition; TestTopGroups feeds merge hand-built results, so it is a standalone utility; the merge is well defined whenever each shard returned all its candidates (a window capped at maxDoc); the boundary pair shows the limit is the wrap; and the same module treated the same pattern as a bug. The works-as-designed reviewer: "experimental status is not an input-domain restriction". Against it: the standard collection recipe always computes one positive int window before the merge, and the request is extreme, though it uses the conventional "no limit" sentinel that the zero-offset call accepts. Verdict: a defect, low severity, narrow reach.

One fix was built and validated with gradle; two alternatives are compared below.

Compute docOffset + docTopN in long, cap it at Integer.MAX_VALUE, and pass that to both TopDocs.merge calls. The finder wrote this patch; the first verifier independently chose the same saturation semantics (inlined at both calls, without the local).

// docOffset + docTopN can exceed Integer.MAX_VALUE (e.g. docTopN = Integer.MAX_VALUE for "all
// remaining documents"): saturate instead of wrapping to a negative size.
final int docWindow = (int) Math.min((long) docOffset + docTopN, Integer.MAX_VALUE);
...
mergedTopDocs = TopDocs.merge(docWindow, shardTopDocs);
...
mergedTopDocs = TopDocs.merge(docSort, docWindow, (TopFieldDocs[]) shardTopDocs);

Trade-offs

Keeps the method's design (merge from position 0, take maxScore from scoreDocs[0], slice locally) and every non-wrapping result. Nothing is lost by saturating: a group's merged hits are one Java array. One long addition per call, before the per-group loop. It repairs the positive overflow only and is not input validation: for every pair whose sum is at most Integer.MAX_VALUE, negative values included, it passes exactly what main passes today, and merge still neither documents nor checks docOffset >= 0 and docTopN >= 0 (option C's checks would add that).

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, JDK 25.0.3: stock, the repro fails (assertions on and off); with the fix it passes, with TestTopGroups; all of :lucene:grouping:test (15 suites, 57 tests) passes with assertions on and off; :lucene:grouping:check -x test passes.
• Standalone runner with the class shadowed: the repro passes; on main a9779ef3c193 the finder's 2 tests, the first verifier's 5 tests and the audit's 3 tests passed with the same fix.

Let core skip the offset and drop the local slice (TopGroups.java:250-260).

Trade-offs

Core has the same addition (int requestedResultWindow = start + size, TopDocs.java:314, LFV-262: a page of null ScoreDocs), so it needs that fix first. It would also break maxScore for relevance order: TopGroups.java:244-245 reads the best score from scoreDocs[0], which would then be the first document after the offset.

Validate the two values and use Math.addExact, or throw IllegalArgumentException.

Trade-offs

A clean message, but it still refuses the "all remaining" request that docOffset 0 and core TopDocs.merge accept.

Method: semi-formal. A hunt unit on the grouping module's int offset + limit sums (GroupingSearch, FirstPassGroupingCollectorManager, TopGroups.merge) wrote a z3 model of each addition site. Query Q5 is satisfiable at docOffset 1, docTopN Integer.MAX_VALUE (sum -2^31); Q8 shows the wrap region is exactly docOffset + docTopN >= 2^31. The site had been noted, but never run, by an earlier unit that worked on core TopDocs.merge (LFV-262).

The unit then drove the documented sharding recipe on the stock main jars with a boundary grid of (docOffset, docTopN) up to Integer.MAX_VALUE, compared with a finite docTopN: (0, MAX) and (1, MAX-1) matched; (1, MAX), (2, MAX-1) and (2, MAX) threw, for relevance and field order. It reduced that to a public-API JUnit (2026-10-28), later minimised to the one-test repro.