TopSuggestDocs.merge sizes its queue by topN before it reads any shard, so merge(Integer.MAX_VALUE, shards) throws even for a handful of hits

TopSuggestDocs.merge(int topN, TopSuggestDocs[] shardHits) combines per-shard document-suggester results into one top list; its javadoc calls it "Analogous to TopDocs#merge(int, TopDocs[])" and sets no bound on topN. It creates its PriorityQueue with topN slots before it looks at any shard, and PriorityQueue pre-allocates maxSize + 1 slots and refuses maxSize >= 2147483631 (on the tested JVM). So merge(Integer.MAX_VALUE, shards) ("keep them all") throws IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647 for two one-hit shards and even for a single empty shard (5.1.0 to 7.4.0: NegativeArraySizeException), while TopDocs.merge(Integer.MAX_VALUE, ...) returns the hits, because it sizes its queue by the number of shards and its result by the hits it received. Below the limit a large topN can succeed but allocates topN + 1 slots whatever the hit count (about 400 MB at topN = 10^8). The method has been like this since it was added in 5.1.0 (2015); Lucene itself never calls it; this bug never returns wrong results (merge either throws or over-allocates) and the failure is loud. If the shards were collected with the same huge n, the per-shard collector throws first, so only a merge topN above the per-shard n, or hand-built shard results, reach it.

• Contract. TopSuggestDocs.java:107-113: "Returns a new TopSuggestDocs, containing topN results across the provided TopSuggestDocs, sorting by score. Each TopSuggestDocs instance must be sorted. Analogous to TopDocs#merge(int, TopDocs[]) for TopSuggestDocs". No bound on topN, no validation. The class is @lucene.experimental (:30).
• The pre-size. PriorityQueue.usingComparator(topN, SUGGEST_SCORE_DOC_COMPARATOR) (TopSuggestDocs.java:116-117), before the loop that inserts each shard's hits (:118-124).
• The queue. PriorityQueue "pre-allocates an array of length maxSize+1" (PriorityQueue.java:34-35, @lucene.internal at :39) and, "to prevent confusing OOME", throws IllegalArgumentException for maxSize >= ArrayUtil.MAX_ARRAY_LENGTH (:121-124), which is Integer.MAX_VALUE minus the JVM's array header size (ArrayUtil.java:33-34): 2147483631 with compressed object pointers, as on the tested JVM, 2147483623 without; otherwise it allocates new Object[maxSize + 1] (:129, :134).
• The analogue. TopDocs.merge(int topN, TopDocs[]) (TopDocs.java:203) sizes its queue by the number of shards (:283, :286), counts the hits it received (:303) and sizes its result Math.min(size, availHitCount - start) (:313): Michael McCandless's rule from commit 4e92af15376 (2011), "when merging TopDocs, the most hits we can produce is the sum of how many hits we actually received".
• The per-shard side. SuggestIndexSearcher.suggest(query, n, ...) builds new TopSuggestDocsCollector(n, ...) (SuggestIndexSearcher.java:52), which also creates PriorityQueue.usingComparator(num, ...) (TopSuggestDocsCollector.java:81). A shard collected with n = Integer.MAX_VALUE therefore fails before merge runs.
• No caller. Nothing in Lucene calls merge, tests included.

Bytes are allocated by the calling thread for one merge after a warm-up (not retained heap), with 4-byte object references (compressed object pointers); with 8-byte references the topN + 1 slots take about twice as much. With -Xmx256m, merge(100_000_000, 3 hits) ends in OutOfMemoryError at TopSuggestDocs.java:117; the out-of-memory part depends on the heap, the topN + 1 slots do not.

• Who is hit. Applications that keep document suggestions in several indexes and combine the per-shard TopSuggestDocs themselves, passing a merge topN larger than the n their shards were collected with (an independent check: real shards collected with n = 5, merged with Integer.MAX_VALUE, throw), or that merge hand-built TopSuggestDocs. "All" means every hit in the shards passed in: shards collected with suggest(query, 5, false) hold at most 5 each.
• Reach caveat. In the plain reduce pattern (per-shard TopSuggestDocsCollector(n), then merge(n, ...) with the same n), Integer.MAX_VALUE already fails in the collector, so merge is never reached. The one public caller found, nrtsearch's MyTopSuggestDocsCollectorManager.reduce, uses that symmetric pattern.
• Default vs opt-in. Calling merge needs no setting, but the API is @lucene.experimental, and real TopSuggestDocs come from a SuggestField, whose index needs a completion postings format in the codec.
• What goes wrong. On the tested JVM every merge with topN >= 2147483631 throws, even with no hits, and merge(Integer.MAX_VALUE, ...) threw on every release tested since 5.1.0. The message names maxSize, a parameter of an internal class rather than merge's topN, and the internal array limit 2147483631; only its got: value is the caller's. A legal large topN costs topN + 1 reference slots per call, whatever the hit count.
• Scale. None: two one-hit shards, or one empty shard, are enough.
• Workaround. Cap topN at the total number of shard hits before calling merge; this works on stock code.
• How likely. Rare: loud and immediate, no caller in Lucene, no earlier public report found.

Copy the test into lucene/suggest/src/test/org/apache/lucene/search/suggest/document/ of a Lucene checkout and run the command below. It sits in that package only because the module's tests live there; it uses public constructors only. The catch only turns the exception into an assertion failure that states expected vs actual, with the original as the cause.

Seeds DEADBEEF, 5EED1234 and 1002FACE each fail on main with -ea and with -da, and C0FFEE42 with -da. Two lines are longer than 100 characters: before committing the test in a PR, run ./gradlew tidy, which rewraps the comment and the throw line and changes nothing else.

cp $REPRO/TestTopSuggestDocsMergeAllHits.java $LUCENE/lucene/suggest/src/test/org/apache/lucene/search/suggest/document/
cd $LUCENE && ./gradlew :lucene:suggest:test --tests org.apache.lucene.search.suggest.document.TestTopSuggestDocsMergeAllHits -Ptests.seed=DEADBEEF

# gradle, fresh worktree at main 645d4fe14a20, -Ptests.seed=DEADBEEF (module prefixes trimmed)
TestTopSuggestDocsMergeAllHits > testMergeWithTopNMaxValue FAILED
    java.lang.AssertionError: expected the 2 merged suggestions for topN = Integer.MAX_VALUE, but got java.lang.IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:E1AAF9DB920CFC30]:0)
        at org.apache.lucene.search.suggest.document.TestTopSuggestDocsMergeAllHits.testMergeWithTopNMaxValue(TestTopSuggestDocsMergeAllHits.java:37)

        Caused by:
        java.lang.IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
            at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:123)
            at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:80)
            at org.apache.lucene.util.PriorityQueue.usingComparator(PriorityQueue.java:63)
            at org.apache.lucene.search.suggest.document.TopSuggestDocs.merge(TopSuggestDocs.java:117)
            at org.apache.lucene.search.suggest.document.TestTopSuggestDocsMergeAllHits.testMergeWithTopNMaxValue(TestTopSuggestDocsMergeAllHits.java:34)

# standalone runner (tests_run=1 failures=1 each)
main 645d4fe14a20  -ea and -da, seeds DEADBEEF, 5EED1234, 1002FACE   the same exception at TopSuggestDocs.java:117
10.6.0, 10.5.2     -da, seed DEADBEEF                                 the same exception, via SuggestScoreDocPriorityQueue.<init> at TopSuggestDocs.java:105

• Contract. TopSuggestDocs.java:107-113 promises "topN results across the provided TopSuggestDocs" and states only a sortedness precondition. The pre-allocation note sits on the @lucene.internal queue (PriorityQueue.java:34-35, :39); core documents pre-allocation where it applies, on its collector managers (TopScoreDocCollectorManager.java:45, :83), and merge has no such note.
• Lucene's precedents put the cap in the code that sizes the queue. LUCENE-504 (#1581): Nadav Har'El, "It is the caller's duty to create a priority queue with a sensible size", with Mark Miller agreeing; Uwe Schindler fixed FuzzyQuery, the caller. LUCENE-2119: IndexSearcher caps n at maxDoc (IndexSearcher.java:721, :729). SOLR-5661: Michael McCandless, "somewhere it should limit the size it passes to the PQ constructor?".
• The guard is not a public bound. It exists "to prevent confusing OOME" (PriorityQueue.java:122); it came with SOLR-5661 (2014) and was made to work for Integer.MAX_VALUE in 4c646dab3fb (2018, Mike McCandless, "thanks rsaavedraf", who diagnosed the overflow in lucene-solr PR #411).
• History. No issue, test, commit message or review comment says the pre-size is intended. In the LUCENE-6339 review (#7400) Michael McCandless proposed a TopSuggestDocs.merge method; none of the 35 comments discusses its queue size. #14817 (2025) only swapped the queue class.

The strongest case for "works as designed" (first raised by reviewers from another model family, OpenAI GPT-5.6, then developed by an intended-behaviour devil's advocate): "Analogous" may describe only the purpose, since merge also differs from TopDocs.merge in totalHits, tie-breaking and the missing start; the document suggester pre-sizes by n everywhere, and TopSuggestDocsCollector rejects Integer.MAX_VALUE the same way; Lucene's answer to Integer.MAX_VALUE queues reached through Solr's shard merge was exactly this clean exception (SOLR-5661), which Shalin Shekhar Mangar judged enough there; Cassandra Targett closed SOLR-5674, whose item 3 is this fix idea for Solr's code, as "a set of improvement requests"; and Adrien Grand's LUCENE-8542 comment that Lucene targets top-k "for some reasonable value of k" predicts the same answer. The queue's own refusal is also tested on purpose (TestPriorityQueue.testMaxIntSize, TestPriorityQueue.java:378-382), though that test says nothing about how merge sizes its queue. The cross-model fact check (OpenAI GPT-5.6, reading only) went further and judged the bug noise: an absurd capacity request that fails loudly, on an experimental API with no caller in Lucene and a trivial caller-side workaround.

Why it fails: those decisions concern Solr's code or core collectors, not this method; the guard (and its test) protects an internal class, and Lucene's own precedents fix the code that sizes the queue, here merge; the analogue it names already follows the "at most what you received" rule; and a collector must size before it sees matches, while merge already holds them all. The argument bears on reach and on the likely reply ("improvement, not a bug", "document the limit" or "remove merge"), hence priority Low.

Three options; A was validated with gradle in a fresh scratch worktree at 645d4fe14a20, B and C were not executed. Credit where the fix is presented: the sizing rule is Michael McCandless's (TopDocs.merge, 4e92af15376, 2011); Raintung Li first proposed it for Solr's merge (SOLR-5674 item 3, 2014: "Create the PriorityQueue after check all shard return sizes"); Nik Everett diagnosed and fixed the same failure, with the same message, in an Elasticsearch reduce step (elastic/elasticsearch#76492, reported by deathcave; fix #76578). Not part of this report: TopSuggestDocsCollector also pre-sizes by num (TopSuggestDocsCollector.java:81), but a collector must size before it sees matches, so it needs a different fix.

In TopSuggestDocs.merge (:116-117) sum scoreLookupDocs().length over the shards in a long and pass (int) Math.min(topN, availHitCount) to PriorityQueue.usingComparator. +8/-1 lines in one file; two of the added lines are a comment.

public static TopSuggestDocs merge(int topN, TopSuggestDocs[] shardHits) {
  long availHitCount = 0;
  for (TopSuggestDocs shardHit : shardHits) {
    availHitCount += shardHit.scoreLookupDocs().length;
  }
  // The result holds at most the hits received, as in TopDocs#merge, so do not size the queue
  // by topN alone: it may be far larger, e.g. Integer.MAX_VALUE for "all hits"
  int queueSize = (int) Math.min(topN, availHitCount);
  PriorityQueue<SuggestScoreDoc> priorityQueue =
      PriorityQueue.usingComparator(queueSize, SUGGEST_SCORE_DOC_COMPARATOR);

Trade-offs

• No change in results. When topN is at most the hit count the queue size is topN, as before; when it is larger, a queue of exactly the hit count never rejects a hit, and neither did the larger queue.
• For non-null input, topN = 0 and negative topN behave as before: 0 returns EMPTY, a negative value still throws from PriorityQueue. No hits gives a size-0 queue, which PriorityQueue accepts (PriorityQueue.java:117-119).
• Null input still throws, but the exception can change: the counting loop reads every shard before the queue exists, so merge(-1, shards with a null entry), for example, throws NullPointerException where it threw IllegalArgumentException before.
• Cost: one extra pass over the shard array, reading each array's length. The long sum cannot overflow.

What the checks validated

• Fresh scratch worktree at 645d4fe14a20, JDK 25.0.3, gradle with -Ptests.seed=DEADBEEF (2026-10-30): without the fix the minimal test fails; with fix.diff applied, :lucene:suggest:test passes: 32 suites, 237 tests, 7 skipped, 0 failures, including the minimal test and the document package's TestSuggestField (20), TestContextQuery (15), TestPrefixCompletionQuery (9), TestContextSuggestField (6), TestRegexCompletionQuery (5) and TestFuzzyCompletionQuery (3).
• :lucene:suggest:checkGoogleJavaFormat passes for the fix (it flagged only the copied test's two long lines; after tidy rewrapped them, the fix byte-identical, the check and the test pass).
• The standalone runner passes the test with the fixed class shadowed, -ea and -da; an allocation test (3 hits, topN = 100,000,000, -da) drops from 400,001,208 to 1,216 bytes.
• The finder's earlier version of the same idea passed its 3-test JUnit (including the empty-shard case) and the minimal test with -ea and -da.
• An edge-case probe gives the same results with and without the fix for every legal topN (including 4 hits with a three-way score tie, at topN 1 to 100), EMPTY for topN 0 and for an empty merge, and IllegalArgumentException for topN -1. These cases stay out of the regression test, which is the one minimal test.

Collect the hits and sort them, or use a growable heap. Not executed.

Trade-offs

Same results, more code, and no gain over A: merge already knows how many hits it holds.

Add a javadoc note or a clear IllegalArgumentException for large topN. Not executed.

Trade-offs

Keeps the topN + 1 allocation for large legal values and makes the "keep them all" call a documented error, although TopDocs.merge handles it (its result holds at most the hits it received, TopDocs.java:313). Not recommended.

Method: executed differential testing (no formal model), in a hunt round on numeric limits of public API arguments, on the stock main jars.

• First signal. A planning probe generalised two earlier findings in the grouping module (a shard-merge helper and a grouping search, both pre-sized by an unbounded limit) to top-N APIs outside grouping. On main a9779ef3c193, 10.5.2 and the 10.6 release candidate (2026-10-28) it printed FAIL TopSuggestDocs.merge(MAX, {empty shard}) with the control TopDocs.merge(MAX, {empty shard}) -> 0.
• Hunt. A finder re-ran it as a differential check against TopDocs.merge, wrote a 3-test JUnit with that control (2 of 3 fail), located the pre-size by reading the code and wrote the fix sketch.
• Checks. Independent reproducers, an intended-behaviour devil's advocate and an adjudication confirmed it; the single test shown here is the audited reduction.