AnalyzingInfixSuggester, WFSTCompletionLookup and FreeTextSuggester size their results by num, so lookup(key, false, Integer.MAX_VALUE) throws even when one entry matches

Lookup.lookup(key, onlyMorePopular, num) documents num only as the "maximum number of results to return". Three implementations allocate storage for num results before they know how many entries match: AnalyzingInfixSuggester builds new TopFieldCollectorManager(SORT, num, null, 1), whose collectors pre-allocate num slots, and never caps num at maxDoc the way IndexSearcher.search(Query, int, Sort) does for the same manager; WFSTCompletionLookup and FreeTextSuggester call new ArrayList<>(num). So lookup("luc", false, Integer.MAX_VALUE) on a 1-entry suggester throws OutOfMemoryError: Requested array size exceeds VM limit on main (on 7.5.0 to 10.6.0 AnalyzingInfixSuggester throws IllegalArgumentException from PriorityQueue's size guard instead; older releases throw other errors), while TSTLookup and FSTCompletionLookup return the match. A large num below the array limit works as long as the heap can hold num slots and costs memory in proportion to num: 114.5 MiB and 38.1 MiB per lookup at num = 10,000,000 for one match. Each allocation came with its class (3.6.0, 4.4.0, 4.6.0). Default constructors; the failure is loud, and every recorded successful call returned the right results.

• Contract. Lookup.java:190: "@param num maximum number of results to return" (also :206, :219). No bound, no validation. Lookup and the three classes are @lucene.experimental.
• AnalyzingInfixSuggester. TopFieldCollectorManager c = new TopFieldCollectorManager(SORT, num, null, 1); (AnalyzingInfixSuggester.java:739), then searcher.search(finalQuery, c) (:741). The manager's javadoc: its collectors "pre-allocate a full array of length numHits" (TopFieldCollectorManager.java:44-45). On main FieldValueHitQueue.create builds the weight sort's comparator first (FieldValueHitQueue.java:146), and LongComparator allocates new long[numHits] (LongComparator.java:38), which the JVM refuses for Integer.MAX_VALUE elements. On 7.5.0 to 10.6.0 the PriorityQueue constructor ran first and rejected the size ("Throw exception to prevent confusing OOME"), hence the IllegalArgumentException; the order changed with #14873 (LFV-993).
• The cap the suggester skips. IndexSearcher.search(Query query, int n, Sort sort) (IndexSearcher.java:677) reaches final int limit = Math.max(1, reader.maxDoc()); and final int cappedNumHits = Math.min(numHits, limit); (:721, :729) before it builds the same manager (:733). On a 1-document index, searcher.search(q, Integer.MAX_VALUE, sort) returns the 1 hit, while searcher.search(q, new TopFieldCollectorManager(sort, Integer.MAX_VALUE, null, 1)) throws the same OutOfMemoryError (executed). The suggester's own comment just below the search call still shows the capped call as the "slower way": // hits = searcher.search(query, null, num, SORT); (AnalyzingInfixSuggester.java:743-744).
• WFSTCompletionLookup. An unbuilt suggester (:162-164) or an unmatched prefix (:179-181) returns an empty list; otherwise List<LookupResult> results = new ArrayList<>(num); (WFSTCompletionLookup.java:183). num also goes to Util.shortestPaths (:198), which pre-sizes nothing by it: its TopNSearcher keeps candidate paths in a TreeSet and results in an unsized list (Util.java:205, :296), which grow with the paths found, up to num (:215, :247-248, :467).
• FreeTextSuggester. List<LookupResult> results = new ArrayList<>(num); (FreeTextSuggester.java:519), before any n-gram model runs.
• The siblings that work. TSTLookup (with onlyMorePopular = false) loops to maxCnt = Math.min(num, list.size()) (TSTLookup.java:193); FSTCompletion streams lookup(key).limit(num).toList() (FSTCompletion.java:204-208).

38.1 MiB is almost all Object[num], 10^7 four-byte references (compressed object pointers on). 114.5 MiB is dominated by the comparator's long[num] (76.3 MiB) and the queue's Object[num + 1] (38.1 MiB, PriorityQueue.java:129, :134), plus small per-call overhead (27,000 bytes in all at num = 1000). These are measurements on OpenJDK 25.0.3, Linux aarch64, not portable constants. With 8 entries (first call at each num, no warm-up) the figures are within 0.1 MiB (114.5, 38.2, 38.2 MiB), and BlendedInfixSuggester, which asks its parent for num x 10, allocates 1144.4 MiB. How far below the limit a lookup still works depends on the heap: with -Xmx1g a probe got OutOfMemoryError: Java heap space from AnalyzingInfixSuggester at num = 10^8 and from all three at 429,496,730.

• Who is hit. Callers of lookup on the three suggesters, and on BlendedInfixSuggester, which inherits the AnalyzingInfixSuggester collector, multiplied by numFactor (default 10).
• Default vs opt-in. Default constructors and settings; nothing to opt into.
• What goes wrong. On a built suggester, Integer.MAX_VALUE fails on every call to AnalyzingInfixSuggester and FreeTextSuggester, even when nothing matches, and on every call to WFSTCompletionLookup whose prefix matches an entry. The error is an OutOfMemoryError, which callers often do not catch, or, for AnalyzingInfixSuggester on 7.5.0 to 10.6.0, an IllegalArgumentException that blames an argument the javadoc allows. A num in the millions returns the right answer but allocates num slots on every call; concurrent calls multiply that.
• Scale. None needed: one entry is enough.
• Workaround. Pass a finite count. Capping num at getCount() is safe for AnalyzingInfixSuggester and WFSTCompletionLookup (at most one suggestion per entry) but not for FreeTextSuggester: the single entry lucene lucid lucky gives getCount() = 1, while lookup("luc", false, 1000) returns 3 suggestions and lookup("luc", false, getCount()) returns 1 (executed).
• How likely. Rare: Integer.MAX_VALUE fails on the first call, so developers meet it in testing and pick a finite count; no public report was found.

Copy the test into lucene/suggest/src/test/org/apache/lucene/search/suggest/ of a Lucene checkout and run the command below. On main the first assertion (AnalyzingInfixSuggester) already throws, so the other two lines are not reached; copies holding only the WFSTCompletionLookup line or only the FreeTextSuggester line fail the same way, at WFSTCompletionLookup.java:183 and FreeTextSuggester.java:519.

LuceneTestCase still randomizes codec, locale and the like; seeds DEADBEEF, 5EED1004 and 1004FACE all fail. The error is the JVM's array-length limit, not heap exhaustion: it is the same at -Xmx64m, 2g and 40g.

cp $REPRO/TestLookupMaxNum.java $LUCENE/lucene/suggest/src/test/org/apache/lucene/search/suggest/
cd $LUCENE && ./gradlew :lucene:suggest:test --tests org.apache.lucene.search.suggest.TestLookupMaxNum -Ptests.seed=DEADBEEF

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF (module prefixes and two overload frames trimmed)
TestLookupMaxNum > testLookupWithMaxNum FAILED
    java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at org.apache.lucene.search.comparators.LongComparator.<init>(LongComparator.java:38)
        at org.apache.lucene.search.SortField.getComparator(SortField.java:503)
        at org.apache.lucene.search.FieldValueHitQueue.create(FieldValueHitQueue.java:146)
        at org.apache.lucene.search.TopFieldCollectorManager.newCollector(TopFieldCollectorManager.java:114)
        at org.apache.lucene.search.TopFieldCollectorManager.newCollector(TopFieldCollectorManager.java:29)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:752)
        at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.lookup(AnalyzingInfixSuggester.java:741)
        ...
        at org.apache.lucene.search.suggest.Lookup.lookup(Lookup.java:195)
        at org.apache.lucene.search.suggest.TestLookupMaxNum.lookupAll(TestLookupMaxNum.java:45)
        at org.apache.lucene.search.suggest.TestLookupMaxNum.testLookupWithMaxNum(TestLookupMaxNum.java:37)

# standalone runner (tests_run=1 failures=1 each)
main 645d4fe14a20  -ea seed DEADBEEF, -da seed 5EED1004   OutOfMemoryError (as above)
10.5.2             -da seed DEADBEEF                      IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
10.6.0             -ea seed DEADBEEF                      the same IllegalArgumentException

• Contract. Lookup.java:190: num is a maximum; nothing calls it a buffer size or says it must stay near the number of entries.
• Siblings. TSTLookup (with onlyMorePopular = false), FSTCompletionLookup, AnalyzingSuggester and FuzzySuggester return the matches for the same call (executed).
• Tests. TestFSTCompletion.testFullMatchList passes Integer.MAX_VALUE to FSTCompletion.lookup and expects every match (TestFSTCompletion.java:157, :165). That is the API under FSTCompletionLookup, not Lookup.lookup itself.
• LUCENE-2119 (#3195, 2009), opened by Michael McCandless: "Note that this is a nonsense value to pass in, since our PQ impl allocates the array up front. ... Better, maybe: in IndexSearcher, if that n is ever > maxDoc(), set it to maxDoc(). This trips users up fairly often because they assume our PQ doesn't statically pre-allocate (a reasonable assumption...)." Michael McCandless's commit ae72952fa41 added the cap still in IndexSearcher. CHANGES credits the report to Paul Taylor.
• SOLR-5661 (2014), reported with a patch by Raintung Li and committed by Michael McCandless, added only the PriorityQueue size guard. On it Michael McCandless noted that "Lucene's IndexSearcher tries to prevent this, when you ask for Integer.MAX_VALUE as the topN hits to the search method, it drops that to the maxDoc for the reader", and asked of the Solr caller: "I'm not sure how to fix Solr here: somewhere it should limit the size it passes to the PQ constructor?" AnalyzingInfixSuggester is such a caller: it builds the manager itself.
• SOLR-2378 (2011): the original FSTLookup capped its pre-size with Math.min(10, num) (Dawid Weiss).

The strongest case for "works as designed" (raised by reviewers from another model family and by an intended-behaviour devil's advocate): Lucene's top-N structures pre-allocate and TopFieldCollectorManager documents it; Lookup and the three classes are @lucene.experimental; a comment in the document suggester assumes "num is smallish" (TopSuggestDocsCollector.java:149); and LUCENE-2119 itself calls Integer.MAX_VALUE "a nonsense value to pass in".

Why it fails: the manager's note binds whoever builds the manager, here the suggester, and the caller of lookup never sees it; the same LUCENE-2119 text calls the user's assumption reasonable and fixed the caller, not the user; @lucene.experimental is about API stability, not about failing on allowed input; the sibling suggesters and the module's own test treat Integer.MAX_VALUE as "all". The argument bears on priority, not on correctness.

Three options; A was validated with gradle in a fresh scratch worktree at 645d4fe14a20, B and C were not executed. None fixes LFV-998 (BlendedInfixSuggester computes num * numFactor in int), which needs its own fix for Integer.MAX_VALUE to work through BlendedInfixSuggester. Same pattern, not part of this report: TSTLookup with onlyMorePopular = true sizes a PriorityQueue by num (TSTLookup.java:195-196) although maxCnt = Math.min(num, list.size()) is computed on :193.

In AnalyzingInfixSuggester.lookup (:739) pass Math.min(num, Math.max(1, searcher.getIndexReader().maxDoc())) to TopFieldCollectorManager, and keep passing the caller's num to createResults (:745). In WFSTCompletionLookup.lookup (:183) and FreeTextSuggester.lookup (:519) create the list with new ArrayList<>(). +5/-3 lines in three files, one added line a comment.

// AnalyzingInfixSuggester.lookup
// The collector pre-allocates numHits slots: cap it at maxDoc, as IndexSearcher.search does
int numHits = Math.min(num, Math.max(1, searcher.getIndexReader().maxDoc()));
TopFieldCollectorManager c = new TopFieldCollectorManager(SORT, numHits, null, 1);

// WFSTCompletionLookup.lookup and FreeTextSuggester.lookup
List<LookupResult> results = new ArrayList<>(); // was: new ArrayList<>(num)

Trade-offs

• No change in results: the cap only bounds the collector by what the index can return, and createResults still gets the caller's num, which BlendedInfixSuggester relies on.
• A result list for many matches grows as it fills instead of being allocated once; the runtime effect was not measured.
• Does not fix LFV-998: BlendedInfixSuggester at Integer.MAX_VALUE still throws IllegalArgumentException: numHits must be > 0; at 10,000,000 its allocation drops from 1144 MiB to 0 MiB.
• Optional hardening (the finder's extra hunk): with the pre-size gone, FreeTextSuggester reaches num + seen.size() (:608) with num = Integer.MAX_VALUE, which wraps in int once seen is not empty; the finder saturates it with (int) Math.min((long) num + seen.size(), Integer.MAX_VALUE). One scratch two-token lookup gave the same results without it (gradle seed DEADBEEF, so assertions off), and so did an assertions-on run of the key lucene se (three bigrams, 3 results at 1000 and Integer.MAX_VALUE); that covers only these cases, so keep the saturation or justify leaving it out.

What the checks validated

• Fresh scratch worktree at 645d4fe14a20, JDK 25.0.3, gradle with -Ptests.seed=DEADBEEF (2026-10-30; this seed runs with assertions off, and a re-run of both steps with -Ptests.asserts=true gave the same results): without the fix the minimal test fails with OutOfMemoryError at AnalyzingInfixSuggester.java:741; with fix.diff applied, :lucene:suggest:checkGoogleJavaFormat and :lucene:suggest:test pass: 32 suites, 237 tests, 7 skipped, 0 failures, including TestLookupMaxNum, TestAnalyzingInfixSuggester (23), TestBlendedInfixSuggester (9), TestFreeTextSuggester (11, 1 skipped) and TestWFSTCompletion (7).
• With the three fixed files shadowed, the standalone runner passes the test with -ea and with -da, and allocation probes on 8 entries and on 1 entry return all matches at num = 10,000,000 and Integer.MAX_VALUE with 0.0 MiB allocated (BlendedInfixSuggester at Integer.MAX_VALUE still throws: LFV-998).
• The finder's version, which also has the FreeTextSuggester saturation hunk, passed the finder's JUnit 4 of 4 and the independent reproducer's 8 of 8.

Replace the hand-built manager with searcher.search(finalQuery, num, SORT), the call the comment at :744 names, which caps num at maxDoc; plus the two list changes of option A. Not executed.

Trade-offs

Reuses IndexSearcher's cap, but that path counts hits up to TOTAL_HITS_THRESHOLD (1000) instead of the suggester's 1 before it can skip non-competitive documents, so it may be slower on large suggesters (not measured), for no gain here.

Add a javadoc note to Lookup.lookup that implementations may allocate num slots up front, or throw IllegalArgumentException above some bound. Not executed.

Trade-offs

Turns the "all suggestions" idiom that TSTLookup and FSTCompletionLookup support into a documented error and keeps the memory cost of a large num. Not recommended.

Method: executed differential testing (no formal model), in a hunt round on numeric limits of public API arguments, on the stock main jars.

• First signal. A planning probe called lookup("luc", false, num) on 8-entry suggesters of the eight Lookup classes of the module in nine configurations, -Xmx1g: at Integer.MAX_VALUE the three threw OutOfMemoryError and BlendedInfixSuggester an IllegalArgumentException (its own int overflow, LFV-998); at num = 10^8 they allocated 381 MiB (WFST, FreeText) or ran out of heap after 762 MiB (AnalyzingInfix), where num = 1000 returns 8; TSTLookup, FSTCompletionLookup, AnalyzingSuggester and FuzzySuggester passed.
• Sweep. The finder widened it to 11 configurations, each with 28 num values chosen with an SMT model of the arithmetic on num plus 7 prefix checks: 385 checks, against the small-num result and a 64 MiB allocation bound. Failures came only from these three sites, from BlendedInfixSuggester (LFV-998) and from one AnalyzingSuggester value (a separate overflow, LFV-1003).
• Reduction. Code reading located the allocations; the finding was reduced to a public-API JUnit test, later to the single test shown here.