BlendedInfixSuggester: num x numFactor overflows an int, so asking for all suggestions throws and some large num values return fewer

BlendedInfixSuggester over-fetches: to return num suggestions it asks its parent AnalyzingInfixSuggester for num * numFactor hits (numFactor is 10 by default), re-scores them by the position of the matched word, and divides the count back ("we reduce the num to the one initially requested", int actualNum = num / numFactor). The multiplication is a plain int product (BlendedInfixSuggester.java:199 on main 645d4fe14a20), so for num > Integer.MAX_VALUE / numFactor (214,748,364 for the default factor) it wraps. lookup("luc", false, Integer.MAX_VALUE), a natural way to ask for every suggestion, then fails on any built suggester (executed with one entry) with IllegalArgumentException: numHits must be > 0, a message about an argument the caller never passed. At other values the wrapped count is small and positive and the lookup silently returns too few results: with 8 matching entries, num = 429,496,731 returns 1, while num = 1000 returns all 8; num = 429,496,730 throws NoSuchElementException. On stock code the same huge values that do not wrap fail with OutOfMemoryError in the parent's collector pre-size (the sibling bug LFV-1004), so the short list is the correctness impact (at Integer.MAX_VALUE stock code fails either way) and returning every match needs both fixes: a saturating long product here and a collector cap at maxDoc there, validated together. The defect is as old as the class (LUCENE-5354, 4.7.0, 2014) and is in every release up to 10.6.0 and main.

1. Contract. Lookup.lookup(key, onlyMorePopular, num): "@param num maximum number of results to return" (Lookup.java:190; also :206, :219). No bound and no check. The class is @lucene.experimental (BlendedInfixSuggester.java:54); its javadoc says "it increases the number of elements searched and applies the ponderation after" (:49-52). DEFAULT_NUM_FACTOR = 10 (:64, set at :90); the two longer constructors take numFactor unchecked (:114, :152).
2. The product, once, in int. Three lookup overloads pass num on unchanged ("Don't * numFactor here since we do it down below, once, in the call chain", :162, :174, :186). The last one multiplies: "We need to do num * numFactor here only because it is the last call in the lookup chain" and return super.lookup(key, contextQuery, num * numFactor, allTermsRequired, doHighlight); (:198-199). Java int arithmetic wraps modulo 2^32: 2,147,483,647 x 10 = 21,474,836,470 is -10; 429,496,731 x 10 = 4,294,967,310 = 2^32 + 14 is 14.
3. Negative count. The parent builds new TopFieldCollectorManager(SORT, num, null, 1) with the wrapped value (AnalyzingInfixSuggester.java:739), which throws for numHits <= 0 (TopFieldCollectorManager.java:62-65).
4. Small positive count. The search runs and the parent hands the same wrapped value to createResults (AnalyzingInfixSuggester.java:745). The override divides it back: "we reduce the num to the one initially requested", int actualNum = num / numFactor; (BlendedInfixSuggester.java:226-227), which restores the caller's num only when the product did not wrap. boundedTreeAdd keeps at most actualNum results (:281, :294-306); with actualNum = 0 the check results.size() >= num is true on the empty set and results.first() (:297-298) throws NoSuchElementException.
5. Never more, never outside the filter. A wrapped product is always below the true one, so the list can only get shorter. The context filter is part of the query (AnalyzingInfixSuggester.java:692-715), independent of num.

With numFactor = 1000 it starts 100 times earlier: 2,147,484 throws the IllegalArgumentException, 4,294,968 the NoSuchElementException, and 4,294,969 returns [lucas]. Over every positive int (numFactor 10, 8 matches), 1,073,741,824 values (half of them) throw the IllegalArgumentException, 4 the NoSuchElementException, and 28 return a short list; with numFactor 1000, 3,493 values return a short list. By the same arithmetic the short-list windows widen with more matches (the count above fixed 8).

On stock code the parent pre-sizes its collector by the fetch count (LFV-1004, suggester-lookup-presizes-by-num), so huge values that do not wrap fail too: with a 2 GB heap, num = 214,748,364 (product 2,147,483,640) throws OutOfMemoryError: Java heap space in the parent and in the blended class alike. The wrap turns that loud failure into a misleading exception or a short list. Each fix alone leaves Integer.MAX_VALUE failing; together they return every match.

• Who is hit. Callers of lookup on a BlendedInfixSuggester with num above Integer.MAX_VALUE / numFactor: 214,748,364 for the default factor, 2,147,483 for numFactor 1000, 3,579,139 for numFactor 600. For example Integer.MAX_VALUE as "all suggestions" (the module's own TestFSTCompletion uses it that way on the separate FSTCompletion API, TestFSTCompletion.java:157, :165), or a count forwarded from a request without a bound.
• Default or opt-in. The default constructor and default numFactor; the class itself is an opt-in, @lucene.experimental suggester. Counts in the tens to thousands are not affected.
• Effect. At Integer.MAX_VALUE every lookup on a built suggester fails (the collector is created before the search, AnalyzingInfixSuggester.java:739; executed with one entry), with an IllegalArgumentException that blames numHits and suggests TotalHitCountCollector, neither of which the caller used. At specific values in the hundreds of millions (millions with a large numFactor) the lookup silently returns fewer suggestions than exist, and a larger num can return fewer than a smaller one. That short list is the correctness impact: on stock code Integer.MAX_VALUE fails anyway (LFV-1004), so there the wrap swaps one error for another.
• Workaround. Pass a finite count no larger than Integer.MAX_VALUE / numFactor; with LFV-1004 also present, small enough for the collector the parent pre-allocates (in practice the number of entries, getCount()).
• How likely: rare (registry likelihood rare, priority low). The natural trigger fails loudly on the first call, so developers meet it in testing and pick a finite count; the silent values are narrow windows; the recorded searches found no public report since 4.7.0.

TestBlendedInfixSuggesterNumOverflow (46 lines with the license header, one test method, package org.apache.lucene.search.suggest.analyzing like the upstream suggester tests). The try/catch only turns the exception into a message that states expected vs actual; without it the test fails the same way with the raw IllegalArgumentException. LuceneTestCase still randomizes its environment (codec, locale, time zone); the test failed with every seed tried (DEADBEEF, and 5EED1234 five times), with assertions on and off.

The silent short list (num = 429,496,731 returns 1 of 8) is not in the test on purpose: it is the same int product at other values with the same fix, and it needs 8 entries and a num chosen to land in a window. The same test is the suggested regression test; it passes with the fix.

cp $REPRO/TestBlendedInfixSuggesterNumOverflow.java $LUCENE/lucene/suggest/src/test/org/apache/lucene/search/suggest/analyzing/
cd $LUCENE && ./gradlew :lucene:suggest:test --tests org.apache.lucene.search.suggest.analyzing.TestBlendedInfixSuggesterNumOverflow -Ptests.seed=DEADBEEF -Ptests.asserts=true
# with -Ptests.asserts=false it fails the same way

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true
TestBlendedInfixSuggesterNumOverflow > testLookupWithIntegerMaxValueNum FAILED
    java.lang.AssertionError: expected 1 suggestion for num=Integer.MAX_VALUE, got java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count
        at org.apache.lucene.search.suggest.analyzing.TestBlendedInfixSuggesterNumOverflow.testLookupWithIntegerMaxValueNum(TestBlendedInfixSuggesterNumOverflow.java:41)
        Caused by:
        java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.TopFieldCollectorManager.<init>(TopFieldCollectorManager.java:63)
            at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.lookup(AnalyzingInfixSuggester.java:739)
            at org.apache.lucene.search.suggest.analyzing.BlendedInfixSuggester.lookup(BlendedInfixSuggester.java:199)
            at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.lookup(AnalyzingInfixSuggester.java:536)
            at org.apache.lucene.search.suggest.analyzing.BlendedInfixSuggester.lookup(BlendedInfixSuggester.java:175)
            at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.lookup(AnalyzingInfixSuggester.java:516)
            at org.apache.lucene.search.suggest.analyzing.BlendedInfixSuggester.lookup(BlendedInfixSuggester.java:163)
            at org.apache.lucene.search.suggest.Lookup.lookup(Lookup.java:195)
            at org.apache.lucene.search.suggest.analyzing.TestBlendedInfixSuggesterNumOverflow.testLookupWithIntegerMaxValueNum(TestBlendedInfixSuggesterNumOverflow.java:38)
:lucene:suggest:test (FAILURE): 1 test, 1  failure
# the seed frame is left out; with -Ptests.asserts=false gradle fails the same way
# standalone runner, seed DEADBEEF: main -ea and -da, 10.6.0 -da, 10.5.2 -da: 1 run, 1 failed, the same exception

• Contract. num is a maximum (Lookup.java:190); the BooleanQuery overload says "@param num number of items to return" (AnalyzingInfixSuggester.java:617). Nothing bounds it or says the subclass multiplies it.
• The code's own intent. "we reduce the num to the one initially requested" (BlendedInfixSuggester.java:226) holds only without overflow.
• A positive num must not produce these errors. The IllegalArgumentException message is false for the caller's input, the NoSuchElementException is undocumented, and a larger num can return fewer results (1,000 gives 8, 429,496,731 gives 1), with the count jumping between nearby values (429,496,730 throws, 429,496,732 gives 2); a top-N lookup should not shrink when N grows.
• History: a count that does not match num was a bug here. SOLR-7865 ("BlendedInfixSuggester was returning more results than requested", Arcadius Ahouansou's patch, committed by Michael McCandless as 602a1f50e72, 2015) fixed the opposite symptom, num * numFactor applied several times along the call chain, and left one multiplication, still in int: today's line 199.
• History: overflow in this class's arithmetic was a bug. On LUCENE-8343 (#9390) Adrien Grand asked of the score arithmetic "It might even introduce overflows?", and Alessandro Benedetti fixed it (e0232f10450, 2018, "weight long overflow fix + test"). Reviewing LUCENE-6339 (#7400), Michael McCandless asked of a suggester's queue-size product whether it "could possibly overflow int? Maybe use long".
• Tests. TestBlendedInfixSuggester uses num up to 10 only; no test accepts the wrap.

The strongest "works as designed" case (first raised by the finder's refute-lens reviewer from another model family, OpenAI GPT-5.6, then argued by the intended-lens reviewer and the devil's advocate): num is only a maximum; a num in the hundreds of millions is unreasonable for a suggester; the class is @lucene.experimental and warns that it "might be costly"; the small-num design is explicit in LUCENE-5354 (Remi Melisson: "If I want 10 suggestions, then I search against the current ordered index for the 100 first results"); and on stock code the parent already throws OutOfMemoryError at these values.

Why it fails: that bears on likelihood and priority, not on correctness. No doc bounds num or numFactor; a design for small num does not make a false error message or a wrong count acceptable for a large one; @lucene.experimental is about API stability; and once the parent's pre-size is fixed (LFV-1004), the wrap is the only thing that still fails. The adjudication (confirmed, high confidence, priority low), the independent reproduction, the cross-model review and the intended-behaviour check treat it as a real defect; the refute-lens review's "not real" (low) was procedural, as it could not run Java.

One fix was built and validated (two sites, together with LFV-1004's fix); three alternatives were considered and not built.

Past the threshold the fetch count becomes Integer.MAX_VALUE, so createResults keeps Integer.MAX_VALUE / numFactor results (214,748,364 for the default) instead of a wrapped count, and every larger num behaves like num = 214,748,364. Below the threshold nothing changes. The parent still hands the uncapped count to createResults (AnalyzingInfixSuggester.java:745); only the collector is capped. It is the end state of the neighbouring queue-size product in NRTSuggester (#11923 made that product long; its Math.min saturation was already there), and the fix recommended for LFV-999.

// BlendedInfixSuggester.lookup(key, contextQuery, num, ...), line 199:
// Multiply in long and saturate: in int, num * numFactor wraps once num > MAX_VALUE / numFactor
int fetch = (int) Math.min((long) num * numFactor, Integer.MAX_VALUE);
return super.lookup(key, contextQuery, fetch, allTermsRequired, doHighlight);

// AnalyzingInfixSuggester.lookup, line 739 (LFV-1004's fix):
int numHits = Math.min(num, Math.max(1, searcher.getIndexReader().maxDoc()));
TopFieldCollectorManager c = new TopFieldCollectorManager(SORT, numHits, null, 1);

Trade-offs

• Both parts are needed for Integer.MAX_VALUE. Saturation alone sends 2,147,483,647 to the parent, which pre-sizes its collector and throws OutOfMemoryError: Requested array size exceeds VM limit (executed, standalone, 512 MB heap). The cap alone gives min(-10, maxDoc) = -10 and the same IllegalArgumentException (executed). So this fix goes with LFV-1004's fix, in the same change or after it.
• Cap only the collector. If the count handed to createResults were capped too, the division would give maxDoc / numFactor and truncate again (code reading).
• Ceiling. A lookup returns at most Integer.MAX_VALUE / numFactor results, far above any realistic suggester. The fix removes the false errors; it does not make billions of results cheap (memory and time still bound them).
• Equivalent to the finder's sketch, which saturates at the largest multiple of numFactor that fits an int: both give createResults the same Integer.MAX_VALUE / numFactor.
• numFactor <= 0, not addressed. The constructors do not check numFactor (a remark of the intended-lens reviewer). The fix changes a result only where the true product exceeds Integer.MAX_VALUE; with a negative numFactor that takes a negative num (-2,147,483,647 x -2: -2 today, Integer.MAX_VALUE with the fix), and a negative product can still wrap (Integer.MAX_VALUE x -10 gives 10 before and after). A patch could validate numFactor > 0 in the constructors; not part of this report.

What the checks validated

• Fresh scratch worktree of 645d4fe14a20, gradle, OpenJDK 25: the repro fails on the stock tree and passes with the fix, with assertions on and off.
• With the fix and the repro test, the whole lucene/suggest module passes with assertions on (seed DEADBEEF) and off (seed 5EED0998): 32 suites, 237 tests, 7 skipped, 0 failures, TestBlendedInfixSuggester (9 tests) and TestAnalyzingInfixSuggester (23) among them.
• :lucene:suggest:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis) passes.
• The finder's 4-case test (the silent 1-of-8 values for numFactor 10 and 1000, the NoSuchElementException value, a negative wrap) passes 4 of 4 with this fix.
• Earlier, by the hunt: the finder's variant passes its test 4 of 4 and 144 upstream suggest tests with all the hunt unit's fixes (main a9779ef3c193); the diagram probe's fix runs return all 8 suggestions at every probed value.
• Not run: nightly tests, a benchmark (one multiplication per lookup).

Throw ArithmeticException instead of a misleading IllegalArgumentException or a short list.

Trade-offs

Honest, but it rejects a legal request (num is only a maximum) that the saturating fix answers correctly.

num = Math.min(num, getCount()) in BlendedInfixSuggester would also keep the parent's collector small.

Trade-offs

A second searcher acquire per lookup (getCount() acquires one, AnalyzingInfixSuggester.java:962-974); still needs the long product for very large suggesters; leaves the parent's own pre-size (LFV-1004) unfixed for AnalyzingInfixSuggester users.

Pass the caller's num to createResults separately from the fetch count, so nothing is recovered by num / numFactor.

Trade-offs

Removes the fragile round trip, but changes the protected createResults contract that subclasses may override: a bigger change than the bug needs.

Method: non-formal first signal, then a small formal model (the hunt's hunt unit for suggester num sizes, 2026-10-28, main a9779ef3c193).

• First signal. A numeric-limits probe ran 9 Lookup configurations (8 classes) at growing num on the stock main jars (8 entries, 1 GB heap): BlendedInfixSuggester threw IllegalArgumentException at Integer.MAX_VALUE and 214,748,365 and NoSuchElementException at 429,496,730, where num = 1000 returns 8.
• Model. A Z3 32-bit bit-vector model of the num arithmetic proved the wrap-free region num <= Integer.MAX_VALUE / numFactor, showed the bound tight, and produced the silent-truncation witnesses the probe had not shown (429,496,731 gives 1 of 8; 4,294,969 with numFactor 1000). An exhaustive evaluation of every positive int counted the outcomes, and a differential sweep covered 11 Lookup configurations.
• Execution. Each witness ran on the stock jars, then became a JUnit; the minimal-repro step reduced it to one method.