Word2VecSynonymFilter: maxSynonymsPerTerm = Integer.MAX_VALUE passes validation, then every in-model word throws

Word2VecSynonymFilter (@lucene.experimental, since 9.7.0) adds, for every token that has a vector in a word2vec model, the model's most similar words as synonyms, at most maxSynonymsPerTerm of them (default 5) and only above minAcceptedSimilarity (default 0.8). Its factory rejects only maxSynonymsPerTerm <= 0, so Integer.MAX_VALUE, meant as "no count cap, let the threshold decide", builds an analyzer without complaint. On the first token that has a vector, Word2VecSynonymProvider.getSynonyms asks HnswGraphSearcher.search for maxSynonymsPerTerm + 1 hits (the query word is its own nearest hit). The sum is int arithmetic and is never capped at the model size, and the search allocates two queues of that many entries before it starts. At Integer.MAX_VALUE the + 1 wraps to -2,147,483,648 and the queue throws IllegalArgumentException: initialCapacity must be > 0 and < 2147483630; got: -2147483648; values from 2,147,483,630 throw the same exception without overflowing, and values from about heap-bytes/16 (1e8 at a 1 GiB heap) throw OutOfMemoryError, even for a 2- or 3-word model. The failure is loud and immediate, with no wrong results. It is in every release with the filter, 9.7.0 to 10.6.0, and on main 645d4fe14a20. Capping the search size at the model size, computed in long, fixes it; validated with gradle on 645d4fe14a20.

1. Validation. Word2VecSynonymFilterFactory reads the value with getInt(args, "maxSynonymsPerTerm", DEFAULT_MAX_SYNONYMS_PER_TERM) (Word2VecSynonymFilterFactory.java:53; the default is 5, :41) and rejects only <= 0: "maxSynonymsPerTerm must be a positive integer greater than 0. Found: ..." (:72-76). The factory is @lucene.experimental and registered under the SPI name Word2VecSynonym (:32-33, :39), so CustomAnalyzer and settings-driven analyzers reach it by name.
2. Per token. Word2VecSynonymFilter.incrementToken calls getSynonyms(term, maxSynonymsPerTerm, minAcceptedSimilarity) for every token (Word2VecSynonymFilter.java:95-96).
3. The search size. Only a term that has a vector searches (Word2VecSynonymProvider.java:78-79). Then HnswGraphSearcher.search(scorer, maxSynonymsPerTerm + 1, hnswGraph, null, Integer.MAX_VALUE) (:83-90), under the comment "The query vector is in the model. When looking for the top-k it's always the nearest neighbour of itself so, we look for the top-k+1" (:85-86). The + 1 (:87) is plain int addition (the shipped bytecode is iload_2, iconst_1, iadd), and nothing compares it with word2VecModel.size().
4. Eager allocation. The static HnswGraphSearcher.search (HnswGraphSearcher.java:177-186) builds new TopKnnCollector(topK, visitedLimit, null) (:180) and new NeighborQueue(topK, true) (:183) before it searches; both create a TernaryLongHeap(topK) (TopKnnCollector.java:45, NeighborQueue.java:59-60).
5. The heap. TernaryLongHeap throws IllegalArgumentException for a capacity < 1 or >= ArrayUtil.MAX_ARRAY_LENGTH ("Throw exception to prevent confusing OOME", TernaryLongHeap.java:57-63); otherwise it allocates new long[initialCapacity + 1] (:66-68). Two such queues cost about 16 bytes per unit of maxSynonymsPerTerm, whatever the model size.

Boundary probes on main, assertions off, a 3-word model unless noted (the repro shows only the 2,147,483,647 row):

Tokens without a vector analyse fine, because no search runs. On 9.7.0 to 10.3.2 the message comes from LongHeap ("maxSize must be > 0 and < 2147483630; got: -2147483648"). The exact start of the IllegalArgumentException range depends on the JVM's maximum array length, and the OutOfMemoryError range on the heap: all thresholds are observations on one JVM and heap size. The rows at 2,147,483,629 and from 2,147,483,630 to 2,147,483,646 (3-word model) came from direct getSynonyms calls, endpoints only; single timings include building the analyzer.

• Who is hit. Users of the @lucene.experimental Word2VecSynonymFilter who configure maxSynonymsPerTerm as Integer.MAX_VALUE, the "no cap" value analysis-common itself uses as the default of CapitalizationFilter's maxWordCount and maxTokenLength (CapitalizationFilter.java:34-35), or as any value from about 100 million up (heap-dependent). The default 5 and ordinary values (the upstream tests use at most 10) never fail. There is no evidence that anyone configures it this way (no public report).
• Where. Index time and query-parse time, on the first token that has a vector, whatever the model size. Analyzers built by SPI name (CustomAnalyzer, settings files) pass the value to the factory unchanged.
• How it shows. An IllegalArgumentException that names an internal heap and a negative capacity rather than the setting, or an OutOfMemoryError; both only after the analyzer was built without complaint. No wrong results and nothing written to the index.
• A cost even when it works. Below the failure range every in-model token still allocates two queues sized by the setting rather than by the model: 1e7 is about 160 MB per token (arithmetic); one token took 46 ms on a 3-word model (a single timing that includes building the analyzer). The fix removes that too.
• How likely: rare (registry likelihood rare, priority low): only extremely large values trigger it (near Integer.MAX_VALUE, or from about 100 million up, heap-dependent), and it shows on first use. No public report since 9.7.0 (2023) in a bounded search.

TestWord2VecLargeMaxSynonymsPerTerm (60 lines with the license header, one test method, package org.apache.lucene.analysis.synonym.word2vec like the upstream word2vec tests, but public API only). It writes a 2-word plain-text dl4j model (syn0.txt in a zip) into createTempDir(), builds the analyzer by SPI name and expects alpha to analyse to [alpha, beta]. If the factory rejected the value when the analyzer is built, the test would return early and pass, so a fix may take either route. Nothing depends on the seed.

This exact file ran unchanged on 9.7.0, 9.12.3, 10.0.0, 10.3.2, 10.4.0, 10.5.2, 10.6.0 and main (the other 18 releases ran the finder's longer test). It is also the regression test, except that :lucene:analysis:common:check rejects its named, unused catch parameter (ecj lint: the value of the exception parameter is not used) and accepts it written as _ (Lucene's convention on main); with _ the check passes, and the test passes with the fix.

cp $REPRO/TestWord2VecLargeMaxSynonymsPerTerm.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/synonym/word2vec/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.synonym.word2vec.TestWord2VecLargeMaxSynonymsPerTerm -Ptests.seed=DEADBEEF
# fails the same way with -Ptests.asserts=false

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF (assertions on; -Ptests.asserts=false prints the same; module prefixes and the SeedInfo frame dropped)
TestWord2VecLargeMaxSynonymsPerTerm > testMaxIntMaxSynonymsPerTerm FAILED
    java.lang.AssertionError: expected [alpha, beta], got java.lang.IllegalArgumentException: initialCapacity must be > 0 and < 2147483630; got: -2147483648
        at org.apache.lucene.analysis.synonym.word2vec.TestWord2VecLargeMaxSynonymsPerTerm.testMaxIntMaxSynonymsPerTerm(TestWord2VecLargeMaxSynonymsPerTerm.java:56)

        Caused by:
        java.lang.IllegalArgumentException: initialCapacity must be > 0 and < 2147483630; got: -2147483648
            at org.apache.lucene.util.TernaryLongHeap.<init>(TernaryLongHeap.java:59)
            at org.apache.lucene.util.hnsw.NeighborQueue.<init>(NeighborQueue.java:60)
            at org.apache.lucene.search.TopKnnCollector.<init>(TopKnnCollector.java:45)
            at org.apache.lucene.util.hnsw.HnswGraphSearcher.search(HnswGraphSearcher.java:180)
            at org.apache.lucene.analysis.synonym.word2vec.Word2VecSynonymProvider.getSynonyms(Word2VecSynonymProvider.java:83)
            at org.apache.lucene.analysis.synonym.word2vec.Word2VecSynonymFilter.incrementToken(Word2VecSynonymFilter.java:96)
            ...
:lucene:analysis:common:test (FAILURE): 1 test, 1  failure

# 9.7.0 to 10.3.2 (LongHeap): IllegalArgumentException: maxSize must be > 0 and < 2147483630; got: -2147483648
# with the recommended fix: 1 test, 0 failures, assertions on and off

• Contract. The factory states the accepted domain in its own check: "maxSynonymsPerTerm must be a positive integer greater than 0" (Word2VecSynonymFilterFactory.java:72-76); TestWord2VecSynonymFilterFactory asserts only the rejection of 0 and -1. The filter javadoc: "@param maxSynonymsPerTerm maximum number of result returned by the synonym search" (Word2VecSynonymFilter.java:58), a cap on results, with no upper bound. package-info.java, CHANGES.txt:3610 (GITHUB#12169, the only entry) and MIGRATE.md say nothing more.
• A cap above the model size is an ordinary, tested input. TestWord2VecSynonymFilter passes 10 for an 8-word and a 4-word model (TestWord2VecSynonymFilter.java:85-97, :115-123).
• Lucene caps requested counts, in the same kind of call. LeafReader.searchNearestVectors does k = Math.min(k, floatVectorValues.size()) before new TopKnnCollector(k, ...) (LeafReader.java:288, :332, :376); IndexSearcher caps numHits (IndexSearcher.java:597, :729). Where a setting drives an allocation, analysis code rejects too-large values when it is set: StandardTokenizer.setMaxTokenLength throws above MAX_TOKEN_LENGTH_LIMIT (StandardTokenizer.java:98-99).
• History. The + 1 is in the first version (#12169, 1f4f2bf5095, 2023); the 38 comments, 33 review comments and 5 review summaries (15 reviews, 5 with a summary) of #12169 do not discuss an upper bound, large values or overflow.
• Why the tests never hit it. The word2vec tests use limits of at most 10, and TestRandomChains draws random int constructor arguments from [-50, 50] only (TestRandomChains.java:176-179).

The strongest "works as designed" case (the study's intended-behaviour checker, and the cross-model reviewer, which rated it "moderate"): the value is trusted configuration on an experimental filter; Integer.MAX_VALUE is not documented as an "unlimited" sentinel; maintainers have called a huge, eagerly allocated HNSW k a pattern to avoid (benwtrent on #11946: the queue "eagerly allocates a new long[heapSize]. This is VERY costly"; jpountz on #12679: k = Integer.MAX_VALUE "will not only run in O(maxDoc) time but also use O(maxDoc) memory. I don't think we should do this"); TestRandomChains bounds int arguments because the full range "could cause huge ram usage ... (e.g. allocate enormous arrays)"; and the failure is loud.

Why it fails: trust decides security, not validity (SECURITY.md:46: out-of-scope issues "are still valid bug reports and hardening suggestions"). A cap needs no sentinel: as a maximum, Integer.MAX_VALUE simply means no cap. The two maintainer remarks are about Lucene's own designs and about exactly the eager allocation the clamp removes; neither speaks to an accepted value overflowing into a negative capacity. The TestRandomChains TODO records a test gap (its neighbour TODO for char says "fix any filters that care to throw IAE instead"). And the error arrives at analysis time, after the analyzer was built, naming an internal heap. Verdict: not intended; the case reduces to severity, which is why the bug is Low priority.

Contested. An independent fact check by a reviewer from another model family (OpenAI GPT-5.6, reading only) judged it noise ("needs absurd parameters and then fails loudly"), at most low-priority hardening. The tie-breaker kept it: as a cap, Integer.MAX_VALUE is the plain "no cap" value, not an absurd one.

One fix was built and validated; two alternatives were considered and not built.

In Word2VecSynonymProvider.getSynonyms (Word2VecSynonymProvider.java:87), pass (int) Math.min(maxSynonymsPerTerm + 1L, word2VecModel.size()) instead of maxSynonymsPerTerm + 1. The search can never return more hits than the model has words, so output is unchanged whenever maxSynonymsPerTerm + 1 fits in the model, and above that the extra queue room could never be filled. It removes the overflow and the allocation sized by the setting. The same pattern as LeafReader.searchNearestVectors (k = Math.min(k, size)).

              // The query vector is in the model. When looking for the top-k
              // it's always the nearest neighbour of itself so, we look for the top-k+1
              // never more than the model holds (also avoids int overflow of the + 1)
              (int) Math.min(maxSynonymsPerTerm + 1L, word2VecModel.size()),

Trade-offs

• Large models. With the clamp, Integer.MAX_VALUE means "every word above the threshold", so each in-model token of a large model searches for the whole model: two queues of model-size entries (about 16 bytes per word) and a walk over most of the graph (arithmetic, not executed). Correct, but slow for models with millions of words; it is the cost of the request itself, and the same as setting the value to the model size today.
• The sibling bug LFV-862 (the provider can return k + 1 synonyms when the query word is missing from its own hits) is not fixed by this; its fix caps the result loop, a different line, and the two compose.

What the checks validated

Gradle in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3: stock, the repro fails with assertions on and off (1 test, 1 failure each). With the fix:

• the repro passes, assertions on and off (1 test, 0 failures each);
• the word2vec test package with the repro: 5 classes, 22 tests, 0 failures, assertions on (seed DEADBEEF) and off (seed 5EED0866);
• the whole :lucene:analysis:common:test task with the repro added, assertions on: 340 classes, 1,910 tests, 0 failures (18 skipped by the module itself);
• :lucene:analysis:common:check -x test (google-java-format, ecj lint, forbidden APIs, license checks) with the fix and the repro, its catch parameter named _: passes (with the name used in the repro file, ecjLintTest rejects the unused exception parameter).

Earlier, the same clamp passed the independent reproduction's 4-test class (4 of 4) and the 21 upstream word2vec tests through the repro runner.

Not built. Gives a configuration-time error instead of an analysis-time one.

Trade-offs

The factory constructor does not know the model size (the model is loaded later, in inform), so the bound would be a fixed constant such as ArrayUtil.MAX_ARRAY_LENGTH - 2: that removes only the IllegalArgumentException range, not the OutOfMemoryError one, and it does not protect code that builds Word2VecSynonymFilter or calls getSynonyms directly (neither validates the value). Rejecting values above the model size instead would turn a sensible "no cap" setting into an error. Possible as an addition to the clamp, not instead of it.

Not built. Start the TopKnnCollector and NeighborQueue heaps small and let them grow.

Trade-offs

Would replace the up-front allocation sized by k with growth bounded by what the search actually collects, for every caller of HnswGraphSearcher.search (on a large graph the queues can still grow large), but would not fix the overflow at Integer.MAX_VALUE (the capacity is already negative when it arrives), and it touches the kNN hot path, so it needs benchmarks. Out of proportion for this bug.

Method: non-formal (a hunt unit for the word2vec synonym provider, 2026-10-28, main a9779ef3c193).

• Context. The unit built a translation-validation harness: the provider's HNSW-based synonyms against an exact cosine ranking over the same model.
• First signal. Reading Word2VecSynonymProvider.getSynonyms for that harness, the finder noticed that the k + 1 self-hit expression is plain int addition and is never capped at the model size, while the factory accepts any positive value.
• Probe. Through the factory on the stock jars: maxSynonymsPerTerm 5 and 1,000,000 return [alpha, beta]; 2,147,483,646 and 2,147,483,647 throw IllegalArgumentException from TernaryLongHeap.
• JUnit. A public-API test then failed on main, branch_10_6, 10.5.2, 10.0.0, 9.12.3 and 9.7.0, and an independent reproducer measured the OutOfMemoryError range. The minimal-repro step reduced the test to one method over a 2-word model.