MoreLikeThis.retrieveInterestingTerms sizes its result list by maxQueryTerms, so Integer.MAX_VALUE throws OutOfMemoryError even for a one-word document

MoreLikeThis.retrieveInterestingTerms returns a document's most interesting words, at most maxQueryTerms of them. Both public overloads first run new ArrayList<>(maxQueryTerms) (MoreLikeThis.java:889 and :914 on main 645d4fe14a20) and only then build the term queue, so the list's array is sized by the limit, not by the words found. With setMaxQueryTerms(Integer.MAX_VALUE), read as "no limit", every call throws java.lang.OutOfMemoryError: Requested array size exceeds VM limit, even for a one-word document (on the tested OpenJDK 25 VM, with a 2 GB and a 16 GB heap alike); like() on the same object returns a one-clause query, because its queue is capped at Math.min(maxQueryTerms, number of words) (:627). Large values below the JVM's array limit return the right words but allocate about 4 bytes per unit of the limit on every call with compressed object pointers (400 MB at 100,000,000), or fail with "Java heap space". The line dates from 2005 (175cf8a9fd8) and is in every release that ships MoreLikeThis, 1.9 through 10.6.0, and main (a source scan of all 147 release tags; 22 releases and main fail when run).

// MoreLikeThis.java:913-925; the docNum overload at :888-901 has the same body
public String[] retrieveInterestingTerms(Reader r, String fieldName) throws IOException {
  ArrayList<String> al = new ArrayList<>(maxQueryTerms);      // :914, before any word is known
  PriorityQueue<ScoreTerm> pq = retrieveTerms(r, fieldName);  // queue capped at min(maxQueryTerms, #words), :627
  ScoreTerm scoreTerm;
  ...
  int lim = maxQueryTerms;
  while (((scoreTerm = pq.pop()) != null) && lim-- > 0) {    // maxQueryTerms used as a cap
    al.add(scoreTerm.word);
  }
  ...

1. setMaxQueryTerms only assigns the field (MoreLikeThis.java:528-530); the default is 25 (:232).
2. new ArrayList<>(n) allocates new Object[n] at once for any positive n (JDK ArrayList.java:157). On the tested OpenJDK 25 HotSpot VM, the JVM refuses an array of Integer.MAX_VALUE elements outright ("Requested array size exceeds VM limit", the same with a 2 GB and a 16 GB heap); below that it allocates about 4 bytes per element with compressed object pointers enabled, as in every run here.
3. The pre-size is never needed: createQueue builds the queue with final int limit = Math.min(maxQueryTerms, this.getTermsCount(perFieldTermFrequencies)); (:627) and adds words only while queue.size() < limit (:663), so the list never holds more than pq.size() words.
4. like() (:555, :584-590), MoreLikeThisQuery.rewrite (MoreLikeThisQuery.java:71) and the XML query parser's <LikeThisQuery maxQueryTerms="..."> (LikeThisQueryBuilder.java:90) all go through createQueue and never allocate by the limit. No Lucene code calls retrieveInterestingTerms.

Bytes allocated by one call after a warm-up call, with compressed object pointers on. The fix column ran with the fixed class shadowed; no bytes were measured at the two largest limits.

The docNum overload, retrieveInterestingTerms(0) on a second one-document index with the same text stored, throws the same error at Integer.MAX_VALUE from :889; with the fix it returns [lucene] at 25, 10,000 and Integer.MAX_VALUE.

• Who is hit. Code that calls retrieveInterestingTerms(int docNum) or retrieveInterestingTerms(Reader, String) with a maxQueryTerms far above the number of words it expects, for example Integer.MAX_VALUE to list every ranked keyword of a document. Before 4.10.0, 0 meant "no limit" for like(); since then 0 throws a NullPointerException (LFV-553), so a large value is today's way to ask for every word.
• Default vs opt-in. The default (25) and ordinary values are safe. It takes a non-default, large limit plus the convenience method; any analyzer, codec, directory and document.
• What goes wrong. At Integer.MAX_VALUE every call fails, on any document (the list is created before the document is read) and, on the tested VM, with a 2 GB and a 16 GB heap alike. The failure is a java.lang.Error; with -XX:+ExitOnOutOfMemoryError the JVM exited (code 3) although the caller caught Throwable. Without that flag nothing is left behind: the same MoreLikeThis and reader keep working. Below the array limit each call allocates about 4 bytes per unit of the limit with compressed object pointers (400 MB at 100,000,000) or fails with "Java heap space". No wrong results and no index change.
• Scale. None needed: one word is enough; fixed input, one thread, assertions on or off.
• Workaround. Pass a modest maxQueryTerms sized to the expected result; the default 25 is safe.
• How likely. Rare: it fails loudly on the first call, so developers meet it in testing, and the documented searches found no public report.

Copy the test into lucene/queries/src/test/org/apache/lucene/queries/mlt/ of a Lucene checkout and run it with gradle (command below). It uses StandardAnalyzer, ByteBuffersDirectory and a plain IndexWriter, so the input is deterministic; LuceneTestCase still randomizes locale and time zone. On the tested OpenJDK 25 HotSpot VM the error is the JVM's array-length limit, not heap exhaustion (the same with a 2 GB and a 16 GB heap); other JVM implementations were not tried. The test covers the (Reader, String) overload only; the docNum overload's failure at :889 comes from the diagram probe and the verifiers' tests.

Executed: gradle on a clean 645d4fe14a20 (fails; done twice, in two fresh scratch worktrees); the standalone runner on the stock main jars with -ea and -da and on the 10.6.0 and 10.5.2 release jars (fails); seeds DEADBEEF and 1234ABCD.

cp $REPRO/TestMoreLikeThisInterestingTermsLargeMaxQueryTerms.java $LUCENE/lucene/queries/src/test/org/apache/lucene/queries/mlt/
cd $LUCENE && ./gradlew :lucene:queries:test --tests org.apache.lucene.queries.mlt.TestMoreLikeThisInterestingTermsLargeMaxQueryTerms -Ptests.seed=DEADBEEF

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF
TestMoreLikeThisInterestingTermsLargeMaxQueryTerms > testUnboundedMaxQueryTerms FAILED
    java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:CEA0F8138A2E1EB2]:0)
        at java.base/java.util.ArrayList.<init>(ArrayList.java:157)
        at org.apache.lucene.queries.mlt.MoreLikeThis.retrieveInterestingTerms(MoreLikeThis.java:914)
        at org.apache.lucene.queries.mlt.TestMoreLikeThisInterestingTermsLargeMaxQueryTerms.testUnboundedMaxQueryTerms(TestMoreLikeThisInterestingTermsLargeMaxQueryTerms.java:48)

# standalone runner, seed DEADBEEF
main 645d4fe14a20   -ea / -da   tests_run=1 failures=1   same OutOfMemoryError at MoreLikeThis.java:914
10.6.0 (release)    -ea / -da   tests_run=1 failures=1   same
10.5.2 (release)    -ea         tests_run=1 failures=1   same

• Contract. setMaxQueryTerms and getMaxQueryTerms document "the maximum number of query terms that will be included in any generated query" (MoreLikeThis.java:512-530), with no bound and no validation. retrieveInterestingTerms(Reader, String) documents "@return the most interesting words in the document" and "@see #setMaxQueryTerms" (:903-912). Nothing mentions an allocation proportional to the limit. The expectation is implied, not spelled out.
• Same object, same setting. like() handles any value because its queue is sized by the words found (:627); only the convenience method fails.
• The clause limit is no bound either. The default's javadoc points at IndexSearcher#getMaxClauseCount (:228), but IndexSearcher.setMaxClauseCount accepts any value >= 1, Integer.MAX_VALUE included (IndexSearcher.java:256-261).
• "No limit" used to be 0. Up to 4.9.x createQuery stopped only if (maxQueryTerms > 0 && qterms >= maxQueryTerms), so 0 meant no limit; LUCENE-5795 (4.10.0) removed that, and 0 now throws a NullPointerException (LFV-553).
• Precedent. LUCENE-2119, under Bug fixes in the "Release 2.9.3 / 3.0.2" section: "Don't throw NegativeArraySizeException if you pass Integer.MAX_VALUE as nDocs to IndexSearcher search methods" (CHANGES.txt:18439-18441). Its issue text (Michael McCandless): "This trips users up fairly often because they assume our PQ doesn't statically pre-allocate (a reasonable assumption...)"; the fix capped the size at maxDoc, still in place (IndexSearcher.java:588, :597, :721, :729). LUCENE-8221 also fixed a large-value overflow in this class as a bug (CHANGES.txt:7456, 7.4.0).
• Lucene avoids this failure on purpose elsewhere. The PriorityQueue that createQueue builds rejects a size >= ArrayUtil.MAX_ARRAY_LENGTH with an IllegalArgumentException and the comment "Throw exception to prevent confusing OOME" (PriorityQueue.java:121-124). An OutOfMemoryError from a size parameter is not a failure Lucene accepts.
• History. The pre-size came with the method (175cf8a9fd8, 2005, "Added convenience routine retrieveInterestingTerms()"), when it still returned every word; later 81087e8bb66 made it return only the top terms and kept the pre-size. No test calls retrieveInterestingTerms, and the documented searches found no commit, issue or list thread that discusses the pre-size.

The strongest "works as designed" case (the intended-behaviour verifier and the cross-model reviewer): the default is 25 and its javadoc points at the clause limit (1024 by default); configuration values are "trusted as given" (SECURITY.md:42); pre-sizing a list to its limit is a common Java idiom; the LUCENE-2119 issue text itself calls Integer.MAX_VALUE "a nonsense value to pass in"; and createQueue's min() came from LUCENE-5795 to make term selection cheaper, not as a guard against large values.

Why it fails: no bound is documented or checked, and the clause limit itself accepts Integer.MAX_VALUE; like() on the same object accepts the value; the method's own loop treats it as a cap; Lucene's own PriorityQueue guards against exactly this "confusing OOME"; LUCENE-2119 called the user's assumption reasonable and fixed the caller. "Trusted configuration" defines security scope, not which accepted settings may fail. The argument bears on priority, not on correctness. The fix is a small robustness change; the createQueue min() is not a deliberate guard.

Three options. Option A was validated with gradle in a scratch worktree at 645d4fe14a20 and with the standalone runner; B and C were not executed. Options A and B do not change LFV-553 (maxQueryTerms = 0 throws a NullPointerException in createQueue); only option C's separate <= 0 check would.

In retrieveInterestingTerms(int docNum) and retrieveInterestingTerms(Reader, String), move the list creation below retrieveTerms(...) and size it by Math.min(maxQueryTerms, pq.size()) (one line removed and one added in each overload).

public String[] retrieveInterestingTerms(Reader r, String fieldName) throws IOException {
  PriorityQueue<ScoreTerm> pq = retrieveTerms(r, fieldName);
  ArrayList<String> al = new ArrayList<>(Math.min(maxQueryTerms, pq.size()));
  // was, before retrieveTerms: ArrayList<String> al = new ArrayList<>(maxQueryTerms);
  ...
// the same change in retrieveInterestingTerms(int docNum), after pq = retrieveTerms(docNum)

Trade-offs

• No change in results: the same loop fills the list, and pq.size() never exceeds maxQueryTerms (MoreLikeThis.java:627, :663), so the min is only defensive.
• A negative maxQueryTerms still throws IllegalArgumentException, now from the queue in createQueue (PriorityQueue rejects a negative size) instead of from ArrayList ("Illegal Capacity"): a visible but obscure change for a caller that inspects the message or the stack. 0 still throws LFV-553's NullPointerException.

What the checks validated

• Gradle, clean scratch worktree at 645d4fe14a20, JDK 25, -Ptests.seed=DEADBEEF: without the fix the minimal test fails; with it, :lucene:queries:checkGoogleJavaFormat :lucene:queries:test passes: 52 suites, 513 tests, 0 failures, 5 skipped, including the minimal test 1/1 and TestMoreLikeThis 11 tests (1 skipped).
• Repeated in a second fresh scratch worktree (./gradlew --no-daemon), with the same results.
• Standalone runner with the fixed class shadowed: the minimal test plus TestMoreLikeThis, 11 run, 0 failed, with -ea and -da (unfixed: 1 failure).
• Allocation probe with the fixed class shadowed: (Reader, String) returns [lucene] at every probed limit up to Integer.MAX_VALUE and allocates 5,368 bytes at each measured limit (25 to 100,000,000); the docNum overload returns [lucene] at 25, 10,000 and Integer.MAX_VALUE. Apart from the minimal test (at Integer.MAX_VALUE), the gradle run does not exercise large limits.
• Earlier, the finder's, the independent reproducer's and the adjudication's tests passed with the same fix.

new ArrayList<>(pq.size()), or new ArrayList<>(), after retrieveTerms(...).

Trade-offs

Equivalent to the recommended option for every value, since the queue is already capped at maxQueryTerms; slightly simpler. Not executed.

Reject values above some bound (and, separately, <= 0).

Trade-offs

An upper bound is arbitrary, turns a setting that like() handles today into an error, and below the bound still leaves an allocation proportional to the limit. Rejecting <= 0 would cover LFV-553, not this bug. Not recommended on its own; not executed.

Method: semi-formal translation validation (no model checker), in a hunt round on MoreLikeThis and CommonTermsQuery against reference semantics.

• A harness ran the real queries jar against a reference term-selection semantics written from the javadoc, with bounded-random documents and settings, and checked several rules; rule R6 was "no exception for any parameter value the setters accept".
• The randomized runs drew maxQueryTerms from 0 to 12, so they could not reach large values. The finder added a boundary probe by hand with Integer.MAX_VALUE, 1,000,000,000 and 100,000,000: the first threw OutOfMemoryError at once, the second ran out of a 1 GB heap, the third returned after allocating about 400 MB.
• Code reading located the pre-size; the finding was reduced to a public-API JUnit test, later to the single test shown here.