RescoreTopNQuery and LatLonPoint.nearest size their queue from n, not from the index, so n = Integer.MAX_VALUE throws even on a one-document index

RescoreTopNQuery(query, source, n) and LatLonPoint.nearest(searcher, field, lat, lon, n) return the top n hits and accept any n >= 1. Before they collect a single hit, each builds a Lucene PriorityQueue of n + 1 slots (RescoreTopNQuery.java:73, NearestNeighbor.java:254), which allocates every slot up front and refuses 2,147,483,631 or more. So n = Integer.MAX_VALUE, the usual way to ask for every hit, throws IllegalArgumentException on an index with a single document, and a large n below the limit costs 4 bytes per slot per call (compressed object references, as in the tested 1 GiB heap; 400 MB at n = 10^8 for 1 hit) or ends in OutOfMemoryError. IndexSearcher.search has capped the same request at maxDoc since LUCENE-2119 (Michael McCandless, 2009), still asserted by TestIndexSearcher.testHugeN, and nearest documents that it is "functionally equivalent" to that search sorted by distance. The fix caps each queue at what the index can return: maxDoc for RescoreTopNQuery, the number of indexed points for nearest (not maxDoc, because nearest returns a multi-valued document once per point). Default settings, any index; nearest is affected from 10.0.0, RescoreTopNQuery from 10.3.0, through 10.6.0 and main. Results are never wrong; the failure is loud.

1. RescoreTopNQuery documents "@param n the number of documents to find" and an IllegalArgumentException only "if n is less than 1" (RescoreTopNQuery.java:55-56), and the constructor checks only that (:58-61). The work happens in rewrite (:67-91), which IndexSearcher.rewrite calls: before any candidate is visited it builds HitQueue queue = new HitQueue(n, false); (:73), although the reader is in hand (:70). The class is @lucene.experimental (:34).
2. LatLonPoint.nearest "Finds the n nearest indexed points", is "functionally equivalent to running MatchAllDocsQuery with a LatLonDocValuesField#newDistanceSort", and does not de-duplicate multi-valued fields, so a document can appear once per point (LatLonPoint.java:380-387). It rejects only n < 1 (:409-411), collects each leaf's PointValues (:418-430) and passes n on unchanged (:432-433). The package-private NearestNeighbor builds new NearestHitQueue(n) (NearestNeighbor.java:254).
3. The queue. HitQueue and NearestHitQueue pass the size to PriorityQueue (HitQueue.java:62-64, NearestNeighbor.java:80-83). PriorityQueue throws for maxSize >= ArrayUtil.MAX_ARRAY_LENGTH ("Throw exception to prevent confusing OOME", PriorityQueue.java:121-125), otherwise allocates new Object[maxSize + 1] (:129, :134). MAX_ARRAY_LENGTH is Integer.MAX_VALUE minus the array header (ArrayUtil.java:33-34), 2,147,483,631 here.
4. The sibling that caps. IndexSearcher.searchAfter caps numHits at Math.max(1, reader.maxDoc()) (IndexSearcher.java:588, :597), and the sorted variant does too (:721, :729). TestIndexSearcher.testHugeN ("should not throw exception", TestIndexSearcher.java:78-117) calls them with Integer.MAX_VALUE, with and without a Sort.

Bytes allocated per call (thread allocation counter, minimum of 3 calls after 2 warm-ups):

Every call that returns finds the 1 hit. On one document with two points, nearest(..., 100) returns 2 hits, so nearest's cap has to be the point count, not maxDoc.

• Who is hit. Callers of new RescoreTopNQuery(..., n) or LatLonPoint.nearest(..., n) with Integer.MAX_VALUE or any n from 2,147,483,631 up: every call throws, on any index, even one with a single document. Also code that derives n from something large (an index-size estimate, a client's size parameter): about 400 MB per call at n = 10^8, and OutOfMemoryError above what the heap holds (the executed runs show the calling operation failing; effects on other work in the same JVM were not tested).
• Default vs opt-in. No setting is needed: default codec, IndexWriterConfig and IndexSearcher. Both APIs are opt-in: RescoreTopNQuery (10.3.0 on, @lucene.experimental) was added to re-rank a kNN query with full-precision vectors (#14009); LatLonPoint.nearest (core since 10.0.0) is a geo top-n helper.
• Scale. Only n matters, not the index size. Ordinary n (tens to thousands) costs kilobytes.
• Who is not. IndexSearcher.search and searchAfter callers (capped), and any n the heap can hold, which returns correct results.
• Severity. Loud: an exception for that call; no wrong results, nothing written to the index.
• How likely. Rare: the throw shows on the first call, so "all" is met in testing; default-branch code search found no use of either API in Solr, Elasticsearch or OpenSearch code, and the documented searches found no prior public report.

The test indexes one document with LatLonPoint("loc", 0, 0) in a ByteBuffersDirectory with a default IndexWriterConfig. One method counts new RescoreTopNQuery(new MatchAllDocsQuery(), DoubleValuesSource.constant(1), Integer.MAX_VALUE) with IndexSearcher.count (the queue is built in rewrite); the other calls LatLonPoint.nearest(searcher, "loc", 0, 0, Integer.MAX_VALUE). Each expects 1 hit, which the fixed code returns. The class has two methods because the bug has two fix sites. Single thread; the outcome is the same for every seed run (DEADBEEF, 5EED0995). The IllegalArgumentException is caught once and rethrown as an AssertionError that states expected against actual, with the exception kept as the cause. Before it goes into a PR, use MatchAllDocsQuery.INSTANCE (the constructor is deprecated on main; it does not affect the failure); the format check already passes. The OutOfMemoryError range is not in the test (it needs a heap-dependent n); its numbers are in the table above.

TestTopNQueuePresizedByN > testRescoreTopNQueryIntegerMaxValue FAILED
    java.lang.AssertionError: expected 1 hit for n = Integer.MAX_VALUE on 1 doc, got java.lang.IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:167D491953B4EF13]:0)
        at org.apache.lucene.search.TestTopNQueuePresizedByN.testRescoreTopNQueryIntegerMaxValue(TestTopNQueuePresizedByN.java:41)

        Caused by:
        java.lang.IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
            at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:123)
            at org.apache.lucene.search.HitQueue.<init>(HitQueue.java:63)
            at org.apache.lucene.search.RescoreTopNQuery.rewrite(RescoreTopNQuery.java:73)
            at org.apache.lucene.search.ConstantScoreQuery.rewrite(ConstantScoreQuery.java:44)
            at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:873)
            at org.apache.lucene.search.IndexSearcher.count(IndexSearcher.java:497)
            at org.apache.lucene.search.TestTopNQueuePresizedByN.testRescoreTopNQueryIntegerMaxValue(TestTopNQueuePresizedByN.java:39)

TestTopNQueuePresizedByN > testLatLonPointNearestIntegerMaxValue FAILED
    java.lang.AssertionError: expected 1 hit for n = Integer.MAX_VALUE on 1 doc, got java.lang.IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:451FE0BBEB01E9C]:0)
        at org.apache.lucene.search.TestTopNQueuePresizedByN.testLatLonPointNearestIntegerMaxValue(TestTopNQueuePresizedByN.java:54)

        Caused by:
        java.lang.IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
            at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:123)
            at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:80)
            at org.apache.lucene.document.NearestNeighbor$NearestHitQueue.<init>(NearestNeighbor.java:82)
            at org.apache.lucene.document.NearestNeighbor.nearest(NearestNeighbor.java:254)
            at org.apache.lucene.document.LatLonPoint.nearest(LatLonPoint.java:433)
            at org.apache.lucene.search.TestTopNQueuePresizedByN.testLatLonPointNearestIntegerMaxValue(TestTopNQueuePresizedByN.java:52)
2 tests completed, 2 failed

# the same with -Ptests.asserts=false; the 10.6.0 release jars fail the same way (assertions off); on 10.3.0 nearest fails with OutOfMemoryError: Requested array size exceeds VM limit instead

• Contract. RescoreTopNQuery: "n the number of documents to find", IAE only "if n is less than 1" (RescoreTopNQuery.java:55-56). nearest: "Finds the n nearest indexed points" and "functionally equivalent to running MatchAllDocsQuery with a LatLonDocValuesField#newDistanceSort" (LatLonPoint.java:380-384). That sorted search accepts Integer.MAX_VALUE and returns every document (IndexSearcher.java:721-733; executed on the repro's document with a LatLonDocValuesField added: 1 hit). The javadoc already excludes multi-valued de-duplication, so for single-valued documents the two differ exactly where the queue is sized.
• Tests. TestIndexSearcher.testHugeN asserts that the core top-n entry points accept Integer.MAX_VALUE; using it as the governing precedent was first pointed out by the cross-model reviewer (an independent reviewer from another model family, OpenAI GPT-5.6). Upstream TestRescoreTopNQuery uses small n only, and TestNearest never asks for more than the number of indexed points.
• History. LUCENE-2119 (#3195, ae72952fa41, 2009, "behave better if you pass Integer.MAX_VALUE as nDcos [sic] to search methods"; CHANGES.txt lists it under the 2.9.3 / 3.0.2 bug fixes, first released June 2010) added the cap to IndexSearcher; Michael McCandless wrote there: "This trips users up fairly often because they assume our PQ doesn't statically pre-allocate (a reasonable assumption...)". The same pattern was fixed at other sites: LUCENE-9614 (#413, Julie Tibshirani: "bound k by total number of vectors to prevent oversizing data structures") and #13184 (bugmakerrrrrr: kNN exact search caps its HitQueue at the candidate count). The threads that added these two APIs (#14009, #14860, LUCENE-7069 / #8125, LUCENE-10266 / #897) contain no identified discussion of a large n.
• Not a designed bound. For nearest, n = Integer.MAX_VALUE gave OutOfMemoryError: Requested array size exceeds VM limit up to 10.4.0 and gives the PriorityQueue IAE from 10.5.0, a side effect of #16154 ("Use Lucene's PriorityQueue in NearestNeighbor to replace poll/offer with updateTop"). The message names an internal maxSize, not n.

The strongest case for "works as designed" (built by an intended-behaviour devil's advocate and the adjudication): the LUCENE-2119 issue body itself calls Integer.MAX_VALUE "a nonsense value to pass in, since our PQ impl allocates the array up front"; Uwe Schindler wrote on java-user in 2009 "If you want to have all results, you do something wrong", and Simon Willnauer defended fixed-size top-k queues there in 2011; pre-allocation is documented on PriorityQueue, HitQueue and the collector managers, which take numHits literally; nearest's javadoc says it throws IllegalArgumentException if n is "out-of-bounds"; SOLR-5661 added the PriorityQueue guard on purpose and Shalin Shekhar Mangar accepted an error for a huge rows value there; RescoreTopNQuery is @lucene.experimental.

Why it fails: the same LUCENE-2119 then fixed the case as a bug, and testHugeN still asserts it. These two are convenience entry points like IndexSearcher.search, not collectors whose caller sizes the queue, and their javadocs mention no pre-allocation (PriorityQueue is @lucene.internal). "Out-of-bounds" came with the n >= 1 check in 2016 (ad86feca097), and no upper bound was ever defined. The guard only turns n >= 2,147,483,631 into an IAE; the OutOfMemoryError range below it has no guard. Willnauer defended fixed-size top-k queues and did not address capacity beyond what the index can return; the cap keeps the queue fixed-size. Honest weak spot: a maintainer may accept the IAE at Integer.MAX_VALUE as documented; the per-call allocation and the OOM on a one-document index do not depend on that reading. Verdict: not intended, a low-severity robustness defect. The claim is that the queue is not bounded by what the index can return, not that memory must follow the number of results.

Option A was applied in a private scratch worktree of 645d4fe14a20 and run with gradle (worktree removed after the runs); the hunt's checks validated the same diff on a9779ef3c193. Options B to D were not built.

Two small edits: RescoreTopNQuery.java:73, and LatLonPoint.java:418-433 (sum PointValues.size() in the loop that already visits every leaf, then pass the capped n). The maxDoc cap is the one Michael McCandless gave IndexSearcher on LUCENE-2119 (IndexSearcher.java:588).

// RescoreTopNQuery.rewrite (line 73)
HitQueue queue = new HitQueue(Math.min(n, Math.max(1, reader.maxDoc())), false);

// LatLonPoint.nearest: count the points in the existing leaf loop ...
long totalPoints = 0;
// ... totalPoints += points.size();   (next to totalHits += points.getDocCount();)
// ... and pass the capped n:
NearestNeighbor.nearest(latitude, longitude, readers, liveDocs, docBases,
    (int) Math.min(n, Math.max(1, totalPoints)));

Trade-offs

• Results are unchanged for every n that works today; only the queue gets smaller. maxDoc is at most IndexWriter.MAX_DOCS (Integer.MAX_VALUE - 128), below the queue limit, so the capped size always fits.
• nearest is capped at the point count, not maxDoc, because it returns a multi-valued document once per point: on one document with two points, nearest(..., 100) returns 2 hits (the finder's SMT obligation and a harness mutant found the same; the cross-model reviewers agreed). Points of deleted documents are counted, so it is an upper bound. NearestVisitor uses the same n to tell when the queue is full (NearestNeighbor.java:181); with the cap that happens only after every point was collected, so results do not change.
• Still loose for RescoreTopNQuery when the inner query matches few of many documents (see C).
• Residual at extreme scale, not part of this bug: n = Integer.MAX_VALUE over 2,147,483,631 or more points would still hit the guard; and under the draft long-doc-id change (#16847) maxDoc could exceed the array limit, so the cap would also need a clamp to ArrayUtil.MAX_ARRAY_LENGTH - 1.

What the checks validated

• Gradle, stock 645d4fe14a20, seed DEADBEEF: the repro fails 2 of 2 with -Ptests.asserts=true and false.
• Gradle with the fix: the repro passes 2 of 2 with assertions on and off; the closest suites (TestTopNQueuePresizedByN, TestRescoreTopNQuery, TestRescoreTopNQueryPrefetch, TestRescoreTopNQueryFloat16Prefetch, TestNearest, TestIndexSearcher; seed C0DE0995, assertions on): 6 classes, 27 tests, 0 failures, 3 skipped (1 by its own assumption, 2 nightly-only TestRescoreTopNQuery tests, which pass with the fix when the nightly group is enabled: 7 of 7); the whole org.apache.lucene.search.* and org.apache.lucene.document.* test packages of lucene/core (seed C0DE0995, assertions on): 274 classes, 2,839 tests, 0 failures, 75 skipped (tests a default run skips, such as the nightly group). :lucene:core:checkGoogleJavaFormat passes with the fix, with and without the repro file.
• The allocation probe with the fix: every n from 1,000 to Integer.MAX_VALUE returns the 1 hit at 2,296 bytes (RescoreTopNQuery) and 6,992 bytes (nearest); on one document with two points nearest returns 2 hits for n = 100 and for Integer.MAX_VALUE.
• The hunt's checks of the same diff: the finder's 4 tests pass (-ea, -da), upstream TestRescoreTopNQuery, TestRescoreTopNQueryPrefetch and TestNearest run 10 tests with 0 failures, an independent reproducer's 6 tests pass, and the bounded-exhaustive harness gives 3,081 checks with 0 failures.

Sum size() over the readers list that NearestNeighbor.nearest receives and cap there, next to the queue.

Trade-offs

Same effect as A for nearest: the class is package-private and LatLonPoint.nearest is its only caller.

Count or buffer the inner query's matches first and size the queue by that count.

Trade-offs

Tighter than A when a small candidate set sits in a big index (for example a kNN query with k = 100 over 10^7 documents still gets a 10^7-slot queue for Integer.MAX_VALUE under A), but needs a counting pass or a buffer; more code than the bug needs.

Start small and grow only as hits arrive, instead of allocating n + 1 slots up front.

Trade-offs

Would help every top-n path, but core PriorityQueue has a fixed capacity, so it needs a different structure on a hot path; larger than the bug needs.

A maxDoc cap for nearest drops hits for multi-valued fields. A documented upper bound on n turns the "all" idiom that IndexSearcher.search supports into an error, and helps the OutOfMemoryError range only if the bound is low.

Trade-offs

Both change results or the contract instead of only the queue size.

Method: semi-formal (the finder's label): executed boundary probes gave the first signal, then a bounded-exhaustive differential harness and SMT obligations on the stock jars.

• An accepted-parameter boundary probe (2026-10-26) saw both IllegalArgumentExceptions at n = Integer.MAX_VALUE and noted them next to a kNN row, not filed separately. A planning probe for top-n APIs reproduced both on the stock main jars with controls passing.
• The hunting AI tool wrote a differential harness (39 layouts of 1-3 segments, 1-2 points per document, 11 values of n; oracles: the API's own full result, the exact number of candidates or points, and an allocation bound) and 11 SMT bit-vector obligations. Every layout failed at n = 2,147,483,630 and 2,147,483,647 for both APIs; the obligations proved the maxDoc cap (RescoreTopNQuery) and the point-count cap (nearest) sound and gave the counterexample against a maxDoc cap for nearest, which a harness mutant confirmed.
• It reduced the case to public-API JUnits with controls, and later to the one-class repro.