A points sort budget of 2048 MB or more can make BKDWriter overflow its in-heap buffer size, so a points merge of more than 2 GiB that still fits the budget throws NegativeArraySizeException and closes the IndexWriter

Point fields (numeric, geo, range) are written as BKD trees. While writing a merged segment, BKDWriter keeps a field's points on heap when they fit the sort budget maxMBSortInHeap and spills them to a temporary file otherwise. The default budget is 16 MB; a custom codec can pass a larger one to the public Lucene90PointsWriter(state, maxPointsInLeafNode, maxMBSortInHeap) constructor, the route a committer pointed to in #8825 (for its predecessor, Lucene60PointsWriter).

BKDWriter turns the budget into a point count, (int) (maxMBSortInHeap * 1024 * 1024 / bytesPerDoc), with no cap at the largest Java array, and HeapPointWriter then allocates new byte[bytesPerDoc * size] as an int multiply. With a budget of 2048 MB or more, a writer that receives more than 2^31-1 bytes of points that still fit the budget overflows: 8.4M points of 260 bytes at 2200 MB give NegativeArraySizeException: -2110967296. Through IndexWriter this is a merge of a multi-dimensional point field, which fails as a tragic event and closes the writer; the last commit survives and a smaller budget recovers. It has been so since 8.1.0, when the old "Buffer too large for Java (2047mb max)" check went away; capping the count at ArrayUtil.MAX_ARRAY_LENGTH / bytesPerDoc fixes it.

1. The budget is public. Lucene90PointsWriter's "Full constructor" and its 3-argument constructor take maxMBSortInHeap with no documented range (Lucene90PointsWriter.java:51-60, :106-110); the 1-argument constructor uses 512 and 16.0 (:112-122). For each field it builds a BKDWriter with that budget and totalPointCount = values.size() (:145-153).
2. It becomes a point count with no upper cap. maxPointsSortInHeap = (int) ((maxMBSortInHeap * 1024 * 1024) / (config.bytesPerDoc())) (BKDWriter.java:196). The constructor's only checks on the budget are maxMBSortInHeap >= 0 (:212-216) and that one leaf fits (:198-209); verifyParams also rejects a negative totalPointCount (:217-220).
3. The first add() decides heap or disk. initPointWriter() (:223-233) opens an OfflinePointWriter if totalPointCount > maxPointsSortInHeap, else new HeapPointWriter(config, Math.toIntExact(totalPointCount)) (:231).
4. The heap buffer size is an int product. this.block = new byte[config.bytesPerDoc() * size] (HeapPointWriter.java:43). Above 2^31-1 bytes it wraps.

16 dimensions (8 indexed) x 16 bytes plus a 4-byte doc id = 260 bytes per point. A 2200 MB budget = 2,306,867,200 bytes holds 8,872,566 points, so 8.4M points stay on heap. 8,400,000 x 260 = 2,184,000,000 > 2,147,483,647, and the int product is 2,184,000,000 - 2^32 = -2,110,967,296.

• The first-step window: more than 2^31-1 bytes and no more than the budget. At 2200 MB and 260 bytes that is 8,259,553 to 8,872,566 points; 8,259,552 points still fit one array (executed by the independent reproducer), and a larger count spills at step 3 (executed by the fact check: 8,872,566 points fail with -1988100136, while 8,872,567 and 20,000,000 points spill to a temporary file on the first add()).
• Only budgets of 2048 MB or more: at exactly 2048 MB only when bytesPerDoc is a power of two; 2047 MB can never overflow (arithmetic; the hunt's SMT model agrees over all 256 field shapes).
• At 4096 MB or more the product can also wrap to a small or zero length. The one such case run (4096 MB, 1-D 4-byte points, length 0) failed on the first add() (IllegalStateException with -ea, ArrayIndexOutOfBoundsException with -da); no silent corruption was seen.
• Same root cause, model only, not executed: BKDRadixSelector sizes its heap writers from the same count (BKDRadixSelector.java:551-581): the delta writer can overflow above 2048 MB even for merges larger than the budget, when the histogram bucket holding the partition point (:291) lands in the window, which depends on the data; the partition writers above 4096 MB. With an absurd maxPointsInLeafNode, switchToHeap (BKDWriter.java:1606-1609; 8,259,553 or more for 260-byte points) and OneDimensionBKDWriter's leaf buffer (:687, maxPointsInLeafNode * packedBytesLength(); 134,217,728 or more for 16-byte 1-D points) overflow the same way.

A multi-dimensional field is merged by PointsWriter.mergeOneField, which sums the sources' PointValues.size() (deleted documents included) into totalPointCount and re-adds every point (PointsWriter.java:43-170: the sum at :43-57, size() at :107-110, the re-add at :117-170; writer.add at Lucene90PointsWriter.java:173). 1-D fields use the bulk BKDWriter.merge when every source is a Lucene90PointsReader and the generic path otherwise (Lucene90PointsWriter.java:192-203). Flushes sort a MutablePointTree in place (:155-161) and never call add(). Natural merges take the same path (the availability probe's fresh writers failed on maybeMerge()), and addIndexes(CodecReader...) writes through the same SegmentMerger (IndexWriter.java:3491, :3568-3575; code reading, not run). A merge failure other than an abort (handleMergeException swallows a non-external MergeAbortedException, IndexWriter.java:5043-5053), such as this one, goes to tragicEvent(t, "merge") and is rethrown (:5109-5113).

• Who is exposed. Applications with a custom codec that passes a budget of 2048 MB or more to Lucene90PointsWriter(state, maxPointsInLeafNode, maxMBSortInHeap), or that use BKDWriter directly. The default codec uses 16 MB. No in-tree test uses 2048 MB or more (randomized codecs draw 3-8 MB, RandomCodec 5-8 MB; direct BKD tests use 0.001 to 16 MB), and code search of the major downstream search servers found none that raises the budget.
• Scale. One merge of one point field with more than 2^31-1 bytes of points (points x bytes per point, deleted documents included) that still fit the budget: more than 8,259,552 points of 16 dims x 16 bytes (the widest shape a FieldType allows), about 179M LatLonPoint values (12 bytes each) or about 67M LatLonShape triangles (32 bytes each). At 2200 MB the window for 16x16-byte points is 8,259,553 to 8,872,566 points; larger merges spill to disk at once (executed), though a later radix-selector buffer can still overflow, depending on the data (model only). For users who raised the budget for big point fields, ordinary growth can reach the window.
• What happens (executed on main a9779ef3c193 with -ea and -da and on 10.5.2; the BKD, points-writer and IndexWriter code is identical at 645d4fe14a20): forceMerge(1) of 84,000 documents x 100 points of a 16x8x16 field throws NegativeArraySizeException: -2110967296; isOpen() is false and getTragicException() is that exception; 1,000 uncommitted documents and an uncommitted delete are rolled back; later calls throw AlreadyClosedException; two fresh writers with the same budget fail again on their next ordinary merge.
• What does not happen. The last commit stays readable and CheckIndex reports it clean; a 2000 MB budget merges the same segments into one correct segment of 8,400,000 points. No wrong or extra hits, no hang, and no query path is involved; the overflowing allocation fails before reserving its oversized length.
• Likelihood: rare. It needs both the unusual budget and the large field. It is still worth fixing: the value is accepted, the failure is late, raw and tragic, and the fix is one statement.

The test builds the BKDWriter that Lucene90PointsWriter(state, 512, 2200.0) creates for each field it merges: BKDConfig(16, 8, 16, 512) (260 bytes per point), budget 2200, totalPointCount 8,400,000. totalPointCount is documented as an upper bound, so a single add() is legal. It uses only public constructors and methods; the package is the one of the upstream TestBKD. The test's own code draws no random values, and seeds DEADBEEF and 5EED1234 give the same failure.

It needs Lucene 10.0 or later as written (BKDConfig became a record in 10.0.0); the follow-up's version-specific check covers 8.x and 9.x. Before pasting it, run ./gradlew tidy: google-java-format wraps the long throw new AssertionError(...) line, the only change, and the tidied test behaves the same.

As a regression test it is the same test: it fails on stock and passes with the recommended fix, where the writer spills its one point to a temporary file. The IndexWriter consequence (a forceMerge(1) that closes the writer) needs about 2.1 GB of real points and a 3 GB heap, so it is not part of the test; it is described under "Who is affected".

# gradle, main 645d4fe14a20, seed DEADBEEF (the same with -Ptests.asserts=false)
TestBKDWriterSortBufferOver2GB > testAddWithSortBufferAbove2048MB FAILED
    java.lang.AssertionError: expected add() to keep the point on heap or spill it, got java.lang.NegativeArraySizeException: -2110967296
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:53C6FD9796C8AA09]:0)
        at org.apache.lucene.util.bkd.TestBKDWriterSortBufferOver2GB.testAddWithSortBufferAbove2048MB(TestBKDWriterSortBufferOver2GB.java:33)

        Caused by:
        java.lang.NegativeArraySizeException: -2110967296
            at org.apache.lucene.util.bkd.HeapPointWriter.<init>(HeapPointWriter.java:43)
            at org.apache.lucene.util.bkd.BKDWriter.initPointWriter(BKDWriter.java:231)
            at org.apache.lucene.util.bkd.BKDWriter.add(BKDWriter.java:253)
            at org.apache.lucene.util.bkd.TestBKDWriterSortBufferOver2GB.testAddWithSortBufferAbove2048MB(TestBKDWriterSortBufferOver2GB.java:31)

# the standalone runner gives the same message on main (-ea, -da) and on the 10.0.0 (-ea) and 10.6.0 (-ea, -da) jars
# with the recommended fix: the test passes (assertions on and off)

• The javadoc. Class javadoc: it "uses up to the specified {@code maxMBSortInHeap} heap space for writing" (BKDWriter.java:74-76); DEFAULT_MAX_MB_SORT_IN_HEAP is the "Default maximum heap to use, before spilling to (slower) disk" (:98-99); totalPointCount is "An upper bound on how many points the caller will add (includes deletions)" (:135). That creates a reasonable expectation of a spill rather than a crash; no javadoc or test states an upper bound.
• Validation. The constructor rejects negative budgets and, with a clear IllegalArgumentException, budgets too small for one leaf (:198-216), but has no upper check. The public 3-argument Lucene90PointsWriter constructor that takes the budget belongs to a class with no @lucene.experimental tag (Lucene90PointsWriter.java:40-41; nor has BKDConfig).
• The same limit is guarded elsewhere. OfflineSorter.BufferSize ("Buffer too large for Java (2047MB max)", OfflineSorter.java:84-91); IndexWriterConfig.setRAMPerThreadHardLimitMB rejects >= 2048 MB (IndexWriterConfig.java:382-386); inside BKDWriter, MergeIntersectsVisitor.grow checks MAX_ARRAY_LENGTH (:370-377), checkMaxLeafNodeCount too (:904-911), and initPointWriter uses Math.toIntExact (:231). Only the multiply in HeapPointWriter was missed.
• The suggested route. In #8825 (LUCENE-7774, 2017) mikemccand answered a request for configurable BKD buffers: "The Lucene60PointsWriter ctor already accepts maxMBSortInHeap, which it passes onto BKDWriter, so users could make a custom codec changing this setting themselves?"; the reporter (romseygeek) accepted that and closed the issue. (In 2017 budgets of 2048 MB or more were still rejected up front.)
• History. From 6.1.0 to 8.0.0 the constructor built an OfflineSorter.BufferSize from the budget and rejected 2048 MB or more (6.0.0 did so only when sorting offline). LUCENE-8673 removed that call and LUCENE-8699 introduced the single int-sized array, in the same release (8.1.0); no commit, issue or PR review of those changes mentions the limit. A lost guard, not a design decision.
• Precedent. Int overflows at other BKD sites were fixed as bugs: #11713 (LUCENE-10678, the merge partition point) and #11907 (splitPackedValues).

The strongest case for "intended or unsupported" (the intended-behaviour devil's advocate and the cross-model reviews): the BKD classes are @lucene.experimental (BKDWriter, Lucene90PointsFormat, PointsWriter) or @lucene.internal (HeapPointWriter, BKDRadixSelector); a 2 GB+ sort budget is untested; committers declined new codec knobs (#10966, LUCENE-9927, Won't Fix), and in #8825 rmuir said it "would be better to speed up e.g. offlinesorter ... so that large amounts of ram are not necessary"; and one Java array cannot hold 2 GB anyway.

Why it fails: Lucene90PointsWriter, whose public constructor takes the budget, carries no such tag; experimental means the API may change, not that a value it accepts may crash a merge; #10966 declined adding knobs and did not withdraw the existing public parameter; "cannot be one array" is exactly why the writer must cap or reject the value, as OfflineSorter did for this same parameter until 8.0.0; and the failure is a raw NegativeArraySizeException deep inside a merge that closes the writer, not a documented error. Verdict: not intended, a defect (the independent reproduction, the cross-model review, the intended-behaviour check and the adjudication agree).

One fix was run through gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF); it is the hunt's candidate fix, unchanged. Two alternatives are described, not run.

In the BKDWriter constructor (BKDWriter.java:195-196). BKDRadixSelector receives the same value (BKDWriter.java:975), so its heap writers are covered too.

-    // Maximum number of points we hold in memory at any time
-    maxPointsSortInHeap = (int) ((maxMBSortInHeap * 1024 * 1024) / (config.bytesPerDoc()));
+    // Maximum number of points we hold in memory at any time; the on-heap buffers are single byte[]
+    // of bytesPerDoc * points, so never more than the largest Java array:
+    maxPointsSortInHeap =
+        (int)
+            Math.min(
+                (maxMBSortInHeap * 1024 * 1024) / (config.bytesPerDoc()),
+                ArrayUtil.MAX_ARRAY_LENGTH / config.bytesPerDoc());

Trade-offs

• No hot-path cost: it runs once per writer.
• Budgets of 2048 MB or more stay accepted, but no single in-heap buffer exceeds about 2 GiB; larger inputs spill to disk, which the javadoc's "up to" allows.
• Behaviour change: the existing "at least one leaf fits" check now also rejects a maxPointsInLeafNode above MAX_ARRAY_LENGTH / bytesPerDoc (8,259,553 or more for 260-byte points), whose leaf buffers could not be allocated anyway (a static argument; not executed).
• A smaller one: ArrayUtil.MAX_ARRAY_LENGTH is Integer.MAX_VALUE minus the array header, 2,147,483,631 (ArrayUtil.java:32-34), so a fractional budget above 2,147,483,631 bytes (just under 2048 MB) can also be capped a few points lower than today; a total that stayed on heap then spills (arithmetic; for 260-byte points both limits give 8,259,552).

What the checks validated

• Stock: the repro fails with assertions on and off.
• Fix: the repro passes with assertions on and off.
• Closest suites (org.apache.lucene.util.bkd.*, TestLucene90PointsFormat, TestLucene90PointsFormatV0, TestPointValues, TestPointQueries, and the repro): 13 classes, 193 tests, 0 failures, 11 skipped (seed DEADBEEF; the fact check repeated it with seed 5EED0613).
• :lucene:core:check -x test passes with the fix and the tidied repro in the tree (the fix needed no reformatting). The full core suite was not run.
• Earlier, the same diff shadowed onto the stock jars made the finder's 4 tests pass, including the end-to-end forceMerge(1) of 8.4M points (which then spills to disk), and the independent reproducer's 6 tests; 57 upstream BKD and points-format tests passed with it (one seed).

In verifyParams, throw IllegalArgumentException when maxMBSortInHeap * 1024 * 1024 exceeds ArrayUtil.MAX_ARRAY_LENGTH, as OfflineSorter.BufferSize and the BKDWriter of 6.1.0-8.0.0 did.

Trade-offs

Loud and explicit, and it restores the pre-8.1.0 behaviour, but it rejects configurations that avoid this overflow today (merges under 2 GiB per field never reach the overflowing multiply; code reading, not run), and the regression test would have to expect the exception from the constructor instead.

Compute the buffer size with Math.multiplyExact so the failure is an ArithmeticException with a clear cause.

Trade-offs

The merge still fails and closes the writer; useful only as defence in depth together with the cap.

Method: non-formal (code reading, then an executed test), a hunt unit on the BKD write path in non-default configurations.

• While writing the unit's bounded-exhaustive translation-validation harness, which runs at budgets of 16 MB or less and so cannot witness this, the AI tool read HeapPointWriter's allocation and saw that BKDWriter never caps the count it passes.
• A JUnit on the stock main jar confirmed it: a direct BKDWriter.add at 2200 MB and at 4000 MB, and an end-to-end forceMerge(1) through a custom codec; a 2000 MB control spilled to disk and passed.
• An SMT (Z3) model written afterwards (every int-sized heap buffer of the BKD write path as a linear obligation, over all 256 field shapes; 19 of 19 queries as expected) gave the exact region: budgets of 2048 MB or more, the minimal point count per shape, and the sibling sites in BKDRadixSelector, switchToHeap and OneDimensionBKDWriter.