A delete in a segment of more than 2,147,482,624 documents can never be committed: the live-docs batch loop's int offset wraps (CheckIndex has the same loop)

Lucene90LiveDocsFormat.writeBits, which IndexWriter runs whenever it saves a segment's live-docs bitmap, copies the bits in batches of 1,024 with for (int offset = 0; offset < bits.length(); offset += copy.length()). In a segment of more than 2,147,482,624 documents the step after the last batch overflows the int: at IndexWriter.MAX_DOCS (2,147,483,519) the offset goes from 2,147,482,624 to -2,147,483,648, which still passes the loop test, so a phantom batch of -129 bits is requested and FixedBitSet.clear throws. On 10.4.0 and later the first hard delete in such a segment can therefore never be committed. With assertions off (production) commit() throws, every later commit() throws again and close() throws and rolls the writer back; with assertions on, the AssertionError is a tragic event that closes the writer at once.

CheckIndex.bitsCardinality has had the same loop since 10.3.0, so CheckIndex at level 2 or higher, or a direct CheckIndex.testLiveDocs call, reports the valid live docs of such a segment as an error. The failure is always loud: no wrong bits are written or counted, and the last commit is untouched. It takes one segment within 894 documents of the hard limit (one of the top 895 legal sizes, up to and including MAX_DOCS), so the priority is low. A long loop variable fixes both copies.

1. The loop. Lucene90LiveDocsFormat.writeBits (Lucene90LiveDocsFormat.java:196-216) uses a 1,024-bit scratch set copy (:200) and loops for (int offset = 0; offset < bits.length(); offset += copy.length()) (:201). Each pass computes numBitsToCopy = Math.min(bits.length() - offset, copy.length()) (:202), sets all 1,024 bits, clears the bits past the end with copy.clear(numBitsToCopy, copy.length()) (:206, only for a short batch), applies the live docs with bits.applyMask(copy, offset) (:208) and writes the words (:210-213).
2. The callers. writeLiveDocs calls it at :180. IndexWriter reaches writeLiveDocs through PendingDeletes.writeLiveDocs:193 on commit (ReaderPool.commit:354 via ReadersAndUpdates.writeLiveDocs:291, executed) and when readers are not pooled (ReaderPool.release:205, code reading). Separately, a newly flushed segment that already carries deletions calls the format directly, without PendingDeletes (DocumentsWriterPerThread.java:697, code reading).
3. The numbers at maxDoc = 2,147,483,519. The batch at offset 2,147,482,624 (2^31 - 1024) is the real last one: 895 bits, copied and written correctly. Then offset += 1024 = 2,147,483,648 = 2^31 wraps to -2,147,483,648. The test -2,147,483,648 < 2,147,483,519 holds, so a phantom pass starts: bits.length() - offset = 4,294,967,167 wraps to -129, Math.min(-129, 1024) = -129, and copy.clear(-129, 1024) runs.
4. The exception. With assertions on, FixedBitSet.clear fails its assert (FixedBitSet.java:799-800): startIndex=-129, numBits=1024. With assertions off, startWord = -129 >> 6 = -3 and bits[startWord] (:821) throws Index -3 out of bounds for length 16 (the scratch set has 16 longs).
5. Which sizes. The loop reaches offset 2^31 - 1024 only for a length above 2^31 - 1024, and the step after it always wraps. So maxDoc from 2,147,482,625 to 2,147,483,519 fails: the top 895 legal sizes. 2,147,482,624 is fine. A Z3 bit-vector query proved that for every length up to 2^31 - 1024 the step from any 1,024-aligned in-loop offset never wraps into a running pass, and that above that bound the wrapped pass always has a negative numBitsToCopy. The model covers only this step condition; reading the loop, without a wrap it ends normally.
6. No malformed file is left (on the normal path). The phantom pass resets the scratch set and then throws, before applyMask applies any live-docs bit and before it writes another word; the earlier batches have already written most of the .liv file. PendingDeletes.writeLiveDocs then deletes that partial file and advances only the next write generation (PendingDeletes.java:184-203); the segment keeps its previous delete generation and the deletions stay pending in memory. An executed probe shows exactly that, and a second attempt throws again. The format itself does no cleanup, so a direct caller of LiveDocsFormat.writeLiveDocs would be left with the partial file (code reading).
7. The CheckIndex copy. CheckIndex.bitsCardinality (CheckIndex.java:1369-1383) has the identical loop (:1372, clear at :1377). The public static CheckIndex.testLiveDocs (:1304) calls it at :1317 for a segment with deletions; checkIndex runs that test only at level 2 or higher (:1114-1116); the command-line default is level 1 and a CheckIndex built in code starts with no level set. With failFast off (the default) the error makes checkIndex throw "Live docs test failed" (:1165) and count the segment as broken (:978-980); with failFast on, testLiveDocs rethrows it (:1350-1351). This half was executed only through direct CheckIndex.testLiveDocs calls, never through a full checkIndex() run on a persisted index of this size, so the level gate and these consequences are code reading.
8. The contract it breaks. Bits#applyMask: "offset must be non-negative" (Bits.java:43); here Lucene's own loop produces the negative value.

• Who can be hit. One segment with maxDoc from 2,147,482,625 to 2,147,483,519. Since the whole index is capped at MAX_DOCS, that is an index or shard filled to within 894 documents of the limit and held in one segment (all other segments together then hold at most 894 documents). Lucene's own @Monster test testExactlyAtTrueLimit builds exactly such a segment with forceMerge(1). Natural merges rarely get there: the default TieredMergePolicy merges by byte size and keeps merged segments under about 5 GiB, about 2.5 bytes per document at this count (the end-to-end run's documents are empty); forceMerge(1) overrides that cap.
• Default, not opt-in. An otherwise default IndexWriterConfig with the current default codec (Lucene104Codec, pinned explicitly in the test because the test framework randomizes it) reproduces it end to end; Lucene90LiveDocsFormat is the live-docs format of the default codec.
• Writer half (10.4.0 and later). A hard deletion that removes a document of such a segment and leaves the segment in place (deleteDocuments by term or query, updateDocument, tryDeleteDocument, or hard deletes next to soft deletes) makes commit() fail. deleteAll(), and deleteDocuments(new MatchAllDocsQuery()), which IndexWriter turns into deleteAll(), drop the segment instead. With assertions off (production) the writer stays open with the deletions pending, every later commit() fails the same way, and close() fails and then rolls the writer back (executed: commit and close fail). With assertions on, the AssertionError is treated as a tragic event and the writer rolls back and closes at once (code reading). The last commit is untouched, so a reader reopened from it shows the deleted documents again, as after any failed commit (inferred). Soft-delete-only updates should not reach the loop, because PendingSoftDeletes.writeLiveDocs writes only hard deletes (code reading).
• CheckIndex half (10.3.0 and later). CheckIndex at level 2 or higher, or a direct CheckIndex.testLiveDocs call, reports the valid live docs of such a segment as an error. On 10.4.0+ the writer fails first, so such a .liv file comes from an earlier writer whose loop is still correct (9.x or 10.0-10.3.x, for an index the checking version can still open), or the check meets an NRT reader whose deletions are in memory. A MAX_DOCS .liv file written by 10.3.2 and validated bit by bit is reported as an error by CheckIndex.testLiveDocs on 10.3.0, 10.3.2, 10.4.0, 10.5.2 and main (direct calls only; the full checkIndex() path is code reading).
• Operator warning. CheckIndex -exorcise at level 2 or 3 on such a segment (10.3.0 or later) would treat it as broken and write a new segments_N without it, dropping up to 2.1 billion healthy documents (code reading, not executed).
• Loud, not silent. Every failure is an exception; no wrong bits, counts or search results were observed. The scale is extreme, so the priority is low.

The test sits in org.apache.lucene.codecs.lucene90 and calls the same writeLiveDocs that PendingDeletes.writeLiveDocs calls on commit. It instantiates the format directly because LuceneTestCase randomizes the default codec (the Codec argument of SegmentInfo is only stored), and writes to an FSDirectory so the 256 MiB .liv file stays off the heap; it runs under the default 512 MB test heap. The 256 MiB bitmap is inherent: the wrap needs more than 2^31 - 1024 bits. It asserts only that the write succeeds. It covers the writer half; the CheckIndex half is the same loop with the same fix and was shown by the checkers' own tests.

End to end, with public API only (FSDirectory, new IndexWriterConfig().setCodec(TestUtil.getDefaultCodec()), an otherwise default config pinned to the default codec): 2,147,483,519 addDocument calls (the test does not assert the segment layout; the failure shows the segment holding the deleted document has more than 2,147,482,624 of them), commit(), deleteDocuments by term, commit(). That run takes 10-16 minutes and a 3 GB heap; the second commit() fails with the same exception from PendingDeletes.writeLiveDocs, and with assertions off close() throws it again.

As the regression test: the same test. Because it allocates 256 MiB of heap and disk, it may suit @Monster or @Nightly.

# stock 645d4fe14a20, gradle, seed DEADBEEF, assertions on
TestLiveDocsWriteMaxDocs > testWriteLiveDocsOfMaxDocsSegment FAILED
    java.lang.AssertionError: writeLiveDocs with maxDoc=IndexWriter.MAX_DOCS threw java.lang.AssertionError: startIndex=-129, numBits=1024
        Caused by:
        java.lang.AssertionError: startIndex=-129, numBits=1024
            at org.apache.lucene.util.FixedBitSet.clear(FixedBitSet.java:799)
            at org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat.writeBits(Lucene90LiveDocsFormat.java:206)
            at org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat.writeLiveDocs(Lucene90LiveDocsFormat.java:180)
            at org.apache.lucene.codecs.lucene90.TestLiveDocsWriteMaxDocs.testWriteLiveDocsOfMaxDocsSegment(TestLiveDocsWriteMaxDocs.java:55)
1 test completed, 1 failed

# stock, -Ptests.asserts=false
    java.lang.AssertionError: writeLiveDocs with maxDoc=IndexWriter.MAX_DOCS threw java.lang.ArrayIndexOutOfBoundsException: Index -3 out of bounds for length 16
        Caused by:
        java.lang.ArrayIndexOutOfBoundsException: Index -3 out of bounds for length 16
            at org.apache.lucene.util.FixedBitSet.clear(FixedBitSet.java:821)
            at org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat.writeBits(Lucene90LiveDocsFormat.java:206)
            at org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat.writeLiveDocs(Lucene90LiveDocsFormat.java:180)
1 test completed, 1 failed

# with the long-offset fix, assertions on and off
:lucene:core:test (SUCCESS): 1 test

# end to end through IndexWriter (main ea57360912ed, assertions off, 617 s)
testDeleteOneDocumentAndCommit: java.lang.ArrayIndexOutOfBoundsException: Index -3 out of bounds for length 16
   org.apache.lucene.util.FixedBitSet.clear(FixedBitSet.java:821)
   org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat.writeBits(Lucene90LiveDocsFormat.java:206)
   org.apache.lucene.codecs.lucene90.Lucene90LiveDocsFormat.writeLiveDocs(Lucene90LiveDocsFormat.java:180)
   org.apache.lucene.index.PendingDeletes.writeLiveDocs(PendingDeletes.java:193)
   [...]
   Suppressed (from IndexWriter.close()): the same exception, via ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:291)

• The only limit. "Hard limit on the maximum number of documents that may be added to the index." (IndexWriter.java:201-204). No javadoc, CHANGES or MIGRATE entry sets a smaller per-segment limit, and LiveDocsFormat.writeLiveDocs ("Persist live docs bits.") states no size restriction.
• Lucene builds and tests this size. TestIndexWriterMaxDocs.testExactlyAtTrueLimit (@Monster) adds MAX_DOCS documents and calls forceMerge(1); close() commits the merged segment, though the test's reads open only the commit made before the merge. BaseLiveDocsFormatTestCase.testOverflow (:75-78) calls testSerialization(IndexWriter.MAX_DOCS, IndexWriter.MAX_DOCS - 7, false), a write and read-back of MAX_DOCS live docs. It came with LUCENE-8303 (8dc69428e34, 2018-05-09) and two days later was kept as @Monster rather than removed ("LUCENE-8303: Make the overflow test a Monster rather than Nightly.", d764156f913). With monster tests enabled (-Ptests.monster=true -Ptests.heapsize=3g; the default test heap was not tried) it fails on stock 645d4fe14a20 and passes with the fix.
• Speed-ups, not a new limit. #14998 ("Speed up flushing of live docs.") replaced a per-word writer loop and #15045 ("Use FixedBitSet#cardinality for counting liveDocs in CheckIndex") a per-bit count; neither old loop can overflow at MAX_DOCS, and neither PR discusses a size limit.
• Its own callee's contract. Bits#applyMask requires a non-negative offset (Bits.java:43); the negative value is produced by Lucene's loop, not by a caller.

The intended-behaviour checker and the works-as-designed reviewer built the strongest case: segments this close to the limit are unsupported in practice (with the default TieredMergePolicy only forceMerge or tiny documents build one), monster tests do not run in regular CI, LiveDocsFormat and CheckIndex are @lucene.experimental, the unit repro calls the codec directly, CheckIndex runs the check only at level 2 or higher, deleteAll() still works, and with assertions on the writer closes itself rather than staying stuck.

Why it fails: these points lower reach and severity, but none is a contract. The end-to-end run reaches the bug through IndexWriter with defaults, the kept @Monster tests show that a one-segment MAX_DOCS index and its live-docs round trip are meant to work, and the negative offset comes from Lucene's own loop. Both reviewers that rated the argument rated it weak, and every checker found it real.

One option was built and validated; the other two are sketches that were not run.

Iterate with long longOffset and derive final int offset = (int) longOffset inside the body, in Lucene90LiveDocsFormat.writeBits and in CheckIndex.bitsCardinality (the same change in both).

FixedBitSet copy = new FixedBitSet(1024);
// long loop variable: offset += 1024 would overflow int for lengths > 2^31 - 1024
for (long longOffset = 0; longOffset < bits.length(); longOffset += copy.length()) {
  final int offset = (int) longOffset;
  int numBitsToCopy = Math.min(bits.length() - offset, copy.length());
  ... // unchanged
}

Trade-offs

The smallest change; behaviour is unchanged for every length up to 2^31 - 1024, where the old loop's step provably never wraps. One cast per 1,024 bits; no benchmark was run. The two copies of the loop remain. Both loops are on the 10.4+ branches; branch_10_3 has the CheckIndex loop only.

What the checks validated

Gradle in a scratch worktree of 645d4fe14a20 (2026-10-31):

• stock: the repro fails with assertions on and off; upstream TestLucene90LiveDocsFormat.testOverflow with -Ptests.monster=true -Ptests.heapsize=3g fails at writeBits:206;
• with the fix: the repro passes with assertions on and off; testOverflow passes (86 s); codecs.lucene90.*, TestLiveDocs, TestFixedBitSet, TestCheckIndex, TestPendingDeletes, TestPendingSoftDeletes, TestIndexWriterDelete and TestIndexWriterMaxDocs: 26 classes, 940 tests, 0 failures (86 skipped); :lucene:core:check -x test passes.
• Earlier, every checker's tests passed with it, including a correct read-back at MAX_DOCS.

for (int offset = 0, remaining = bits.length(); remaining > 0; offset += 1024, remaining -= 1024) with numBitsToCopy = Math.min(remaining, 1024): remaining never overflows, and the wrapped final offset is never used.

Trade-offs

Stays all-int, but is a slightly larger rewrite and was not validated.

Move the batch walk into one helper that writeBits and bitsCardinality both call, with either fix inside, so a future fix cannot miss one copy (the loop was copied from one PR into the other).

Trade-offs

A new internal method for a two-line fix; harder to backport.

Method: semi-formal.

• While modelling the Bits#applyMask contract, the finder read every caller and saw the int offset += 1024 loops in Lucene90LiveDocsFormat.writeBits and CheckIndex.bitsCardinality.
• A Z3 query over 32-bit bit-vectors gave a counterexample (length 2,147,483,136 at offset 2,147,482,624), the smallest failing length 2,147,482,625 (895 legal maxDoc values up to MAX_DOCS), a proof that for every length up to 2^31 - 1024 the step from any aligned in-loop offset never wraps, and a proof that the wrapped pass always clears from a negative index.
• Same-package unit tests then failed on the stock main jars with assertions on and off, with the 2^31 - 1024 controls passing, and a legal end-to-end run (default IndexWriterConfig and codec, MAX_DOCS documents in one segment, one delete, commit) failed in 617 s (assertions off) and 935 s (on).
• Independent checkers then wrote their own tests from the claim (see the checks).