After a non-tragic addDocument failure, the next document resets the delete slice and the next commit silently lacks an acknowledged delete

After an in-memory segment (a DWPT) has indexed a new document, and before addDocument returns, DocumentsWriterPerThread.finishDocuments applies to the documents buffered before it every delete and doc-values update queued since the segment's last document, its slice of the writer-wide delete queue (DocumentsWriterPerThread.java:291, :300-301, :361-390). DeleteSlice.apply walks the slice node by node and empties it only after the last node (DocumentsWriterDeleteQueue.java:348-367). When a node throws, the slice keeps its unapplied nodes, and the exception is not aborting: finishDocuments runs after allDocsIndexed = true and outside any onAbortingException guard, so isOpen() stays true and getTragicException() null. On the segment's next document updateSlice finds no new operation, and finishDocuments calls deleteSlice.reset() (:385), which marks the unapplied nodes done. The segment flushes without them, and the next commit() returns a sequence number that covers an acknowledged delete it lacks. Since 9.8.0 a legal workload reaches the throw: #12392 made ByteBlockPool.nextBuffer throw ArithmeticException at 2 GiB (ByteBlockPool.java:199), and an idle segment's copy of 2.2 million doc-values updates on a 1 KB term gets there. On 9.7.0 the same workload closes the writer. Treating a throw out of finishDocuments as aborting, as the indexing chain does for a byte-pool overflow while indexing, fixes it.

// DocumentsWriterPerThread.finishDocuments, main 5adff78bb168      DocumentsWriterPerThread.java:361
if (deleteNode != null) {                                         // :374  updateDocument, delete by term
  seqNo = deleteQueue.add(deleteNode, deleteSlice);               // :375
  deleteSlice.apply(pendingUpdates, docIdUpTo);                   // :377
  return seqNo;
} else {
  seqNo = deleteQueue.updateSlice(deleteSlice);                   // :380  negative: new operations arrived
  if (seqNo < 0) {
    seqNo = -seqNo;
    deleteSlice.apply(pendingUpdates, docIdUpTo);                 // :383  throws here (addDocument(next))
  } else {
    deleteSlice.reset();                                          // :385  drops what :383 never applied
  }
}

// DocumentsWriterDeleteQueue.DeleteSlice.apply                   DocumentsWriterDeleteQueue.java:348
Node<?> current = sliceHead;
do {
  current = current.next;
  current.apply(del, docIDUpto);                                  // :364  one node per delete or update
} while (current != sliceTail);
reset();                                                          // :366  only after the last node
// reset(): sliceHead = sliceTail;                                // :369-372

1. addDocument(buffered) leaves the thread's in-memory segment with one document and an empty slice.
2. Each updateNumericDocValue call and the deleteDocuments call append one node to the writer-wide queue. The writer-wide buffer is frozen and applied whenever it reaches the 16 MB RAM buffer, but the segment makes its own copy only at its next document, so its slice spans all 2,200,001 nodes. Nothing counts that backlog (LFV-1077): ramBytesUsed() reads 2,555,912 bytes.
3. addDocument(next): updateSlice moves sliceTail to the queue tail and returns a negative sequence number (DocumentsWriterDeleteQueue.java:284-293). apply copies every update, term bytes included, into the segment's private BufferedUpdates (BufferedUpdates.java:134-144). About 2 GiB in, ByteBlockPool.nextBuffer throws at byteOffset = Math.addExact(byteOffset, BYTE_BLOCK_SIZE) (ByteBlockPool.java:199). Executed stack, from there up to finishDocuments: ByteBlockPool.nextBuffer:199 <- ByteBlockPool.append:301 <- ByteBlockPool.append:228 <- BytesRefArray.append:70 <- FieldUpdatesBuffer.append:215 <- FieldUpdatesBuffer.addUpdate:184 <- BufferedUpdates.addNumericUpdate:139 <- DocValuesUpdatesNode.apply:507 <- DeleteSlice.apply:364 <- DocumentsWriterPerThread.finishDocuments:383. apply never reaches its reset(), so the slice still holds the rest of the updates and the delete of buffered.
4. The exception is not aborting: maybeAbort (DocumentsWriterPerThread.java:310, :594-605) acts only on an exception passed to onAbortingException, and nothing passes this one. DocumentsWriter.updateDocuments skips doAfterDocument (DocumentsWriter.java:434) and returns the segment to the pool (:444). Executed: isOpen()=true, getTragicException()=null, numRamDocs()=2.
5. addDocument(later): nothing was queued since step 3, so updateSlice returns a positive number and finishDocuments calls reset() (:385): sliceHead = sliceTail. The delete of buffered is gone from the segment. doAfterDocument then sees the segment's 2 GiB of pending updates and flushes it (executed: numRamDocs() 2 -> 0).
6. A delete reaches a document that is still buffered only through its segment's slice: the delete queue's javadoc says each DWPT keeps its own head of the queue, for "its segments private deletes" (DocumentsWriterDeleteQueue.java:34-38), while the writer-wide slice records deletes "against all prior (already written to disk) segments" (:73-75). Executed: the committed index still has buffered. commit() returns 2,200,005.

• In normal operation that reset() changes nothing: after every successful apply the slice is already empty (:366), and a positive updateSlice means nothing was queued. It acts only after a failed apply; a bounded exhaustive model of the slice protocol found no other path within its bound (every sequence of up to 7 operations on one modelled segment).
• One transient fault is enough: a fault-injection check that makes one queue node throw once (no overflow, 128 MB heap, through the real IndexWriter/DWPT path) loses the delete the same way (delete sequence number 3 missing from commit 7). It injects the fault through package-private writer internals, so it is evidence for the mechanism, not a public-API repro.
• Read, not executed: for updateDocument and delete by term, add(node, slice) sets slice.sliceTail (DocumentsWriterDeleteQueue.java:166) before tryApplyGlobalSlice() (:168), so a throw there (the writer-wide buffer's overflow) leaves the same unapplied slice for the same reset(). On that path later freezes of the broken writer-wide buffer fail too, so a loss would not be the only symptom.

• What is lost. Deletes, and by code also doc-values updates, of documents buffered in that in-memory segment that were queued behind the failing node. Deletes acknowledged after the failure are applied (executed with the delete-term trigger, 75,000 distinct 30,000-byte delete terms: of three acknowledged deletes, buffered before the failure and b2, b3 after it, only buffered, queued behind the failing node, is missing). CheckIndex is clean, on that commit and on the commit of this doc's doc-values workload, but it checks structure only; it does not see a missing delete. By code, deletes of documents in already-flushed segments go through the writer-wide buffer, not this slice.
• Silent. The exception surfaces on a later addDocument that has nothing to do with the delete, and the writer's own health signals say nothing is wrong: isOpen() is true and getTragicException() null. The addDocument javadoc promises that after an exception "the index will be consistent, but this document may not have been added" (IndexWriter.java:1465-1466); it does not tell the caller to stop using the writer or to roll back.
• Default vs opt-in. An otherwise default IndexWriterConfig (the test pins the default codec with TestUtil.getDefaultCodec()), a plain ByteBuffersDirectory, one thread, assertions on or off. The workload calls only public IndexWriter API; the test class itself uses the test framework (LuceneTestCase, TestUtil).
• Scale, the counterweight. About 2^31 bytes of buffered update or delete terms must pass one in-memory segment that holds documents but receives none, with no commit, NRT reopen or flush in between. Executed: 2.2 million updateNumericDocValue calls on one 1 KB term (3 GB heap), or 75,000 deleteDocuments calls with distinct 30,000-byte terms (8 GB heap, 9.9.0 and later). The writer's accounting does not count the backlog (LFV-1077), and neither safeguard stops the private copy: when buffered deletes reach the 16 MB RAM buffer the writer freezes and applies its writer-wide buffer (FlushByRamOrCountsPolicy.java:45-53), which does not touch the idle segment's slice; the flush-by-RAM check and the 1,945 MB per-thread hard limit (IndexWriterConfig.java:93) both run in doAfterDocument (DocumentsWriterFlushControl.java:215-251), which is reached only after finishDocuments returned (DocumentsWriter.java:428, :434). They first see the 2 GiB copy at addDocument(later), after the reset, and the segment is flushed then. Likelihood: contrived.
• The same throw site in production (related, not a report of this loss): elastic/elasticsearch#160939 (Lucene 10.5.1) shows the same non-tragic ArithmeticException out of DeleteSlice.apply inside finishDocuments, on the writer-wide-buffer path; its reporter says it is still hitting production.

The codec is pinned with TestUtil.getDefaultCodec(), the directory is a plain ByteBuffersDirectory, and nothing on the failing path is randomized, so the mechanism does not depend on the seed: the recorded runs use DEADBEEF, and an independent check also failed it with seeds CAFEBABE and 12345678. The test sits in org.apache.lucene.index like the upstream tests and passes :lucene:core:check -x test (formatting, ecj lint, forbidden APIs). Run it with -Ptests.monster=true -Ptests.heapsize=3g.

On the released 9.7.0 it stops at expectThrows: there addDocument(next) throws ArrayIndexOutOfBoundsException: Index -65536 instead and the writer closes itself, so nothing is lost silently.

Regression test: this same test. It passes with the recommended fix. Because it expects the ArithmeticException, it tests fixes that keep the exception and close the writer (fix A); a fix that prevents the overflow (fix C) or throws another exception type would stop at expectThrows and would need that line changed.

TestDeleteLostAfterFailedAddDocument > testDeleteSurvivesFailedAddDocument FAILED
    java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:52162A1781E7F99A]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.index.TestDeleteLostAfterFailedAddDocument.testDeleteSurvivesFailedAddDocument(TestDeleteLostAfterFailedAddDocument.java:51)

# the same steps as a plain main() (numbers only, assertions off; the probe prints only these 3 stack frames, frames filtered)
addDocument(buffered) seqNo=1
2,200,000 updateNumericDocValue calls: last seqNo=2200001 ramBytesUsed=2555912 (max sampled 15954696)
deleteDocuments(id:buffered) seqNo=2200002
addDocument(next) threw java.lang.ArithmeticException: integer overflow
    at org.apache.lucene.util.ByteBlockPool.nextBuffer(ByteBlockPool.java:199)
    at org.apache.lucene.index.DocumentsWriterDeleteQueue$DeleteSlice.apply(DocumentsWriterDeleteQueue.java:364)
    at org.apache.lucene.index.DocumentsWriterPerThread.finishDocuments(DocumentsWriterPerThread.java:383)
after the failure: isOpen=true tragic=null numRamDocs=2
addDocument(later) seqNo=2200004 numRamDocs=0
commit() seqNo=2200005
committed index: numDocs=3 maxDoc=3 buffered=1 next=1 later=1   (the delete of buffered expects buffered=0)

# with fix A
after the failure: isOpen=false tragic=java.lang.ArithmeticException: integer overflow

• commit(). "it returns the sequence number such that all indexing operations prior to this sequence will be included in the commit point" and "All sequence numbers <= this value will be reflected in the commit, and all others will not." (IndexWriter.java:4202-4204, :4207-4209). The class javadoc introduces sequence numbers as "describing which changes are in the commit point and which are not" (IndexWriter.java:117-121).
• addDocument. "if an Exception is hit (for example disk full) then the index will be consistent, but this document may not have been added." (IndexWriter.java:1465-1466). The exception may cost this document, not an earlier acknowledged delete.
• The design rule. LUCENE-5987: "IndexWriter will now forcefully close itself on aborting exception (an exception that would otherwise cause silent data loss)." (lucene/CHANGES.txt:11614-11616). The indexing chain follows it for a byte-pool overflow while indexing a document's terms: "If we hit an exception in here, we abort all buffered documents since the last flush, on the likelihood that the internal state of the terms hash is now corrupt" (IndexingChain.java:2068-2073; catch (Throwable th) { onAbortingException(th); throw th; } at :2098-2100). finishDocuments is the gap. An older precedent: LUCENE-938 "Fixed cases where an unhandled exception in IndexWriter's methods could cause deletes to be lost." (lucene/CHANGES.txt:20501-20503).
• The reset() was not written for this case. It came with LUCENE-5006 (trunk 75ab79f0aa3, 2013-08-19, "if we don't need to apply we must reset!"; release 4.5.0 got it through the branch_4x commit 8003cc78298), where it also skipped applying while the segment had no documents (applySlice = numDocsInRAM != 0). The block path (updateDocuments) got today's form, a reset only when nothing new was queued, in 76fb616bdf5 (2016); the single-document path kept applySlice = numDocsInRAM != 0 until LUCENE-9276 (#1346, c0cf7bb4b0e, Simon Willnauer, 2020-03-13) sent addDocument through finishDocuments. In today's form the branch changes nothing unless an apply failed. We found no comment, test or PR that discusses a failed apply (in the history of these lines, the test tree and the prior-report searches); TestDocumentsWriterDeleteQueue, the only test that touches DeleteSlice, applies successfully.
• No documented limit was exceeded. Every term is 1 KB (the indexed-term limit is 32,766 bytes, IndexWriter.java:258), the IndexWriterConfig settings are the defaults, and ramBytesUsed() stays below 16 MB. The per-thread hard limit is "a safety limit to prevent a DocumentsWriterPerThread from address space exhaustion due to its internal 32 bit signed integer based memory addressing" (IndexWriterConfig.java:373-378); it is checked only after finishDocuments returns, so it cannot stop this copy.

Unsupported scale. On #15330 a maintainer (rmuir) wrote that "the limits on IndexWriter are 2GB". That remark is about one document's postings pool, which the per-thread hard limit bounds. Here the application never sees a number near 2 GB, and #16867 treats the same 2 GB overflow of the writer-wide buffer as a bug to fix (lucene/CHANGES.txt:324-326). Scale goes to severity (contrived), not to validity.

"After a non-tragic failure, call rollback()." On open #9738 (2019) a maintainer (s1monw) said that non-tragic exceptions from some methods may lose documents and that a user can always call rollback(). Their list (getReader, deleteAll, addIndexes, flushNextBuffer, doFlush, prepareCommitInternal, startCommit) leaves out addDocument, and they describe unrecoverable exceptions during indexing as handled in DocumentsWriter. No javadoc tells a caller to roll back after a failed addDocument; its javadoc promises a consistent index.

The class javadoc promises a self-close only "if you hit an Error, or disaster strikes during a checkpoint" (IndexWriter.java:162-165). Staying open is not the defect; commit() then breaking its own contract is.

Verdict: not intended, a defect. An independent reviewer from another model family (OpenAI GPT-5.6) rated the intended-behaviour argument weak; the devil's-advocate verifier rejected each case above.

Validated with gradle in a clean scratch worktree of 5adff78bb168 (OpenJDK 25.0.3, seed DEADBEEF, 2026-11-03): the repro fails stock with assertions on and off and passes with fix A, on and off. With fix A and the test in place, 7 closest suites (153 tests), the whole org.apache.lucene.index test package (214 classes, 1,397 tests, 0 failures, 57 skipped: nightly, monster or otherwise gated; the monster repro ran separately) and :lucene:core:check -x test pass. The scratch tree was clean after reverting. No stress or performance run was made.

Wrap the body of DocumentsWriterPerThread.finishDocuments in a try whose catch (Throwable t) calls onAbortingException(t) and rethrows. The finally of updateDocuments and updateBatch then calls maybeAbort, which aborts the segment and fires the tragic event (DocumentsWriterPerThread.java:594-605), and the writer closes itself. +17/-10 lines in one file, mostly the re-indented existing body. It mirrors the indexing chain (IndexingChain.java:2098-2100).

long seqNo;
try {
  if (deleteNode != null) {
    ... unchanged ...
  } else {
    ... unchanged, including deleteSlice.reset() ...
  }
} catch (Throwable t) {
  // The slice may now hold deletes that were never applied to pendingUpdates, and a later
  // finishDocuments would reset() past them: this DWPT can no longer honour them.
  onAbortingException(t);
  throw t;
}
return seqNo;

Trade-offs

The segment's buffered documents are lost with the abort (here buffered and next), and the application must open a new writer; the last commit stays the recovery point. That is the stance LUCENE-5987 took for every exception that would otherwise lose data silently. By code it also covers the writer-wide-buffer path (add(node, slice) -> tryApplyGlobalSlice, not executed). It makes any throw aborting, also an AlreadyClosedException from a closed delete queue, but a queue is closed only after every segment using it was flushed (DocumentsWriter.java:707), and the index test package passes. It also makes every other runtime failure out of finishDocuments aborting, where the writer used to stay open; the nearby suites and the index package pass, but no stress, concurrency, full-flush or performance test was run, so a reviewer may want the catch narrowed or argued case by case. Optional cleanup: the comment at the top of finishDocuments (DocumentsWriterPerThread.java:362-365) still describes the earlier order (update the slice, then increment the doc id); today the doc id is assigned first (:291).

What the checks validated

• Repro: fails stock (assertions on and off), passes with A (on and off), through gradle on 5adff78bb168.
• TestDocumentsWriterDeleteQueue, TestIndexWriterExceptions, TestIndexWriterExceptions2, TestIndexWriterDelete, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestIndexWriterWithThreads: 153 tests, 0 failures, 10 skipped.
• :lucene:core:test --tests 'org.apache.lucene.index.*': 214 classes, 1,397 tests, 0 failures, 57 skipped; :lucene:core:check -x test passes.
• Numbers probe with A: isOpen=false, tragic exception the ArithmeticException.
• Bounded exhaustive model of the slice protocol: 0 violations with this rule, 1,073 (transient fault) and 1,767 (persistent fault) without it.

Apply instead of reset() when the slice is not empty, or let apply advance sliceHead node by node so a retry resumes at the failing node.

Trade-offs

The per-thread BufferedUpdates is left mid-update: nextBuffer installs the new block and advances bufferUpto before Math.addExact throws (ByteBlockPool.java:196-199), so the pool is left half-advanced and appending to it again is not safe; the segment would still have to be flushed or aborted. Read, not validated.

Count the bytes an idle segment's slice pins, or flush such a segment early, so its copy never nears 2 GiB.

Trade-offs

Removes the executed per-thread trigger on a default config, and a maintainer may reasonably want it first. It leaves the handling, though: the writer-wide-buffer path and any future non-aborting throw from a queue node still reach the reset(), and the transient-fault check loses a delete with no overflow at all. Do it in addition to A, not instead. Not validated here.

Method: non-formal (code reading witnessed by an executed public-API probe), confirmed with a bounded exhaustive model.

• #16867 (merged 2026-10-30) caps the bytes of buffered deletes and updates in the writer-wide delete queue, because the byte pools use 32-bit offsets, and assumes the per-thread buffers cannot overflow. Reading the code around it for other copies of the same stream found the copy an in-memory segment makes in finishDocuments, which has no cap, and the non-aborting handling when it throws.
• A public-API probe (large distinct delete terms behind an idle in-memory segment, then a delete of the buffered document, 8 GB heap) showed addDocument failing there, the writer staying open and the next commit missing the delete. The case was then reduced to the one 3 GB doc-values-update JUnit test of this page.
• A bounded exhaustive model of one segment's delete slice (every sequence of up to 7 deletes, adds, updates and commits with one injected non-aborting fault: 19,420 fault scenarios per mode) found 1,073 (transient fault) and 1,767 (persistent fault) runs where a commit lacks an acknowledged operation, 0 with the fix and 0 without a fault.