A failure inside prepareFlush is not tragic: IndexWriter stays open, and the next commit silently lacks acknowledged documents, deletes and updates

Flushing an in-memory segment (a DWPT) starts in DocumentsWriter.doFlush, which creates its flush ticket with ticketQueue.addTicket(() -> new FlushTicket(dwpt.prepareFlush(), true)) (DocumentsWriter.java:534-536). prepareFlush first freezes the writer-wide buffered deletes and doc-values updates into a packet and clears them (DocumentsWriterPerThread.java:429, DocumentsWriterDeleteQueue.java:269-273), then applies the segment's own slice of the delete stream (:434), which can throw. When it does, there is no ticket: the catch skips markTicketFailed (DocumentsWriter.java:556-563), so the packet is lost, and the finally drops the DWPT with its buffered documents (:576-578). doFlush issues no abort and no tragic event for this DWPT, so isOpen() stays true and getTragicException() null, and the next successful commit(), or close() with the default commitOnClose, persists an index without operations its sequence number covers. Every failure inside the segment write itself is tragic (DocumentsWriterPerThread.java:585-604). The executed trigger, about 2^31 bytes of update terms behind one idle in-memory segment, reaches this step since 9.8.0 (#12392 made the overflow throw there; 9.7.0 closed the writer instead); an InfoStream that throws on the flush's first message reaches the same handling in every release since 4.0.0. Making a failure on this path tragic in doFlush fixes both.

// DocumentsWriter.doFlush, main 5adff78bb168                         DocumentsWriter.java:502
DocumentsWriterFlushQueue.FlushTicket ticket = null;                 // :506
try {
  try {
    ticket = ticketQueue.addTicket(                                   // :534-536
        () -> new DocumentsWriterFlushQueue.FlushTicket(dwpt.prepareFlush(), true));
    ... flushingDWPT.flush(flushNotifications) ...                    // :541, the tragic-safe part
  } catch (Throwable t) {                                             // :556
    if (ticket != null) {                                             // :557  null: prepareFlush threw
      ticketQueue.markTicketFailed(ticket);                           // :561  skipped: the packet is lost
    }
    throw t;                                                          // :563  no abort, no tragic event
  }
} finally {
  flushControl.doAfterFlush(flushingDWPT);                            // :577  the DWPT and its documents go
}

// DocumentsWriterPerThread.prepareFlush                                DocumentsWriterPerThread.java:427
final FrozenBufferedUpdates globalUpdates = deleteQueue.freezeGlobalBuffer(deleteSlice); // :429 packs + clears
if (deleteSlice != null) {
  deleteSlice.apply(pendingUpdates, numDocsInRAM);                    // :434  throws here
  ...
}

1. The thread that added buffered then only updates and deletes, so its in-memory segment's slice spans all 2,200,001 later operations. commit() reaches doFlush through prepareCommitInternal (IndexWriter.java:3809) and flushAllThreads / maybeFlush (DocumentsWriter.java:688, :496); flush(), forceMerge() (IndexWriter.java:2248, :4402) and an NRT reopen (:586) take the same path.
2. freezeGlobalBuffer applies the global slice, builds a FrozenBufferedUpdates packet and clears the writer-wide buffer (DocumentsWriterDeleteQueue.java:217-237, :262-277, clear at :272). Here the packet holds the delete of victim and the updates since the writer-wide buffer was last applied (it is applied whenever it reaches the 16 MB RAM buffer, FlushByRamOrCountsPolicy.java:52-53).
3. Applying the segment's slice appends each update's 1 KB term to the segment's private buffer: 2.2 million updates need about 2.25 GB, and Math.addExact in ByteBlockPool.nextBuffer throws ArithmeticException at 2 GiB (ByteBlockPool.java:199).
4. addTicket gives its ticket count back and rethrows (DocumentsWriterFlushQueue.java:48-50). ticket is still null, so nothing holds the packet, and the finally drops the DWPT with buffered. doFlush calls neither flushFailed, abort() nor onTragicEvent for this DWPT (a failed full flush's cleanup, finishFullFlush(false) -> abortFullFlushes(), aborts only other DWPTs still queued or blocked for that flush, DocumentsWriter.java:727-745, DocumentsWriterFlushControl.java:685-723; the one-thread repro has none); getPendingNumDocs() stays one too high (2 for an index of 1 document).
5. Contrast: DocumentsWriterPerThread.flush treats any Throwable as aborting (:585-587), and maybeAbort aborts and fires the tragic event ("whatever we do here we have to fire this tragic event up", :594-604). Two steps that can throw an ordinary exception run outside that guard: prepareFlush, and flush's first InfoStream message ("flush postings as segment", :479-483, before the try at :485). A non-tragic exception there is dropped the same way; the ticket already exists then, so only the segment's documents are lost.

Sequence numbers: victim 1, first commit 3, buffered 4, last update 2,200,004, delete 2,200,005; ramBytesUsed() 2,586,576 bytes before the commit. The commit throws; isOpen() true, getTragicException() null, getPendingNumDocs() 2. The retried commit returns 2,200,008 and holds 1 document: buffered 0 hits, victim 1 hit with n = 2,197,624. With the fix the commit throws the same exception and the writer closes itself (getTragicException() is that exception, getPendingNumDocs() 1).

• Who is hit. Applications that keep using an IndexWriter, or close it with the default commitOnClose (the try-with-resources idiom), after commit(), flush(), forceMerge(), an NRT reopen or flushNextBuffer() threw from this step. The writer gives no signal to stop: isOpen() is true and getTragicException() is null, the two signals LUCENE-6217 (#7279) added for that decision (#9738's description says Solr's leader election, for one, decides from getTragicException()). Earlier commits stay intact, and rollback() right after the failure restores the last commit (assertions off).
• What is lost. The in-memory segment's documents and the deletes and doc-values updates in the frozen packet: in the repro the buffered document, the delete of a committed document (it stays searchable) and the last 2,376 of 2,200,000 updates (a stale value). The index stays structurally valid; no hang.
• Executed entry points. commit() (the repro); forceMerge(1), after which the next commit lacks the document and the delete; and close() after a failed commit(), which returns normally and commits the same incomplete state. By code reading, flush(), NRT reopen, IndexWriter.flushNextBuffer() (IndexWriter.java:3758-3771, which treats only an Error as tragic, -> DocumentsWriter.flushOneDWPT, DocumentsWriter.java:246-258) and an indexing call whose document makes its DWPT due for flushing (DocumentsWriter.postUpdate, :400-410) reach the same doFlush path.
• Default vs opt-in. Default IndexWriterConfig, codec and directory, one thread, assertions on or off.
• Triggers of the same handling. (1) Executed: about 2^31 bytes of doc-values-update (or delete) terms behind one idle in-memory segment, for example 2.2 million updates of a 1 KB id; ramBytesUsed() never comes near the 2 GiB the flush will need (it peaks at about 16.8 MB during the updates and reads 2.6 MB just before the commit), so nothing warns (its root, the unbounded per-segment copy, is LFV-1077). Released 9.8.0 to 10.6.0 and main. (2) An over-long delete term makes prepareFlush's freeze throw (LFV-1046, 9.9.0 and later). (3) An InfoStream whose message() throws on "flush postings as segment": the retried commit has 0 documents (executed on main, 10.6.0, 10.5.2 and 9.12.3, and, in a version-neutral main() form with assertions off, on eight releases from 4.0.0 to 9.0.0: 4.0.0, 4.10.4, 5.0.0, 5.5.5, 6.6.6, 7.7.3, 8.11.4 and 9.0.0). (4) In production, elastic/elasticsearch#160855 (Lucene 10.5.1) logs a non-tragic IllegalStateException out of freezeGlobalBuffer <- prepareFlush <- doFlush <- IndexWriter.flushNextBuffer(), during an indexing-buffer flush inside a bulk request (Elasticsearch logs it with the message "Failed to perform engine refresh"), as a warning.
• Likelihood: rare. It needs a non-tragic exception out of this step and then an application that continues or just closes. The 2 GiB trigger alone would be contrived; the other triggers and the production log make it rare.
• Side observation, not part of this finding. With assertions on, a later rollback() or close() fails the internal pendingNumDocs ... != ... totalMaxDoc assertion (#10661 is the generic report).

new IndexWriterConfig() (the test framework picks a random default codec; the failing step, prepareFlush, runs before any codec code, and the test failed with every seed tried), a ByteBuffersDirectory, one thread, no package-private access (it sits in org.apache.lucene.index like the upstream tests). Run it with -Ptests.monster=true -Ptests.heapsize=3g. It passes :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs).

The test expects commit() to throw ArithmeticException and then accepts either a closed writer (the fix) or an open writer whose next commit has buffered once and victim gone. On the released 9.7.0 it stops at expectThrows, because there the commit throws ArrayIndexOutOfBoundsException (and the writer closes itself).

Regression test: this same test. It passes with the recommended fix.

TestFailedPrepareFlushLosesOps > testNextCommitKeepsAcknowledgedOps FAILED
    org.junit.ComparisonFailure: expected:<buffered=[1 victim=0]> but was:<buffered=[0 victim=1]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:DEE1DBA70DD1CA8D]:0)
        at app//org.junit.Assert.assertEquals(Assert.java:117)
        at app//org.junit.Assert.assertEquals(Assert.java:146)
        at app//org.apache.lucene.index.TestFailedPrepareFlushLosesOps.testNextCommitKeepsAcknowledgedOps(TestFailedPrepareFlushLosesOps.java:57)

# the same steps as a plain main() (numbers only, assertions off): where the commit throws
commit 2 threw java.lang.ArithmeticException: integer overflow
    at org.apache.lucene.util.ByteBlockPool.nextBuffer(ByteBlockPool.java:199)
    ...
    at org.apache.lucene.index.DocumentsWriterDeleteQueue$DeleteSlice.apply(DocumentsWriterDeleteQueue.java:364)
    at org.apache.lucene.index.DocumentsWriterPerThread.prepareFlush(DocumentsWriterPerThread.java:434)
    at org.apache.lucene.index.DocumentsWriter.lambda$doFlush$0(DocumentsWriter.java:536)
    at org.apache.lucene.index.DocumentsWriterFlushQueue.addTicket(DocumentsWriterFlushQueue.java:42)
    at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:535)
after: isOpen=true tragic=null pendingNumDocs=2
commit 3 returned 2200008 numDocs=1 buffered=0 victim=1
commit 3 doc 0 n=2197624

• commit(). "it returns the sequence number such that all indexing operations prior to this sequence will be included in the commit point" and "All sequence numbers <= this value will be reflected in the commit" (IndexWriter.java:4202-4204, :4207-4209).
• forceMerge. "If an Exception is hit, for example due to disk full, the index will not be corrupted and no documents will be lost." (IndexWriter.java:2213-2214). It starts with flush(true, true) (:2248); executed, it throws the same ArithmeticException, the writer stays open and the next commit lacks the operations. (That javadoc paragraph is about merging, :2213-2217; here the exception comes from the flush forceMerge starts with, so this is the weaker of the two contracts.)
• The design rule. LUCENE-5987 (#7049): "IndexWriter will now forcefully close itself on aborting exception (an exception that would otherwise cause silent data loss)." (lucene/CHANGES.txt:11614-11616). The segment write follows it for every Throwable (DocumentsWriterPerThread.java:585-604); IndexWriter.flushFailed carries "TODO: this really should be a tragic" (IndexWriter.java:6242); getTragicException() names "disk full while flushing a new segment" as its example (:6108-6109).
• A live upstream test retries after non-tragic commit failures and asserts nothing is lost: TestIndexWriterExceptions.testExceptionDuringSync (LUCENE-1044, TestIndexWriterExceptions.java:943-970, 23 documents). (TestIndexWriterDelete.testErrorAfterApplyDeletes states the same expectation but has been @Ignored since 2017, TestIndexWriterDelete.java:802.)
• History. Before #12392 (9.8.0) the same workload overflowed later, in FrozenBufferedUpdates applied through IndexWriter.forceApply, which closes the writer: on 9.7.0 nothing is lost. #12392 is correct; it moved the throw into the one flush step that is not tragic.

The argument: a maintainer wrote on open #9738 (2019) that exceptions from doFlush(), prepareCommitInternal() and others "might cause documents go missing etc. but we treated them not as fatal or tragic events since a user could always call rollback()", and that "if somebody continues after a failed commit / prepareCommit / reopen they will end up with inconsistency and / or dataloss". TwoPhaseCommit says "If the 2-phase commit fails, rollback() is called" (TwoPhaseCommit.java:31), and the class javadoc promises a self-close only "if you hit an Error, or disaster strikes during a checkpoint" (IndexWriter.java:162-165).

Why it fails: that stance is in no javadoc, MIGRATE or CHANGES entry, and the same maintainer called the current behaviour inconsistent ("I am all for making it even more strict especially since it's inconsistent with what we do if addDocument fails with an aborting exception"). forceMerge's javadoc promises the opposite. close() with the default commitOnClose persists the loss with no retry at all, and under the default KeepOnlyLastCommitDeletionPolicy that commit replaces the last good one, so rollback() can no longer help. isOpen() and getTragicException() tell the caller the writer is fine. An executed review by another model family (OpenAI GPT-5.6, intended-behaviour lens) rated the argument weak, and a devil's-advocate verifier rejected it on the same grounds.

Validated with gradle in a clean scratch worktree of 5adff78bb168 (OpenJDK 25.0.3, seed DEADBEEF, 2026-11-03): the repro fails stock with assertions on and off and passes with fix A, on and off. A check test for the InfoStream step (an InfoStream that throws on "flush postings as segment"; not part of the repro) fails stock and passes with fix A, on and off. With fix A and both tests in place, :lucene:core:test --tests 'org.apache.lucene.index.*' passes (216 test classes, 1,399 tests, 0 failures, 57 skipped: nightly, monster or otherwise gated; the monster repro ran separately) and so does :lucene:core:check -x test. The scratch tree was clean after reverting.

Why one catch covers both unguarded steps: a failure inside DocumentsWriterPerThread.flush's guard has already aborted the DWPT (isAborted() is true), so the new branch runs only for prepareFlush and flush's first InfoStream message; the rest of the block only enqueues events (IndexWriter.java:437-444). The tragic event only records the tragedy; the entry point's maybeCloseOnTragicEvent() in its finally then closes the writer (for commit(), IndexWriter.java:3883 in prepareCommitInternal). This is reasoning from the source plus the runs above; no stress or performance run was made.

In DocumentsWriter.doFlush's catch (Throwable t), after the markTicketFailed block: if flushingDWPT.isAborted() == false, subtract the DWPT's documents from the in-RAM count when no ticket was made, call flushingDWPT.abort() (which also gives back the writer's pending document count, DocumentsWriterPerThread.java:98-100) and, in a finally, flushNotifications.onTragicEvent(t, "flush"); then rethrow. 15 lines added, DocumentsWriter.java only. It mirrors DocumentsWriterPerThread.maybeAbort (:594-604).

} catch (Throwable t) {
  if (ticket != null) {
    ticketQueue.markTicketFailed(ticket);
  }
  if (flushingDWPT.isAborted() == false) {
    // prepareFlush, or DWPT#flush's first log line, failed outside DWPT#flush's aborting guard
    try {
      if (ticket == null) { // else the finally above already subtracted them
        subtractFlushedNumDocs(flushingDWPT.getNumDocsInRAM());
      }
      flushingDWPT.abort();
    } finally {
      flushNotifications.onTragicEvent(t, "flush");
    }
  }
  throw t;
}

Trade-offs

An application that today retries commit() after such a failure now gets AlreadyClosedException and must open a new writer: the stance LUCENE-5987 took for every other flush failure. The frozen packet is still dropped, but the writer closes, so nothing incomplete is committed and the last commit stays the recovery point. Not stress- or performance-tested.

What the checks validated

• The repro passes with assertions on and off (gradle); the same steps as a plain main() show the writer closed with the ArithmeticException as its tragic exception.
• The InfoStream check test passes with assertions on and off.
• org.apache.lucene.index test package with both tests: 216 classes, 1,399 tests, 0 failures, 57 skipped.
• :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs) passes.

Wrap only ticket = ticketQueue.addTicket(... prepareFlush() ...) in a catch that gives back the count, aborts the DWPT, fires onTragicEvent(t, "prepareFlush") and rethrows.

Trade-offs

Narrower, but it leaves the InfoStream step losing the segment's documents (executed: that check test still fails with it).

What the checks validated

On main 645d4fe14a20, whose DocumentsWriter.java is identical to 5adff78bb168's: the repro passes on and off; 6 upstream classes pass (104 tests); a model-based fault injection shows 0 violations over 1,176 injected faults.

Swap the two steps of prepareFlush.

Trade-offs

Keeps the deletes and updates in the writer-wide buffer, but doFlush still drops the segment's documents silently; it still needs a tragic event or a documented contract.

Create the ticket before the failing step so markTicketFailed publishes the packet, and abort the DWPT.

Trade-offs

Keeps the deletes and updates of committed documents, but still loses the buffered documents, so it still needs a tragic event or a documented contract.

Method: non-formal (code reading witnessed by executed public-API probes), on main 645d4fe14a20, 2026-11-02; confirmed with a model-based test.

• #16867 (merged 2026-10-30) caps the bytes of buffered deletes and updates in the writer-wide delete queue at 1,945 MB, because the byte pools use 32-bit offsets. Reading the code around it for other copies of the same stream found the per-segment slice applied in prepareFlush, which has no cap, and the non-tragic handling when that step throws.
• Public-API probes (default configuration) then showed commit() throwing ArithmeticException there, the writer staying open, and the next commit missing the buffered document, a delete of a committed document and updates. The case was reduced to the one JUnit test of this page.
• A model-based test enumerated 1,554 operation sequences of up to 4 steps (adds, deletes of committed and buffered documents, doc-values updates) with a fault injected after 0 to 4 nodes of the segment's slice inside prepareFlush (7,770 scenarios, a test-only copy of the class): every one of the 1,176 faults that fired broke the oracle (a tragic close with the last commit intact, or a next commit equal to the model), and none did with the fix.
• The second unguarded step (DocumentsWriterPerThread.flush's first InfoStream message) was found while testing InfoStream callbacks that throw.