An Error while IndexWriter aborts the queued segments of a failed flush hangs its self-close forever

When several threads index into one IndexWriter, each holds its own in-memory segment. A full flush (DirectoryReader.open(writer), commit(), flush()) queues them all. If flushing the first one fails, DocumentsWriterFlushControl.abortPendingFlushes aborts the rest, but it wraps each abort in catch (Exception _). An Error thrown while aborting one of them (in practice an OutOfMemoryError) ends the loop: the segments it has not reached stay registered in flushingWriters, and nothing ever removes them. IndexWriter then closes itself, as it documents for any Error, and that close waits in waitForFlush for those segments forever, so DirectoryReader.open(writer) never returns. Until LUCENE-8215 (2018, first release 7.4.0) the loop caught Throwable to "keep on aborting the flush queue". Every release since is affected. It is realistically OutOfMemoryError-only, so it is Low priority.

1. Three segments queued. DirectoryReader.open(writer) -> IndexWriter.getReader -> flushAllThreads() (IndexWriter.java:586). markForFullFlush checks out every segment with documents; checkOutForFlush adds each to flushingWriters (DocumentsWriterFlushControl.java:427, :437-441) and all go into flushQueue (:626).
2. The first flush fails. maybeFlush takes segment 1 from the queue (DocumentsWriter.java:494) and flushes it; IndexOutput.close() throws, and doFlush's finally releases segment 1 with doAfterFlush (DocumentsWriter.java:577). Segments 2 and 3 are still queued and registered. Any throwable in a segment's flush is an aborting exception, so the writer is already tragic here (DocumentsWriterPerThread.java:585-603); an IOException would do as well.
3. The abort loop meets an Error. getReader calls finishFullFlush(false) (IndexWriter.java:669-671) -> abortFullFlushes (DocumentsWriter.java:738) -> abortPendingFlushes (DocumentsWriterFlushControl.java:694-723). Aborting segment 2 closes its open stored-fields writer through IOUtils.closeWhileHandlingException (StoredFieldsConsumer.java:112-113), which rethrows Errors on purpose ("we ensure that we bubble up any errors", util/IOUtils.java:155-178).
4. Segment 3 is stranded. catch (Exception _) (:700, and :712 for blocked flushes) does not take the Error. The per-entry finally releases segment 2 only, the Error leaves the loop, and the outer finally clears the queue (:718-722) while segment 3 is still in flushingWriters. Only doAfterFlush (:298-301) removes it, and nothing will call it.
5. The self-close waits forever. getReader's catch (Throwable) marks the Error as tragic and runs maybeCloseOnTragicEvent (IndexWriter.java:711-722) -> rollbackInternalNoCommit -> docWriter.abort() (:2635) -> waitForFlush() (DocumentsWriter.java:226), an untimed wait() while flushingWriters is not empty (DocumentsWriterFlushControl.java:361-369).

A RuntimeException thrown at the same point is caught and the open returns promptly, with no hang (the controls of the cross-model reviewers; 1.3 s in one), so the catch (Exception) boundary is the defect.

• Who. Multi-threaded indexing (IndexWriter is documented as "completely thread safe", IndexWriter.java:167) with NRT reopen, commit or flush, on the default configuration; the injected Error is a fault, not a setting.
• Effect. A silent hang, not an exception: the thread that opened the reader blocks forever [exec]. By reading only: commit() (IndexWriter.java:3875, :3879) and flush() (:4411, :4427-4434) reach the same loop. Executed by the fact check's probe: any later rollback() or close() of that writer never returns either, because the hung self-close has already set closing (:6102, :1379), so they wait in shouldClose(true) -> doWait() (:1387, :5867).
• Data. The hang comes before the write.lock release (:2635 runs before :2684), so the last commit on disk is untouched, write.lock stays held in that JVM until the process exits, and buffered documents are lost, as in any tragic close [read].
• Scale. Three in-memory segments and one Error; no size threshold; it does not depend on assertions.
• How rare. Contrived: a full flush must fail, then an Error must hit the abort of a queued segment that is not the last one. Many servers exit the JVM on OutOfMemoryError. No report of this hang was found since 7.4.0.

TestErrorAbortingQueuedSegments (one method, extends LuceneTestCase, package org.apache.lucene.index of the lucene/core tests). The three threads only build the segments; the failing open runs on the test thread, and the outcome needs no timing. TestUtil.getDefaultCodec() pins the default codec against the test framework's random codec. No package-private member is used; FilterDirectory and FilterIndexOutput are public but marked @lucene.internal.

The same method is the regression test (it fits next to TestIndexWriterOnError). With a fix it passes in under 1.5 s. Older releases need more than package renames: FilterIndexOutput first shipped in 9.6.0 (GITHUB-11868, #11958), so before 9.6.0 wrap IndexOutput by hand, as the hunt's earlier, longer test for 7.x/8.x does; on 7.x/8.x the test framework package is org.apache.lucene.util, and on 7.x RAMDirectory replaces ByteBuffersDirectory.

The fault is harsher than a natural one-off OutOfMemoryError: once fail is set, every IndexOutput.close() throws. By reading, the hang needs only a failed flush of the first segment (any exception) and then one Error while a queued segment that is not the last is aborted. During the hang writer.isOpen() already returns false (IndexWriter.java:6117-6118), so it does not show that the close finished.

The timeout report also lists a leaked thread (the test thread still parked in waitForFlush) and a stdout-limit failure (the timeout's thread dump); both are consequences of the hang.

cp $REPRO/TestErrorAbortingQueuedSegments.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestErrorAbortingQueuedSegments -Ptests.seed=DEADBEEF

# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF (assertions on)
TestErrorAbortingQueuedSegments > testErrorWhileAbortingQueuedSegments FAILED
    java.lang.Exception: Test timeout exceeded (>= 20000 msec).
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:C9B68B5A67B45519]:0)
    ... (the leak report, after the timeout):
        com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestErrorAbortingQueuedSegments:
       1) Thread[id=21, name=TEST-TestErrorAbortingQueuedSegments.testErrorWhileAbortingQueuedSegments-seed#[DEADBEEF]-worker, state=WAITING, group=TGRP-TestErrorAbortingQueuedSegments]
            at org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(DocumentsWriterFlushControl.java:364)
            at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:226)
            at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2635)
            at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2590)
            at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:6103)
            at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:719)
            at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:144)
            at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:123)
            at org.apache.lucene.index.TestErrorAbortingQueuedSegments.lambda$testErrorWhileAbortingQueuedSegments$1(TestErrorAbortingQueuedSegments.java:88)
            at org.apache.lucene.tests.util.LuceneTestCaseParent._expectThrows(LuceneTestCaseParent.java:1587)
            ...
:lucene:core:test (FAILURE): 2 tests, 2  failures
# (the second failure is the suite-level report of the leaked thread and of the timeout's thread dump)

# same, -Ptests.asserts=false
TestErrorAbortingQueuedSegments > testErrorWhileAbortingQueuedSegments FAILED
    java.lang.Exception: Test timeout exceeded (>= 20000 msec).
    ... leaked thread parked at org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(DocumentsWriterFlushControl.java:364)
:lucene:core:test (FAILURE): 2 tests, 2  failures

# with fix A: assertions on, then off
:lucene:core:test (SUCCESS): 1 test
:lucene:core:test (SUCCESS): 1 test

• Contract. "if you hit an Error, or disaster strikes during a checkpoint then IndexWriter will close itself. ... Any subsequent calls will throw an AlreadyClosedException." (IndexWriter.java:162-165). getReader implements this for every Error (:711-722).
• Purpose of the loop. LUCENE-4277 (#5345, b26dd733f77, 2012; CHANGES: "Fix IndexWriter deadlock during rollback if flushable DWPT instance are already checked out and queued up but not yet flushed") created abortPendingFlushes with catch (Throwable ex) { // ignore - keep on aborting the flush queue }. LUCENE-4989 (#6053) and LUCENE-5002 (c0e7a50ab8e, 2013) moved doAfterFlush into a per-entry finally so a failing abort could not skip the release.
• The narrowing. LUCENE-8215 (#9262, 2e35ef2b3da, 2018, "Fix several fragile exception handling places in o.a.l.index") changed both catches to catch (Exception ex). Its discussion is about the new IOUtils helpers and never mentions this loop. The same commit added IOUtils.applyToAll ("even if an exception is thrown. The first exception ... is re-thrown"), the shape of the recommended fix.
• Precedent. Upstream fixed Error-induced writer hangs as bugs: #12654 (fixed by #12751, 2023, an Error during the tragic close left rollback() waiting forever) and LUCENE-7570, whose test TestTragicIndexWriterDeadlock throws an OutOfMemoryError from mergeSuccess and expects commit() to fail fast and close() to return.
• Tests. TestIndexWriterOnError injects fake Errors from I/O but runs "just one thread" (:59 of the test), so it never queues three segments in one full flush; no upstream test pins the hang.

The strongest case for "not a bug": LUCENE-8215 narrowed the catch on purpose so that Errors are not swallowed, and after an OutOfMemoryError the JVM state is undefined, so Lucene owes nothing. Also, it needs two faults in one flush.

Why it fails: the narrowing meant that Errors should propagate, and the fix keeps that (it rethrows the first Error after every segment is released); nothing in #9262 says cleanup may stop halfway. IndexWriter itself documents a self-close after any Error and treats every Error as tragic; a hang breaks that promise and leaves the writer unclosable. The repro uses no InfoStream and no writer-internal hook: the Error comes from IndexOutput.close(), the kind of directory-I/O Error that Lucene's own TestIndexWriterOnError injects. Needing two faults lowers the likelihood (hence Low), not the validity.

Option A was built and checked through gradle on 5adff78bb168; option B is the pre-7.4.0 code, which the hunt ran on 7.4.0. IOUtils.applyToAll is not a drop-in: it rethrows the first throwable of any type (util/IOUtils.java:566-577), so it would turn today's ignored abort exceptions into failures.

Every queued and blocked segment is aborted and released (unless doAfterFlush itself throws); the Error still propagates (after the cleanup), so LUCENE-8215's intent holds and the writer treats it as tragic. Exceptions stay ignored as today. The hunt's variant with explicit addSuppressed is the same idea and passed the repro too.

+    Error firstError = null;
     ... for each queued DWPT, then each blocked DWPT:
         } catch (Exception _) {
           // that's fine we just abort everything here this is best effort
+        } catch (Error e) {
+          // keep aborting so every DWPT reaches doAfterFlush, rethrow below
+          firstError = IOUtils.useOrSuppress(firstError, e);
         } finally {
           doAfterFlush(dwpt);
     ... the outer finally clears flushQueue and blockedFlushes ...
+    if (firstError != null) {
+      throw firstError;
+    }

Trade-offs

If the full flush failed with an exception and the abort then hits an Error, finishFullFlush(false) now throws that Error in place of the flush's exception, as any throw from it does today.

What the checks validated

• Gradle in a scratch worktree at 5adff78bb168 (OpenJDK 25.0.3, seed DEADBEEF): without the fix the repro times out at 20 s with assertions on and off; with fix A it passes with assertions on and off (0.38 s and 0.36 s test time).
• With fix A: the repro, TestIndexWriterOnError (5, 1 @Nightly skipped), TestTragicIndexWriterDeadlock (3), TestIndexWriterExceptions (40, 3 @Nightly skipped), TestIndexWriterExceptions2 (1), TestIndexWriterWithThreads (14), TestIndexWriterOnDiskFull (4), TestNRTReaderWithThreads (1), TestFlushByRamOrCountsPolicy (5) and TestDocumentsWriterStallControl (3): 77 tests, 0 failures, 4 skipped. :lucene:core:checkGoogleJavaFormat passes with the fix and the repro file in place.
• The hunt's variant with explicit addSuppressed passed the same repro on the main jars too (assertions on 1.32 s, off 1.27 s, wall time).

Two lines. The hunt's 7.4.0 run with only the LUCENE-8215 hunk reverted passes (0.62 s wall time, writer closed; run with the earlier, longer version of the test).

Trade-offs

It swallows the abort's Error. The writer still closes itself, because the failed flush already made it tragic (DocumentsWriterPerThread.java:585-603), but if the flush failed with an IOException and the abort then hit an OutOfMemoryError, the OOM would be neither thrown nor recorded (the tragedy and the thrown exception stay the IOException; by reading), against LUCENE-8215's aim of not swallowing Errors.

What the checks validated

Executed on 7.4.0 only (the hunt's history step), not on main.

Method: non-formal (code reading, witnessed by an executable repro).

• The finder asked which throwables can leave abortPendingFlushes and what each leaves behind in flushingWriters, the set waitForFlush waits on. catch (Exception _) lets an Error out, the outer finally clears the queue, and only doAfterFlush removes entries, so every segment after the one that threw stays registered.
• DWPT.abort can throw an Error because IOUtils.closeWhileHandlingException rethrows Errors by design, and the tragic self-close after an Error runs waitForFlush.
• The finder's first test threw the Error from an InfoStream; an independent reproducer then wrote the Directory-based test this repro is reduced from, with no InfoStream.