A throw from IndexWriter's InfoStream as a commit's full flush ends leaves the writer in full-flush mode, so it stops flushing

prepareCommit(), commit() (unless an earlier prepareCommit() is still pending), flush() and an NRT reader open each run a full flush. DocumentsWriter.flushAllThreads() starts it by calling DocumentsWriterFlushControl.markForFullFlush(), which sets fullFlush = true, and it must end in DocumentsWriter.finishFullFlush(boolean). That method first logs "<thread> finishFullFlush success=" + success to the InfoStream; only then does it call flushControl.finishFullFlush() or abortFullFlushes(), the only code that sets fullFlush back to false (the first passes the flushes parked during the full flush on to be flushed, the second aborts them). Its finally only resets a flag and applies deletes.

If the log call throws a RuntimeException, commit() (or the flush or NRT open) rethrows it, and because IndexWriter treats only an Error as tragic, the writer stays open with getTragicException() == null. But fullFlush stays true. From then on every in-memory segment buffer that becomes due for flushing is checked out and blocked instead of flushed, so maxBufferedDocs and the RAM buffer no longer flush anything. With RAM-based flushing the parked bytes count as flushing, the stall control engages at twice the RAM buffer, and an indexing thread waits in addDocument until another thread's full flush releases the parked buffers, or a rollback() or close() closes the writer.

With assertions off, the next full flush from any thread (commit, refresh, flush(), close()) clears the state; in the executed single-thread, add-only cases no document was lost. With assertions on, that next full flush trips the internal assertion "called DWFC#markForFullFlush() while full flush is still running" and the writer closes. By code reading, an Error from the log call is tragic and closes the writer. The trigger is opt-in and contrived, so the bug is Low priority; calling rollback() after the failed commit avoids it, and the fix is to end the full flush before logging.

1. The protocol. The comment in IndexWriter.getReader (IndexWriter.java:572-581) says the full flush "informs the FlushControl (#markForFullFlush()) that it should prevent any new DWPTs from flushing until we are ... done (DocumentsWriter#finishFullFlush(boolean))". The entry points end it in a finally or right after: prepareCommitInternal (IndexWriter.java:3872-3877), getReader (:671, :680) and flush (:4410-4412).
2. Log before release. DocumentsWriter.finishFullFlush (DocumentsWriter.java:727-745): the infoStream.message("DW", ... + " finishFullFlush success=" + success) call, inside an isEnabled("DW") guard (:729-732), is the first thing the try does; flushControl.finishFullFlush() / abortFullFlushes() follow (:733-739); the finally (:740-744) only sets pendingChangesInCurrentFullFlush = false and calls applyAllDeletes().
3. The only way out of full-flush mode. DocumentsWriterFlushControl.finishFullFlush() (:661-676) and abortFullFlushes() (:685-692) are the only code that sets fullFlush = false (:672, :689). Parked buffers return to the flush queue only through pruneBlockedQueue (:647-659), which runs in finishFullFlush() (:666-670) and in the next markForFullFlush (:624); abortFullFlushes() and a rollback() abort them instead (abortPendingFlushes, :694-724).
4. Not tragic. The throw comes from prepareCommitInternal's finally (IndexWriter.java:3872-3877); only the outer catch (Error tragedy) (:3879-3881) records a tragedy. So after a RuntimeException the writer is open and getTragicException() is null.
5. Every later flush is parked. DocumentsWriterFlushControl.checkout (:261-268): while fullFlush is set, a flush-pending buffer goes to checkoutAndBlock (:408-418, into blockedFlushes), and nextPendingFlush (:452-480) hands out only buffers from the (empty) flush queue during a full flush.
6. The stall. setFlushPending (:376-387) moves a buffer's bytes from activeBytes to flushBytes, where parked buffers keep them. updateStallState stalls when activeBytes + flushBytes exceeds twice the RAM buffer while activeBytes stays below it (:114-119, :326). DocumentsWriter.preUpdate (DocumentsWriter.java:386-397) then loops maybeFlush(); waitIfStalled(); (DocumentsWriterStallControl.java:61-78, up to 1 s per round), and nothing can flush; only another thread's full flush, rollback() or close() ends it (a closed writer never stalls, :326). With only maxBufferedDocs (RAM flushing disabled) the stall limit is Long.MAX_VALUE (:116-118): an add-only workload does not stall, it buffers without bound. Deletes are never applied while fullFlush is set (DocumentsWriter.java:174-176), and on main (since #16867, 2026-10-30) buffered delete bytes at maxDeleteBytes (1945 MB) also stall indexing during a full flush (:327-329), so there a delete-heavy workload can stall too.
7. Healing. The next full flush runs markForFullFlush again. With assertions off it moves the parked buffers to the flush queue (:624), flushes them and ends normally (a second commit kept all 11 documents). With assertions on, assert fullFlush == false : "called DWFC#markForFullFlush() while full flush is still running" (:573-574) fails, and as an Error it closes the writer.

That it stays stalled indefinitely is from code reading: nothing on the stalled thread's own path ends it; another thread's full flush releases the parked buffers, and a rollback() or close() ends the stall by closing the writer.

• Who. Applications that set an InfoStream with IndexWriterConfig.setInfoStream (a new config takes InfoStream.getDefault(), LiveIndexWriterConfig.java:142, which is NO_OUTPUT unless replaced process-wide). The sink must throw an unchecked exception on exactly this line: a custom InfoStream, a PrintStream passed to setInfoStream(PrintStream) whose underlying stream throws an unchecked exception (a PrintStream only turns IOException into its error flag), or a logging bridge set to pass appender failures to the caller. The stock PrintStreamInfoStream and JavaLoggingInfoStream do not throw by themselves.
• When. During commit(), prepareCommit(), flush() or an NRT open or reopen (all end their full flush here), and the application keeps using the writer after that exception instead of calling rollback().
• What breaks. The writer stops flushing without any sign: memory grows past maxBufferedDocs or the RAM buffer until some thread runs another full flush. With RAM-based flushing (the default) a lone indexing thread with no other thread to commit, refresh, roll back or close the writer stays blocked in addDocument after about twice the RAM buffer (executed with a 1 MB buffer; about 32 MB with the default 16 MB buffer, by reading). An application that refreshes or commits on another thread is released at the next refresh (no document was lost in the executed add-only cases, assertions off).
• Scale. None needed: one failed commit and a few documents.
• How likely. Contrived: an opt-in log, a sink that throws on this one line, and continued use of the writer instead of rollback(); the recorded searches found no outside report since 4.0 (2012). Before 8.8.0 indexing crawls (about one document per second) instead of stopping.
• Remedy. Calling rollback() after the failed commit (the remedy a maintainer gives in #9738) avoids it: it discards the buffered documents and closes the writer. Continuing to use the open writer is the case this bug is about.

The test sits in org.apache.lucene.index only for placement next to TestIndexWriterExceptions; it uses only public classes and methods (IndexWriterConfig, InfoStream, ByteBuffersDirectory, numRamDocs()), though InfoStream is marked @lucene.internal (a debugging API), the default RAM buffer and merge policy (the test framework randomizes the default codec, which by reading plays no part), and empty documents. It is deterministic: the throw is keyed on the message text, and the seed does not matter (this one-method test failed with seeds DEADBEEF and 5EEDF1F0; the independent reproduction's earlier, longer test, a different file of the same name, failed with 5EEDF1F0, 5EEDF1F1, A0D14941 and A0D14942).

Regression test: this same test; it passes with either fix and fits in TestIndexWriterExceptions.

A variant without the anonymous InfoStream (an internal API): a stock java.io.PrintStream over an OutputStream whose write throws an IllegalStateException once the text contains "finishFullFlush success=", passed to setInfoStream(PrintStream), fails the same way with assertions on and off (the intended-lens reviewer's test).

cp $REPRO/TestCommitLogFailureStopsFlushing.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestCommitLogFailureStopsFlushing -Ptests.seed=DEADBEEF
# the same result with -Ptests.asserts=false

# gradle, stock e1faeba08bde, -Ptests.seed=DEADBEEF (assertions on; the same with -Ptests.asserts=false)
TestCommitLogFailureStopsFlushing > testFlushesAfterCommitFailedOnLogMessage FAILED
    java.lang.AssertionError: numRamDocs=10
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:CBAFFB174CCC737E]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.index.TestCommitLogFailureStopsFlushing.testFlushesAfterCommitFailedOnLogMessage(TestCommitLogFailureStopsFlushing.java:59)
1 test completed, 1 failed

# with either fix: 1 test, 0 failures (assertions on and off)

• The flush contract. setMaxBufferedDocs (LiveIndexWriterConfig.java:213): "When this is set, the writer will flush every maxBufferedDocs added documents." The class javadoc (IndexWriter.java:162-165) names when the writer closes itself: "if you hit an Error, or disaster strikes during a checkpoint then IndexWriter will close itself". Here it does not close (getTragicException() is null), so its documented flush behaviour still applies.
• The code's own invariant. The full flush prevents new DWPTs from flushing "until we are ... done (DocumentsWriter#finishFullFlush(boolean))" (IndexWriter.java:572-581). Here "done" does not come until the next full flush.
• Lucene's tests. TestIndexWriterExceptions.testExceptionDuringRollback (:1980-2052) throws RuntimeException("BOOM!") from InfoStream.message at the ordinary message "rollback: done finish merges" (IndexWriter.java:2626) or at the test point "rollback before checkpoint", and asserts "even though we hit exception: we are closed, no locks or files held, index in good state" (:2042). testOutOfMemoryErrorRollback (:1150-1166) throws an Error at "startFullFlush". None throws at "finishFullFlush". These tests check cleanup and close after a logging failure, not continued use of an open writer after a failed commit: they show that a throwing InfoStream is a case Lucene already exercises, not a documented contract.
• The public API. IndexWriterConfig.setInfoStream(PrintStream) (IndexWriterConfig.java:440-446) is public and states no no-throw condition; MIGRATE.md:1147-1148 presents JavaLoggingInfoStream for logging IndexWriter events through java.util.logging, with a Log4j adapter.
• History. LUCENE-3348 (78d186e081f, 2011) added the log line as the method's first action, when the info stream was a plain PrintStream; the pluggable InfoStream (LUCENE-2621, 7877d1ff072, 2011-11, which also converted this line to infoStream.message("DW", ...)) and its rework in LUCENE-3598 (2011-12) kept the order. The recorded searches found no commit or issue that discusses a throw from it.
• Precedent. #6053 (LUCENE-4989, 2013): an exception in abortPendingFlushes skipped cleanup and threads hung in waitIfStalled; fixed with a finally (via #6066). #10547 (LUCENE-9508, 2020): "we can't let blocked flushes go unless the full flush is over" (s1monw).

The intended-lens reviewer (OpenAI GPT-5.6) and the intended-behaviour check built the case for "not a bug": InfoStream, PrintStreamInfoStream and JavaLoggingInfoStream are @lucene.internal, and InfoStream is a "Debugging API" (InfoStream.java:23-29). A sink that throws is abnormal application behaviour; Lucene does not wrap every InfoStream call and promises no strong exception safety after an arbitrary callback RuntimeException. A maintainer may add #9738 (LUCENE-8692): non-tragic exceptions from prepareCommitInternal "might cause documents go missing", and rollback() is the remedy. The cross-model reviewers rated this argument moderate.

Why it fails: the trigger is reachable through the public setInfoStream(PrintStream) overload with a stock PrintStream over a failing OutputStream (executed); Lucene's own tests already inject RuntimeExceptions through InfoStream.message (during rollback); the writer reports itself open and healthy while silently breaking a documented setting; and the fix is a reorder with no measured cost (it moves the message after the release; it does not make the other InfoStream call sites safe). #9738 is about a failed commit losing documents, which the caller sees; here the writer keeps accepting documents and quietly stops flushing them. Still, rollback() after the failed commit is the practical remedy, and a maintainer could reasonably call the fix low-priority hardening.

Both options were validated in a scratch worktree of e1faeba08bde with gradle (OpenJDK 25.0.3, seed DEADBEEF, 2026-11-02): the minimal test fails on stock and passes with each fix, with assertions on and off. Neither fixes the sibling InfoStream sites (LFV-842, LFV-844, LFV-846), which need their own changes. Open PR #14837 (open, last updated 2025-10-16) rewords this log line and keeps it before the release; if it merges first, either fix needs a trivial rebase.

One file, DocumentsWriter.java, +6/-4 (fix-reorder.diff): the infoStream.isEnabled("DW") / message(...) block moves from the start of the try to just after flushControl.finishFullFlush() / abortFullFlushes(), with a comment saying why.

      assert setFlushingDeleteQueue(null);
      if (success) {
        // Release the flush lock
        flushControl.finishFullFlush();
      } else {
        flushControl.abortFullFlushes();
      }
      // log only once the full flush has ended: a failing InfoStream must not leave the writer in
      // full-flush mode, where every later flush is blocked
      if (infoStream.isEnabled("DW")) {
        infoStream.message(
            "DW", Thread.currentThread().getName() + " finishFullFlush success=" + success);
      }

Trade-offs

Smallest change. Without a throw the behaviour is identical except the log order: "finishFullFlush success=" now comes after the release (and after a "done stalling flushes" line, if the release logs one). No test keys on this message. With a throw, commit() still fails with the logging exception, but the writer is out of full-flush mode.

What the checks validated

• Stock: the minimal test fails with assertions on and off.
• With the fix: the minimal test passes with assertions on and off.
• Closest suites with the fix (assertions on, seed DEADBEEF): TestIndexWriterExceptions (40 tests), TestIndexWriterExceptions2, TestIndexWriterCommit, TestIndexWriterReader, TestIndexWriterNRTIsCurrent, TestIndexWriterFromReader, TestFlushByRamOrCountsPolicy, TestDocumentsWriterStallControl, TestIndexWriterWithThreads, TestIndexWriter, TestInfoStream, TestIndexWriterDelete: 12 classes, 265 tests, 0 failures, 13 skipped.
• :lucene:core:checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass.
• Earlier: the finder's test, the independent reproduction's test (with the independent reproduction's own one-hunk reorder) and the cross-model reviewers' tests pass with it, assertions on and off; the refute reviewer also ran two existing TestIndexWriterExceptions rollback tests with it (pass).
• Not run: nightly tests, benchmarks.

DocumentsWriter.java, +14/-10 (fix-finally.diff): wrap the log call in try { ... } finally { assert setFlushingDeleteQueue(null); finishFullFlush() or abortFullFlushes(); }.

Trade-offs

Keeps the log order and ends the full flush whatever the log call throws, at the cost of one more nesting level. If the release itself also throws, its exception replaces the logging one (ordinary finally semantics).

What the checks validated

• The minimal test passes with assertions on and off (same gradle run).
• The same 12 closest suites: 265 tests, 0 failures, 13 skipped; checkGoogleJavaFormat and ecjLintMain pass.

Catch RuntimeException around the message and ignore it.

Trade-offs

Hides the failure from the caller and would have to be repeated at every sibling site. IndexWriter swallows an InfoStream failure only where it already has a primary exception to report (onTragicEvent adds it to the tragedy as suppressed). Not run.

Method: bounded-exhaustive single-fault injection, then code reading (the hunt's "semi-formal" tier).

• A hunt unit studies retrying a commit after a non-tragic exception from a public callback. Its sweep ran the real lucene-core jar and made the k-th call, for every k, of InfoStream.message or isEnabled (and of other public callbacks) throw once during commit, prepareCommit, DirectoryReader.open(writer), openIfChanged, flush or flushNextBuffer, over 6 configurations, then retried the call.
• Its message pass with assertions on (2,502 cases) reported RETRYFAIL at DocumentsWriter.finishFullFlush:730 in 10 cases over commit, prepareCommit, flush and NRT open and reopen: the retried call hit "called DWFC#markForFullFlush() while full flush is still running".
• Code reading then showed the assertions-off consequence (blocked flushes, maxBufferedDocs ignored, the stall); probes confirmed it, and the finder reduced it to a small JUnit test. The independent reproducer wrote its own test; the one-method test used here is a trimmed copy of it.