IndexWriter deadlocks when another thread's merge fails while commit() waits for SerialMergeScheduler

With merge-on-commit (maxFullFlushMergeWaitMillis > 0, 500 ms by default since 9.3.0), commit() holds IndexWriter's commitLock while it calls mergeScheduler.merge(mergeSource, MergeTrigger.COMMIT). SerialMergeScheduler.merge is synchronized and runs merges on the calling thread, so if another application thread is already inside it running a merge, commit() waits at the monitor, still holding commitLock. If that merge fails (a full disk, any I/O error), IndexWriter treats the failure as tragic and closes itself on the merging thread; the close (rollbackInternal) needs commitLock while that thread still holds the scheduler's monitor. The two threads deadlock: the JVM deadlock detector lists both, Thread.interrupt() does not free them, close() and rollback() from other threads hang, and the write lock stays held until the process restarts; the last commit stays intact. The same failure closes the writer cleanly with ConcurrentMergeScheduler or with merge-on-commit off. The commit-time merge call came with LUCENE-8962 in 8.6.0 (opt-in); LUCENE-10078 made merge-on-commit the default in 9.3.0. A small change in SerialMergeScheduler (a timed lock that checks the writer for a tragedy while it waits, as ConcurrentMergeScheduler already does) makes commit() throw instead; with it the repro passes, and 184 tests in the 14 closest suites and 1,395 tests of the whole org.apache.lucene.index package pass.

1. Commit holds commitLock across its commit merges. commitInternal takes synchronized (commitLock) (IndexWriter.java:4242) and calls prepareCommitInternal (:4253), which takes it again (:3775), flushes and selects point-in-time merges (preparePointInTimeMerge(toCommit, ..., MergeTrigger.COMMIT, ...), :3863-3864). Then, still under commitLock: eventListener.beginMergeOnFullFlush (:3891), mergeScheduler.merge(mergeSource, MergeTrigger.COMMIT) (:3893), pointInTimeMerges.await(maxCommitMergeWaitMillis, ...) (:3894).
2. SerialMergeScheduler serializes merges on its own monitor. public synchronized void merge(...) (SerialMergeScheduler.java:32) loops getNextMerge() / mergeSource.merge(merge) on the calling thread (:33-39). Its javadoc says the method is "synchronized" "so that even if the application is using multiple threads, only one merge may run at a time" (:27-29).
3. A merge failure is tragic and closes the writer on the merging thread. IndexWriter.merge catches any Throwable and calls tragicEvent(t, "merge") (IndexWriter.java:5109-5113), then maybeCloseOnTragicEvent (:6090-6106), then rollbackInternal, which takes synchronized (commitLock) (:2586-2590: "Make sure no commit is running, else e.g. we can close while another thread is still fsync'ing"). The thread is still inside SerialMergeScheduler.merge, holding its monitor.
4. The cycle. The committer holds commitLock and waits for the scheduler's monitor; the merger holds the monitor and waits for commitLock. Both are blocked on monitor entry, which Thread.interrupt() cannot break.
5. Why ConcurrentMergeScheduler escapes. LUCENE-10078 made getNextMerge() and hasPendingMerges() throw once the writer has a tragedy (IndexWriter.java:2538-2542, :2558-2562). A committer stalled in ConcurrentMergeScheduler waits in wait(250) and re-checks hasPendingMerges() (ConcurrentMergeScheduler.java:609-636, :647-655), so it throws and releases commitLock. With SerialMergeScheduler the committer never gets past monitor entry.
6. Lucene avoids this cycle elsewhere. commit() and prepareCommit() run their post-commit maybeMerge outside commitLock: "we must do this outside of the commitLock else we can deadlock" (IndexWriter.java:3738, :4264, LUCENE-7570). getReader waits for its merge-on-refresh merges outside fullFlushLock "since the merge might hit a tragic event and that must not be reported while holding that lock" (:626-632, :738). The merge-on-commit call at :3893 is the one merge call still made under commitLock.

From the verifiers' probes on main and 10.5.2: Thread.interrupt() leaves both threads BLOCKED; rollback() or close() from a third thread hangs in shouldClose / doWait; a new IndexWriter or CheckIndex on the same path fails with LockObtainFailedException; other threads' addDocument, commit and NRT reopen throw AlreadyClosedException at once; a copy of the last commit opens and passes CheckIndex (the live directory stays locked). With merge-on-commit off, the same failure makes addDocument throw the IOException and commit() throw "this writer hit an unrecoverable error; cannot complete commit". With ConcurrentMergeScheduler, commit() throws "this writer hit an unrecoverable error" and both threads finish.

• Who. Applications that configure SerialMergeScheduler and call commit() while another thread's indexing call (addDocument, flush, maybeMerge, forceMerge) runs a merge inside it. Merge-on-commit is on by default since 9.3.0 (500 ms; the stock TieredMergePolicy and LogMergePolicy select small-segment merges at commit). In 8.6.0-9.2.x it needs an explicit nonzero wait and a policy that implements findFullFlushMerges. ConcurrentMergeScheduler, the default, is not affected.
• Trigger. Any merge failure at that moment: a full disk, another I/O error, or an exception from inside a merge. The window: the other thread's merge must fail while a commit waits at its commit-merge call (IndexWriter.java:3893), after the commit has selected commit merges and before that merge ends.
• Effect. A permanent hang instead of a clean tragic close: two threads stuck, close() and rollback() hang, the write lock stays held, so the index cannot be written again until the process restarts. Documents added since the last commit are lost, as with the clean close.
• Not affected. The last commit stays intact: a copy of it opens and passes CheckIndex (the live directory stays locked). No corruption and no wrong search results were seen.
• How likely. Rare: SerialMergeScheduler with concurrent commits is uncommon and merge failures are rare events. It was first seen unforced in a randomized stress run. Lucene's own randomized tests pick SerialMergeScheduler rarely and always with a merge-on-commit wait of at least 200 ms.
• Workaround. With SerialMergeScheduler, setMaxFullFlushMergeWaitMillis(0); or ConcurrentMergeScheduler (setMaxMergesAndThreads(1, 1) for one merge at a time, as mikemccand suggested in #6727). Executed: the verifiers' wait-0 and default-ConcurrentMergeScheduler controls, and a setMaxMergesAndThreads(1, 1) run (commit() throws, the writer closes, no deadlock).
• Not shown. prepareCommit() and close() with commitOnClose reach the same prepareCommitInternal (by reading, not executed).

The two hooks only make the order deterministic: FilterDirectory (@lucene.internal) throws the full-disk stand-in from createOutput for a merge, and IndexWriterEventListener (@lucene.experimental; beginMergeOnFullFlush is called under commitLock just before the commit-merge call, IndexWriter.java:3891) tells the fault when to fire. The threads are daemons and the class is @ThreadLeakScope(NONE) because deadlocked threads cannot be stopped. setMinMergeDocs(2) is needed, else the first merge packs all three segments and the commit has nothing to merge; the policy is switched live after the three flushes so they are not merged early. On stock the test waits 10 s for commit() and then fails; with the fix it passes in under a second. It asserts only that no thread is in a JVM monitor deadlock; what each call returns is shown by a separate outcome check (Fixes). It uses the 10.x IOContext.context() accessor, so it compiles against 10.x and main only; a version-portable main() covers 8.6.0-9.12.3. Its unnamed catch variable (catch (Exception _)) needs Java 22 or later: fine on main, but name the variable in a 10.x source checkout, which compiles with --release 21.

Regression test: this same test (a monitor-deadlock check), next to testDeadlockStalledFullFlushMerges in TestTragicIndexWriterDeadlock (same package).

cp $REPRO/TestSerialMergeSchedulerCommitDeadlock.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestSerialMergeSchedulerCommitDeadlock -Ptests.seed=DEADBEEF
# the same result with -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF (assertions on; the same with -Ptests.asserts=false)
TestSerialMergeSchedulerCommitDeadlock > testMergeFailureDuringMergeOnCommit FAILED
    java.lang.AssertionError: expected:<[]> but was:<[committer, merger]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:6E99721BC49A8BED]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.index.TestSerialMergeSchedulerCommitDeadlock.testMergeFailureDuringMergeOnCommit(TestSerialMergeSchedulerCommitDeadlock.java:91)

# thread stacks at that point (a copy of the test that only adds a stack print, -da;
# test and lambda frames left out; the Object is IndexWriter's commitLock)
merger BLOCKED on java.lang.Object@5afd30d owned by committer
  at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2590)
  at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:6104)
  at org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:6094)
  at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:5112)
  at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6884)
  at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:38)
  at org.apache.lucene.index.IndexWriter.executeMerge(IndexWriter.java:2451)
  at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2446)
  at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2439)
committer BLOCKED on org.apache.lucene.index.SerialMergeScheduler@72665766 owned by merger
  at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:34)
  at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3893)
  at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4253)
  at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4215)

# with the fix: 1 test, 0 failures (assertions on and off), in 0.58 s with assertions on and 0.77 s off, instead of a 10 s wait

• The contract. "IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently" (IndexWriter.java:167-170). getTragicException: "If this IndexWriter was closed as a side-effect of a tragic exception, e.g. disk full while flushing a new segment, this returns the root cause exception" (:6108-6111). A disk-full merge is meant to close the writer, not hang it.
• IndexWriter's own rule. "we must do this outside of the commitLock else we can deadlock" (IndexWriter.java:3738, :4264), from LUCENE-7570 (#8621, 2b073a2f296, 2016: "don't run merges while holding the commitLock to prevent deadlock when merges are stalled and a tragic merge exception strikes"). Merge-on-commit (2020) added a new merge call under commitLock.
• The test. TestTragicIndexWriterDeadlock.testDeadlockStalledFullFlushMerges (TestTragicIndexWriterDeadlock.java:106-172, LUCENE-10078) asserts that commit() throws "this writer hit an unrecoverable error" when a merge fails during merge-on-commit. It uses ConcurrentMergeScheduler only.
• A supported combination. SerialMergeScheduler is public and not deprecated, its javadoc names multi-threaded applications (SerialMergeScheduler.java:27-29), IndexWriterConfig.java:495-497 documents it with merge-on-commit, and LuceneTestCase combines the two (LuceneTestCaseParent.java:1710-1711, :1788).

The strongest case: the setMaxFullFlushMergeWaitMillis javadoc says "If SerialMergeScheduler is used and a non-zero timeout is configured, full-flush merges will always wait for the merge to finish without honoring the configured timeout" (IndexWriterConfig.java:495-497). In the 2020 review of this exact call (apache/lucene-solr#1552), s1monw asked "what if we has a MergeScheduler configured that blocks on this call like SerialMergeScheduler? ... I think we should make sure that this is not trappy", and then concluded "its rather a rare combination ... I think we are fine as it is"; LUCENE-10078 (#921) added the javadoc note. "This blocking in SMS is by design" (mikemccand, #6374, answering s1monw's proposal there to "use a tryLock operation instead of syncing the entire method"), mikemccand also proposed removing SerialMergeScheduler (#6727), and merge scheduling is an expert API.

Why it fails: the note and the review accept a wait for a merge to finish. Here the merge has already failed, and its thread can never leave SerialMergeScheduler.merge, because the tragic close it runs needs the lock the waiting commit holds; nobody in #1552 or #921 discussed a failing merge, the tragic rollback or commitLock. Blocking by design is a wait that ends when the merge ends; the recommended fix keeps that blocking (its tryLock only stops waiting once the writer has a tragedy), so it is not the #6374 change. #6727 is still open (rmuir objected), and the class is public and not deprecated. Lucene fixed this lock pair as a bug at the post-commit merge call (LUCENE-7570), and LUCENE-10078, which turned full-flush merges on by default, added the tragedy checks and a test that make the ConcurrentMergeScheduler variant throw instead of hang. Verdict: not intended. (s1monw flagged this call for SerialMergeScheduler in that review, as a wait, not a deadlock, and proposed a tryLock in #6374; credit them where the issue cites either.)

One fix was validated in a scratch worktree of 645d4fe14a20; the others are directions, not attempted. The cycle can be broken on either side: the committer must not wait uninterruptibly for the scheduler while it holds commitLock, or the tragic close of a merging thread must not wait for commitLock while it holds the scheduler.

One file, +27/-8 in SerialMergeScheduler.java. Merges stay serial (one ReentrantLock, reentrant like the monitor). A waiting thread retries tryLock(250, MILLISECONDS) and calls mergeSource.hasPendingMerges() in between, which throws IllegalStateException ("this writer hit an unrecoverable error; cannot merge") once the writer has a tragedy (IndexWriter.java:2558-2562): the same check that lets ConcurrentMergeScheduler's stalled committer escape. Without a tragedy the behaviour is unchanged: the committer waits for the other thread's merges, then runs its commit merges. Excerpt below (the new field and merge()).

  private final ReentrantLock mergeLock = new ReentrantLock();

  @Override
  public void merge(MergeSource mergeSource, MergeTrigger trigger) throws IOException {
    // Do not block forever while another thread merges: IndexWriter calls this for merge-on-commit
    // while it holds its commit lock, and if the other thread's merge hits a tragic event, its
    // rollback waits for that commit lock. hasPendingMerges() throws once the writer hit a tragedy.
    try {
      while (mergeLock.tryLock(250, TimeUnit.MILLISECONDS) == false) {
        mergeSource.hasPendingMerges();
      }
    } catch (InterruptedException ie) {
      throw new ThreadInterruptedException(ie);
    }
    try {
      while (true) {
        MergePolicy.OneMerge merge = mergeSource.getNextMerge();
        if (merge == null) {
          break;
        }
        mergeSource.merge(merge);
      }
    } finally {
      mergeLock.unlock();
    }
  }

Trade-offs

• A subclass whose merge override is itself synchronized keeps the deadlock: it locks the same scheduler object before super.merge reaches the new lock. Lucene's tests have three overrides that call super.merge, two synchronized (TestIndexWriterMergePolicy.java:568, BaseMergePolicyTestCase.java:83) and one not (:631); they pass with the fix, but the two would need synchronized dropped to be covered.
• Compatibility: SerialMergeScheduler no longer synchronizes on itself, so code that locked the scheduler object to keep merges out would no longer do so (worth a CHANGES note).
• A waiting thread wakes every 250 ms (the period ConcurrentMergeScheduler.doStall uses).
• After a tragedy a waiting thread throws at its next check, at most 250 ms later (in the outcome check the committing thread ended 262 ms after it started), instead of after the failing thread's rollback; it is the exception getNextMerge() would throw to it anyway.
• Interrupting a waiting thread now throws ThreadInterruptedException, as the IndexWriter javadoc describes for waits; the monitor ignored interrupts.
• It fixes the SerialMergeScheduler case only: a custom MergeScheduler that blocks the same way under commitLock keeps the hazard. The escape relies on hasPendingMerges() throwing after a tragedy, which IndexWriter's own MergeSource does (a method synchronized on the writer).

What the checks validated

• Stock: the minimal test fails with assertions on and off.
• With the fix: the minimal test passes with assertions on and off (0.58 s and 0.77 s instead of a 10 s wait), and 10 of 10 iterations with -Ptests.iters=10.
• Closest suites with the fix: TestTragicIndexWriterDeadlock, TestIndexWriterMergePolicy (its merge-on-commit and merge-on-getReader tests subclass SerialMergeScheduler), TestIndexWriterOnDiskFull, TestIndexWriterCommit, TestIndexWriterMerging, TestIndexWriterExceptions2, TestIndexWriterDelete, TestAddIndexes, TestDeletionPolicy, TestIndexFileDeleter, TestIndexWriterConfig, TestTieredMergePolicy, TestLogMergePolicy, TestIndexWriterOnError: 14 classes, 184 tests, 0 failures, 15 skipped.
• The whole org.apache.lucene.index package with the fix: 214 classes, 1,395 tests, 0 failures, 56 skipped (the 214 include TestClassloadingDeadlock, a plain JUnit test that Gradle reports under the name "Gradle Test Run :lucene:core:test").
• :lucene:core:checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass.
• Outcome check (not the repro): commit() returns after 262 ms with "this writer hit an unrecoverable error; cannot merge" (cause: the IOException), maybeMerge() throws the IOException, the writer is closed, and a new IndexWriter opens the same directory.
• Not run: nightly tests, other modules, benchmarks; one seed (DEADBEEF) for the suites.

Return at once for MergeTrigger.COMMIT (one of the options s1monw listed in the apache/lucene-solr#1552 review). The commit then waits at most maxFullFlushMergeWaitMillis and the pending merges run at the next merge call.

Trade-offs

Turns merge-on-commit off for SerialMergeScheduler users and changes behaviour that IndexWriterConfig.java:495-497 documents. Not attempted.

Run the commit's point-in-time merges without holding commitLock, as LUCENE-7570 did for the post-commit merge.

Trade-offs

Fixes every scheduler, but commitLock protects the commit point from prepare to finish (prepareCommit/commit pairing, rollback exclusion); a larger change in a delicate path. Not attempted.

Record the tragedy, leave the scheduler, and close afterwards (or let the commit thread close the writer).

Trade-offs

Changes when the writer closes on a tragedy for every scheduler; LUCENE-6790 added the commitLock wait on purpose ("block commit while rollback runs"). Not attempted.

With SerialMergeScheduler, setMaxFullFlushMergeWaitMillis(0); or use ConcurrentMergeScheduler (setMaxMergesAndThreads(1, 1) for one merge at a time, as mikemccand suggested in #6727).

Trade-offs

Executed: the verifiers' wait-0 and default-ConcurrentMergeScheduler controls, and a setMaxMergesAndThreads(1, 1) run; the same failure then closes the writer cleanly.

Method: non-formal baseline (a randomized stress harness, not a model checker).

• The harness (stage 3: merge on full flush with SerialMergeScheduler, injected merge I/O faults) hung in round 29 of a local run; its JVM thread dump showed commit() BLOCKED at SerialMergeScheduler.merge from prepareCommitInternal and an indexing thread BLOCKED in rollbackInternal from tragicEvent.
• The finder reduced it to a deterministic JUnit with a scripted merge policy; an independent reproducer wrote its own main(), and a devil's-advocate verifier wrote a stock-policy JUnit (default wait, LogDocMergePolicy, a simulated "No space left on device"). The minimal test here is that stock-policy test cut down to one method.