Each thread holds one of the writer's internal locks and waits for the lock the next thread holds.
- The IndexWriter lock (the writer's monitor) is held by synchronized writer methods such as
tryDeleteDocument, and while a flushed segment is added to the index. - The DocumentsWriter lock is the monitor of the writer's in-memory buffer manager. It is held while a sequence number is handed out, and while
deleteAllaborts the in-memory buffers. - The purge lock guards the queue of flushed segments that wait to be added to the index. The thread that publishes them holds it until each one has been added under the IndexWriter lock.

- Open a writer with
maxBufferedDocs=2, add 1 document and open an NRT reader. Thread 1 callstryDeleteDocument(reader, 0): it holds the IndexWriter lock (and thebufferedUpdatesStreamlock) while it deletes the document and drops the now-empty segment (the test pauses it there once), and only then asks for a sequence number. - Thread 2 adds 2 documents. The second add flushes a new segment and publishes it itself: it takes the flush queue's purge lock, then waits for the IndexWriter lock, which thread 1 holds.
- Thread 3 calls
deleteAll(). It takes the DocumentsWriter lock, then waits for the purge lock, which thread 2 holds. - Thread 1 continues. You expect all three calls to return within milliseconds (they do with the fix, 20 of 20 iterations). Instead thread 1's sequence number needs the DocumentsWriter lock, which thread 3 holds: a cycle.
- The JVM's deadlock detector reports 3 deadlocked threads, with assertions on and off, in every recorded run. The writer calls throw nothing (the test reports the cycle with an
AssertionError), and a probe shows latercommit(),getReader(),rollback()andclose()calls on that writer blocking too.
Applications that call deleteAll() while other threads index and, at the same time, use addIndexes or the Expert tryDeleteDocument/tryUpdateDocValue. By code reading no non-default setting is needed, and no scale. Rare, since three calls must meet in short windows, but when hit the writer is lost: later commit(), getReader(), rollback() and close() calls block too (executed).
Mechanism, terms and numbers546 words
Three threads share one IndexWriter: one calls deleteAll(), one calls addDocument() and that call flushes and publishes a new segment, and one calls tryDeleteDocument() on an NRT reader (or addIndexes(...)). The IndexWriter javadoc says instances are "completely thread safe, meaning multiple threads can call any of its methods, concurrently". Instead they can deadlock: deleteAll() holds the DocumentsWriter monitor (in lockAndAbortAll) and waits for the flush-ticket purge lock, the publishing thread holds that purge lock and waits for the IndexWriter monitor (publishFlushedSegment), and tryDeleteDocument() (or addIndexes()) holds the IndexWriter monitor and waits for the DocumentsWriter monitor to get a sequence number (getNextSequenceNumber). The writer calls throw nothing, with assertions on and off, and later commit(), getReader(), rollback() and close() calls on that writer block too (executed). The edge that closes the cycle came with LUCENE-9304 in 8.6.0 (2020); every release since, through 10.6.0, and main are affected (seven releases and main executed, the rest by code reading).
| edge | holds | waits for | where |
|---|---|---|---|
| 1 | DocumentsWriter monitor | purge lock | deleteAll (IndexWriter.java:2750) takes fullFlushLock (:2764) and calls docWriter.lockAndAbortAll() (:2765). That method is synchronized (DocumentsWriter.java:266) and starts with ticketQueue.forcePurge(...) (:271), which calls purgeLock.lock() (DocumentsWriterFlushQueue.java:118). |
| 2 | purge lock | IndexWriter monitor | After a flush, afterSegmentsFlushed() calls publishFlushedSegments(false) (IndexWriter.java:448-449), through DocumentsWriter.purgeFlushTickets (DocumentsWriter.java:188-195) to tryPurge (tryLock(), DocumentsWriterFlushQueue.java:128). innerPurge runs the consumer while it holds the purge lock (:82-114, call at :99); the consumer calls the synchronized publishFlushedSegment (IndexWriter.java:6290, declared at :2953). |
| 3 | IndexWriter monitor | DocumentsWriter monitor | public synchronized long tryDeleteDocument (IndexWriter.java:1688) calls the synchronized tryModifyDocument (:1791), which ends in return docWriter.getNextSequenceNumber(); (:1824). getNextSequenceNumber is synchronized on the DocumentsWriter (DocumentsWriter.java:583-586). |
Other callers of edge 3 (read): tryUpdateDocValue (IndexWriter.java:1721, the same tryModifyDocument), addIndexes(Directory...) (synchronized (this) at :3194, getNextSequenceNumber at :3201) and addIndexes(CodecReader...) when its merge succeeds (synchronized (this) at :3350, getNextSequenceNumber at :3369; its branch for an empty merge specification takes the sequence number at :3337, outside the IndexWriter monitor). A separate executed test with the merging addIndexes(CodecReader...) in place of tryDeleteDocument forms the same cycle (main, assertions off); the other two paths were only read. deleteAll itself also calls getNextSequenceNumber inside the IndexWriter monitor (:2801), after lockAndAbortAll has returned and released the DocumentsWriter monitor; that is not part of this cycle.
The window the test widens. Inside tryDeleteDocument's closure the writer calls rld.delete, then isFullyDeleted(rld) (IndexWriter.java:1694-1695), which asks MergePolicy.keepFullyDeletedSegment (:6428-6431), then dropDeletedSegment and checkpoint() (:1696-1697), all inside the IndexWriter monitor and before getNextSequenceNumber. The test pauses once in that callback; the pause takes no new lock, but it runs while tryDeleteDocument already holds the IndexWriter monitor and the bufferedUpdatesStream monitor (:1822).
The thread dump on main (gradle run):
tryDelete BLOCKED on DocumentsWriter owned by deleteAll at getNextSequenceNumber <- tryModifyDocument <- tryDeleteDocument
deleteAll WAITING on ReentrantLock owned by indexer at forcePurge <- lockAndAbortAll (locked DocumentsWriter)
indexer BLOCKED on IndexWriter owned by tryDelete at publishFlushedSegment <- innerPurge <- tryPurge <- afterSegmentsFlushedExposure and likelihood351 words
- Effect. The three calls hang with no exception or log line, assertions on and off (executed). The writer is then unusable. A probe that runs after the cycle has formed (main, assertions off, InfoStream off as in a default config) shows later
commit(),DirectoryReader.open(w)and a seconddeleteAll()blocked onfullFlushLock, a fourth lock that the stuckdeleteAlltook atIndexWriter.java:2764, androllback()andclose()blocked on the IndexWriter monitor inshouldClose(:1374) (executed). With the test framework's InfoStream,commit()blocks on the IndexWriter monitor first, in the synchronizedsegString()(:5840). Later automatic flushes do not wait for the purge lock (tryPurgegives up,DocumentsWriterFlushQueue.java:128), so their segments stay unpublished (read). Monitor waits andReentrantLock.lock()ignore interrupts, so no in-process way to recover was found (read). - Who. Applications that call
deleteAll()while other threads index, and that also calltryDeleteDocumentortryUpdateDocValue(Expert: delete or update by NRT doc id) oraddIndexes(...)(not Expert) at the same time.deleteAll's own javadoc anticipates concurrentaddIndexes(CodecReader[])calls (IndexWriter.java:2743-2745).deleteDocuments(new MatchAllDocsQuery())is routed todeleteAll()(IndexWriter.java:1886-1890), so it is a second way in (read, not executed). - Default vs opt-in. By code reading no non-default setting is needed. The indexing thread needs only an automatic flush that it completes and publishes itself (a
maxBufferedDocsor RAM-buffer flush); an explicitflush()takesfullFlushLockand publishes on another path (IndexWriter.java:4399-4412). The repro setsmaxBufferedDocs=2(so that the second add flushes) and a merge policy that pauses one thread once, only to fix the schedule; it runs with the test framework's randomized codec and also fails with the plainLucene104codec forced (executed). - Scale. None: three documents. At the start there is one 1-document segment; at the deadlock it has been dropped and the new 2-document segment is flushed but not yet published.
- Likelihood: rare.
deleteAllmust enterlockAndAbortAllwhile the indexer is publishing, and the third thread must be inside its IndexWriter-monitor section just before it asks for a sequence number. No public report of it was found (see prior reports).
One JUnit test, three threads (the cycle needs three), three empty documents, public API only. The only pause, in the public MergePolicy.keepFullyDeletedSegment callback, takes no new lock and ends before the deadlock forms. The test waits on thread states, not on time; its only bug signal is the JVM's deadlock detector (a guard assertion fails if a thread ends early).
package org.apache.lucene.index;
import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadMXBean;
import java.util.Arrays;
import java.util.concurrent.Callable;
import java.util.concurrent.Semaphore;
import org.apache.lucene.document.Document;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.util.IOSupplier;
@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
public class TestDeleteAllDeadlock extends LuceneTestCase {
// Expected: deleteAll, a flushing addDocument and tryDeleteDocument all return; actual: they deadlock.
public void testDeleteAllWithFlushAndTryDelete() throws Exception {
Semaphore resume = new Semaphore(0);
MergePolicy mp =
new FilterMergePolicy(NoMergePolicy.INSTANCE) {
@Override
public boolean keepFullyDeletedSegment(IOSupplier<CodecReader> reader) {
if (Thread.currentThread().getName().equals("tryDelete")) {
resume.acquireUninterruptibly();
}
return false;
}
};
IndexWriter w =
new IndexWriter(
new ByteBuffersDirectory(),
new IndexWriterConfig().setMergePolicy(mp).setMaxBufferedDocs(2));
w.addDocument(new Document());
DirectoryReader r = DirectoryReader.open(w);
Thread tryDelete = start("tryDelete", () -> w.tryDeleteDocument(r, 0));
awaitState(tryDelete, Thread.State.WAITING);
Thread indexer =
start(
"indexer",
() -> {
w.addDocument(new Document());
return w.addDocument(new Document());
});
awaitState(indexer, Thread.State.BLOCKED);
Thread deleteAll = start("deleteAll", w::deleteAll);
awaitState(deleteAll, Thread.State.WAITING);
resume.release();
ThreadMXBean mx = ManagementFactory.getThreadMXBean();
while (tryDelete.isAlive() && mx.findDeadlockedThreads() == null) {
Thread.onSpinWait();
}
long[] deadlocked = mx.findDeadlockedThreads();
if (deadlocked != null) {
fail(Arrays.toString(mx.getThreadInfo(deadlocked, true, true)));
}
indexer.join();
deleteAll.join();
r.close();
w.close();
}
private static Thread start(String name, Callable<?> body) {
Thread t =
new Thread(
() -> {
try {
body.call();
} catch (Exception e) {
throw new RuntimeException(e);
}
},
name);
t.start();
return t;
}
private static void awaitState(Thread t, Thread.State state) {
while (t.getState() != state) {
assertTrue(t.getName() + " ended early", t.isAlive());
Thread.onSpinWait();
}
}
}
cd $LUCENE && ./gradlew :lucene:core:test --tests TestDeleteAllDeadlock -Ptests.seed=DEADBEEF
- Expected
- All three calls return:
tryDeleteDocumentgets its sequence number, the flushed segment is published anddeleteAllfinishes (as with the fix). - Observed
AssertionErrorlisting 3 JVM-detected deadlocked threads, stuck ingetNextSequenceNumber,forcePurgeandpublishFlushedSegment, with assertions on and off.
More files, environment, full output481 words
The test uses IndexWriter, IndexWriterConfig, FilterMergePolicy, DirectoryReader.open(IndexWriter) and ByteBuffersDirectory. It sets maxBufferedDocs=2 (so that the second add flushes) and a merge policy that pauses one thread once, only to fix the schedule; the codec is the test framework's randomized one (it also fails with the plain Lucene104 codec forced). It sits in org.apache.lucene.index only for placement next to TestIndexWriterDelete; it has no license header (add the ASF header).
It is deterministic: it starts each thread only after the previous one has reached its state (WAITING in the callback, BLOCKED in the publish, WAITING for the purge lock), and the test itself draws no random values (the seed only drives the test framework's own randomization). It failed in every recorded run on main (gradle and a standalone runner, assertions on and off, seed DEADBEEF; 5 of 5 separate JVMs with seeds 5EED0001 to 5EED0005) and on the 10.6.0, 10.5.2 and 9.12.3 release jars, and it passes with the fix (20 of 20 iterations). Its only bug signal is the detector's fail (line 57); a guard assertion fails the test if a thread ends before reaching its expected state (lines 80-85), and a worker rethrows any exception (line 72). As a regression test it has no deadline of its own (its waits and join() calls are unbounded; the test framework's timeout applies), and @ThreadLeakScope(NONE) is there because deadlocked threads cannot be stopped.
Regression test: this same test; it passes with the fix and fits in TestIndexWriterDelete.
| lucene | main 5adff78bb168 (12.0.0-SNAPSHOT): fails; with the fix: passes. 10.6.0, 10.5.2 and 9.12.3 release jars: fail (assertions on and off) |
|---|---|
| jdk | OpenJDK 25.0.3 (build 25.0.3+9-LTS) |
| os | Linux, aarch64 (the only platform run; the code path is pure Java) |
cp $REPRO/TestDeleteAllDeadlock.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestDeleteAllDeadlock -Ptests.seed=DEADBEEF
# the same result with -Ptests.asserts=false
# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF -Ptests.asserts=false (the same with true);
# trimmed: the JVM-generated $$Lambda frames, the test's thread-start frames (lambda$start$0, Thread.runWith) and the randomizedtesting seed line; each '...' is the JVM's own cut
TestDeleteAllDeadlock > testDeleteAllWithFlushAndTryDelete FAILED
java.lang.AssertionError: ["tryDelete" prio=5 Id=21 BLOCKED on org.apache.lucene.index.DocumentsWriter@2d046055 owned by "deleteAll" Id=23
at app//org.apache.lucene.index.DocumentsWriter.getNextSequenceNumber(DocumentsWriter.java:585)
- blocked on org.apache.lucene.index.DocumentsWriter@2d046055
at app//org.apache.lucene.index.IndexWriter.tryModifyDocument(IndexWriter.java:1824)
- locked org.apache.lucene.index.BufferedUpdatesStream@7d4a9b11
- locked org.apache.lucene.index.IndexWriter@7632cea5
at app//org.apache.lucene.index.IndexWriter.tryDeleteDocument(IndexWriter.java:1690)
- locked org.apache.lucene.index.IndexWriter@7632cea5
at app//org.apache.lucene.index.TestDeleteAllDeadlock.lambda$testDeleteAllWithFlushAndTryDelete$0(TestDeleteAllDeadlock.java:37)
...
, "deleteAll" prio=5 Id=23 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@26cd8a9e owned by "indexer" Id=22
at java.base@25.0.3/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@26cd8a9e
at java.base@25.0.3/java.util.concurrent.locks.LockSupport.park(LockSupport.java:223)
at java.base@25.0.3/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:790)
at java.base@25.0.3/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1030)
at java.base@25.0.3/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:154)
at java.base@25.0.3/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:323)
at app//org.apache.lucene.index.DocumentsWriterFlushQueue.forcePurge(DocumentsWriterFlushQueue.java:118)
at app//org.apache.lucene.index.DocumentsWriter.lockAndAbortAll(DocumentsWriter.java:271)
- locked org.apache.lucene.index.DocumentsWriter@2d046055
...
, "indexer" prio=5 Id=22 BLOCKED on org.apache.lucene.index.IndexWriter@7632cea5 owned by "tryDelete" Id=21
at app//org.apache.lucene.index.IndexWriter.publishFlushedSegment(IndexWriter.java:2960)
- blocked on org.apache.lucene.index.IndexWriter@7632cea5
at app//org.apache.lucene.index.IndexWriter.lambda$publishFlushedSegments$0(IndexWriter.java:6290)
at app//org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:99)
at app//org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:130)
at app//org.apache.lucene.index.DocumentsWriter.purgeFlushTickets(DocumentsWriter.java:193)
at app//org.apache.lucene.index.IndexWriter.publishFlushedSegments(IndexWriter.java:6266)
at app//org.apache.lucene.index.IndexWriter$1.afterSegmentsFlushed(IndexWriter.java:449)
...
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync@26cd8a9e
]
at org.junit.Assert.fail(Assert.java:89)
at org.apache.lucene.index.TestDeleteAllDeadlock.testDeleteAllWithFlushAndTryDelete(TestDeleteAllDeadlock.java:57)
# with the fix (option A): TestDeleteAllDeadlock tests=1 failures=0 (assertions on and off);
# -Ptests.iters=20: tests=20 failures=0- IndexWriter's javadoc: "completely thread safe, meaning multiple threads can call any of its methods, concurrently".
deleteAll's javadoc expects concurrentaddIndexescallers and namesMergeAbortedExceptionas the outcome, not a hang.- It came in 8.6.0 with a sequence-number fix (LUCENE-9304) whose review material does not discuss lock order; an earlier
deleteAlldeadlock (#6066) was fixed as a bug.
The full argument229 words
- Contract (
IndexWriter.java:167-170): "IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently. If your application requires external synchronization, you should not synchronize on the IndexWriter instance as this may cause deadlock; use your own (non-Lucene) objects instead." The test does not synchronize on the writer. - deleteAll's javadoc (
IndexWriter.java:2743-2745): "If other threads are running forceMerge, addIndexes(CodecReader[]) or forceMergeDeletes methods, they may receive MergePolicy.MergeAbortedExceptions." - The code implies a lock order elsewhere.
innerPurge(DocumentsWriterFlushQueue.java:93-98): "if we block on publish -> lock IW -> lock BufferedDeletes we don't block concurrent segment flushes";publishFrozenUpdates(IndexWriter.java:2931-2932): "Do this as an event so it applies higher in the stack when we are not holding DocumentsWriterFlushQueue.purgeLock". These comments and the implementation imply that the purge lock comes before the IndexWriter monitor (there is no written lock-order contract);lockAndAbortAlltakes it under the DocumentsWriter monitor, and edge 3 takes the DocumentsWriter monitor under the IndexWriter monitor. - Tests.
TestIndexWriterDelete.testDeleteAllNoDeadLock(TestIndexWriterDelete.java:326) runsdeleteAllwhile other threads index; it has notryDeleteDocumentoraddIndexesthread. No test or comment accepts this cycle. - History. LUCENE-5002 (#6066), an earlier
deleteAlldeadlock, was fixed in 4.3.1. The edge that closes this cycle came with a fix for a sequence-number race (LUCENE-9304, apache/lucene-solr#1427), whose review material (10 review comments, 6 reviews, 1 issue comment) does not discuss lock order.
The devil's-advocate check149 words
Strongest case: "the test forces a schedule by pausing inside tryDeleteDocument, and tryDeleteDocument is Expert; real windows are tiny."
Why it fails: (1) the pause takes no new lock and is lifted before the deadlock forms. It runs inside the IndexWriter monitor that tryDeleteDocument already holds, which is the window itself: it only lengthens a window that exists anyway (the delete, dropDeletedSegment and checkpoint() run inside the IndexWriter monitor before the sequence number is taken), as preemption or slow file I/O would; after the release only Lucene's own locks are involved and the JVM itself reports the cycle. (2) Expert is not "unsupported": the thread-safety contract covers every method, and a separate executed test with the non-Expert addIndexes(CodecReader...) forms the same cycle. (3) Rare is not intended: Lucene has fixed rare IndexWriter deadlocks before (#6066, #12654). The cross-model reviewers (OpenAI GPT-5.6) rated the "works as designed" case weak.
Recommended A: purge the ticket queue before taking the DocumentsWriter monitor (sketch; validated by the runs below, not proven safe)
In DocumentsWriter.lockAndAbortAll, run the ticket-queue forcePurge before taking the DocumentsWriter lock (+9/-2 lines), which removes one edge of the cycle. Tested, not proven safe: the test passes, and so do 290 tests of the 12 closest suites and the whole org.apache.lucene.index package (1,397 tests). Design it together with LFV-474's fix.
Closeable lockAndAbortAll() throws IOException { // was: synchronized
// ... infoStream message unchanged
// Purge before taking this monitor: a thread holding the purge lock may be waiting for the
// IndexWriter monitor, whose owner may be waiting for this monitor (getNextSequenceNumber).
assert Thread.holdsLock(this) == false;
ticketQueue.forcePurge(
ticket -> {
if (ticket.getFlushedSegment() != null) {
pendingNumDocs.addAndGet(-ticket.getFlushedSegment().segmentInfo.info.maxDoc());
}
});
return lockAndAbortAllAfterPurge();
}
private synchronized Closeable lockAndAbortAllAfterPurge() throws IOException { /* rest of the old body */ }All options and trade-offs3 options ยท 634 words
One fix sketch was tested with gradle in a scratch worktree; two alternatives were considered and not tried.
A: purge the ticket queue before taking the DocumentsWriter monitor (sketch; validated by the runs below, not proven safe) (recommended)
lockAndAbortAll stops being synchronized: it runs ticketQueue.forcePurge(...) first and then calls a new synchronized lockAndAbortAllAfterPurge() with the rest of the old body. This removes edge 1 (DocumentsWriter monitor -> purge lock); edges 2 and 3 stay. The snippet is condensed.
Why it looks safe, and what it changes (read): lockAndAbortAll has one caller, deleteAll (IndexWriter.java:2765), which still holds fullFlushLock during the purge. Tickets are added under the ticket queue's own monitor (DocumentsWriterFlushQueue.java:38), and a segment flush adds its ticket without the DocumentsWriter monitor (DocumentsWriter.java:535), so the purge under that monitor never kept new tickets out: for adding tickets the fix opens no new window. It does open one for the other users of the DocumentsWriter monitor: getNextSequenceNumber, applyDeleteOrUpdate (deletes and doc-values updates) and resetDeleteQueue (DocumentsWriter.java:583, :153, :588) can now run between the purge and the abort. That is exactly what breaks the cycle (tryDeleteDocument gets its sequence number), and fullFlushLock does not restore that exclusion.
Closeable lockAndAbortAll() throws IOException { // was: synchronized
// ... infoStream message unchanged
// Purge before taking this monitor: a thread holding the purge lock may be waiting for the
// IndexWriter monitor, whose owner may be waiting for this monitor (getNextSequenceNumber).
assert Thread.holdsLock(this) == false;
ticketQueue.forcePurge(
ticket -> {
if (ticket.getFlushedSegment() != null) {
pendingNumDocs.addAndGet(-ticket.getFlushedSegment().segmentInfo.info.maxDoc());
}
});
return lockAndAbortAllAfterPurge();
}
private synchronized Closeable lockAndAbortAllAfterPurge() throws IOException { /* rest of the old body */ }Trade-offs
- A maintainer must confirm that no invariant needs the purge and the abort under one hold of the DocumentsWriter monitor.
- LFV-474 (
deleteAllmisses a segment flush that is already running) has its fix site in the same method; its sketch adds a secondforcePurgethere, which would re-create edge 1 if it ran inside the monitor. The two fixes need one design (moving that purge out too is one option; a maintainer may choose another lock order or serialization); review them together. - The full core suite was not run.
What the checks validated
- Gradle
--no-daemon, OpenJDK 25.0.3, scratch worktree of5adff78bb168, 2026-11-02 (removed afterwards):- stock: the minimal test fails with assertions on and off;
- with the fix: the minimal test passes with assertions on and off, and 20 of 20 iterations (seed 5EED1137);
- 12 closest suites (
TestIndexWriterDelete,TestIndexingSequenceNumbers,TestIndexWriterWithThreads,TestTryDelete,TestAddIndexes,TestIndexWriter,TestIndexWriterExceptions,TestIndexWriterReader,TestIndexWriterThreadsToSegments,TestRollingUpdates,TestFlushByRamOrCountsPolicy,TestDocumentsWriterStallControl): 290 tests, 0 failures, 16 skipped; - the whole
org.apache.lucene.indexpackage of core: 214 classes, 1,397 tests, 0 failures, 56 skipped; checkGoogleJavaFormat,ecjLintMain,forbiddenApisMainon core: BUILD SUCCESSFUL.
- Earlier, independent checks with the same diff: the finder's test, the independent reproducer's test and the refute reviewer's tests pass. The intended-lens reviewer's first run with the fix failed only because its test still expected the old
BLOCKEDstate (with the fix that thread finishes); its corrected test passes.
Give tryModifyDocument and addIndexes their sequence number inside the IndexWriter monitor without taking the DocumentsWriter monitor (removes edge 3).
Trade-offs
Not recommended. getNextSequenceNumber became synchronized to close the LUCENE-9304 race ("the delete queue might change concurrently"), and later sequence-number work (#13169) relies on it. On 8.6.0, changing tryModifyDocument's call back to deleteQueue.getNextSequenceNumber() makes the deadlock go away, which shows causation but reopens that race.
Say that deleteAll must not run concurrently with tryDeleteDocument, tryUpdateDocValue or addIndexes.
Trade-offs
Contradicts the class javadoc and deleteAll's own note on concurrent addIndexes.
A service that lets clients request a delete-everything operation while it indexes and runs addIndexes or tryDeleteDocument could see its writer hang; whether clients may request that is the service's own choice.
A proof-obligation audit of DocumentsWriter's lock order (the obligation: the writer's locks are always taken in one order) found three edges that form a cycle. An executable JUnit test witnessed it, and an independent reproducer, cross-model reviews and an adjudication confirmed it.
The method in full138 words
Method: a proof-obligation audit of DocumentsWriter's lock order, witnessed by an executable repro (the hunt's "semi-formal" tier).
- The audit looked at the places where DocumentsWriter and IndexWriter take one of their locks while holding another, and found three such edges that close a cycle: DocumentsWriter monitor -> purge lock (
lockAndAbortAll), purge lock -> IndexWriter monitor (the inline publish) and IndexWriter monitor -> DocumentsWriter monitor (getNextSequenceNumberundertryModifyDocumentandaddIndexes). - The finder then wrote a JUnit test that holds each thread at a point it reaches anyway and showed the cycle on main, 10.6.0 and 9.12.3; the fix sketch made it pass.
- An independent reproducer wrote its own test from the claim alone, with the single pause in a public merge-policy callback; the minimal repro on this page is a trimmed version of that idea.
No report of this cycle (searched 2026-11-02). Closest: open #6070 (2013) says IndexWriter has no clear lock-order rules but names no cycle and predates two of the three edges, so the overlap is only at class level; #6066 is an earlier, different deleteAll deadlock, fixed in 4.3.1.
Searches4 searches
- Class-level overlap: open #6070 (LUCENE-5006, mikemccand, 2013): "there are no clear rules about lock order to avoid deadlocks like #6066", and a request to "define the allowed concurrent behavior eg when an app calls deleteAll while other threads are indexing". It names no cycle; edges 1 and 3 came in 2017 and 2020.
- Nearest items, none of them this bug: #6066 (LUCENE-5002, a different
deleteAlldeadlock in DocumentsWriterFlushControl, fixed in 4.3.1; precedent); #10403, #10512, #9829 (testDeleteAllNoDeadLockfailures by assertion or exception, fixed, not hangs); #13571, #13127, #13169 (sequence-number races); #12654 (another IndexWriter deadlock, fixed by #12751); lucene-solr#1427 (the introducing change). - Not fixed upstream: upstream main
1c2072f0284(2026-11-02) has no commit toDocumentsWriter.javaorDocumentsWriterFlushQueue.javasince 2026-10-01. - Limits: keyword search, at most 50 results per query; private reports are invisible.
| Where | When | How | Found |
|---|---|---|---|
| Scripted prior-report search, 2 runs | 2026-11-02 | 51 and 43 queries: GitHub issues, PRs and commits; the JIRA archive; the dev@, java-user@, issues@ and commits@ lists; Elasticsearch, OpenSearch and Solr; the web | 193 and 1673 distinct hits; the only high-overlap hit is #6070 |
| GitHub, manual (apache/lucene, apache/lucene-solr, downstream projects) | 2026-11-02 | 34 searches, e.g. lockAndAbortAll, forcePurge, purgeLock, getNextSequenceNumber, tryDeleteDocument, "deleteAll deadlock"; re-checked | no report of this cycle |
| Open apache/lucene PRs | 2026-11-02 | the changed files of all 36 open PRs updated since 2026-10-29 | none changes lockAndAbortAll, forcePurge, publishFlushedSegment, tryModifyDocument or getNextSequenceNumber |
| JIRA archive and lists | 2026-11-02 | LUCENE and SOLR JQL; the comments of LUCENE-9304; builds@ for lockAndAbortAll | nothing about this cycle; 0 builds@ mails mention lockAndAbortAll |
Introduced in 8.6.0 by LUCENE-9304 (18af6325ed98, branch_8x 47ed21324c41, 2020-04-14), which made the sequence-number call take the DocumentsWriter lock inside the IndexWriter lock. 8.5.2 passes; the seven releases tested from 8.6.0 to 10.6.0, and main, deadlock. Not fixed upstream.
Versions tested5 rows
Introduced by 18af6325ed98b50c23a1f026ab9d05f17039c731 (master, first released in 9.0.0) and its branch_8x copy 47ed21324c41d4414cd259ceaba15f5d45863ac6 (first released in 8.6.0, 2020-07-13): "LUCENE-9304: Fix IW#getMaxCompletedSequenceNumber() (#1427)" by Simon Willnauer, author date 2020-04-14 (apache/lucene-solr#1427; LUCENE-9304 is #10344). It added the synchronized DocumentsWriter.getNextSequenceNumber and switched IndexWriter's sequence-number calls to it, among them those in tryModifyDocument, deleteAll and both addIndexes methods. In tryModifyDocument, addIndexes(Directory...) and the path of addIndexes(CodecReader...) that adds the merged segment, the call runs inside the IndexWriter monitor: edge 3.
The older edges: edge 1 from 58105a203a19 (LUCENE-7868, Mike McCandless, 2017-06-21, first in 7.0.0), which added ticketQueue.forcePurge to the synchronized lockAndAbortAll; edge 2 by design since 4.0.
The release runs used the independent reproducer's earlier, longer test of the same scenario (same threads, same pause point; assertions on, seed DEADBEEF); the minimal test itself ran on main and the 10.6.0, 10.5.2 and 9.12.3 jars. The releases not executed rest on code reading: all three edges are present in the source of every release from 8.6.0 to 10.6.0 (read: the history clone's tags 8.6.0 to 8.8.1 and 9.0.0 to 10.6.0; 8.8.2 to 8.11.4, which the clone lacks, from the releases/lucene-solr/<version> tags of apache/lucene-solr on GitHub, read by the fact check). Not new in 10.6.
| Version or commit | Verdict | Result |
|---|---|---|
main 5adff78bb168 | affected | the minimal test fails with gradle and a standalone runner, assertions on and off; passes with fix A |
| 10.6.0, 10.5.2, 9.12.3 | affected | the minimal test fails on the release jars, assertions on and off (3 deadlocked threads) |
| 8.6.0, 8.11.4, 9.0.0, 10.0.0 | affected | the earlier test of the same scenario deadlocks (3 JVM-detected threads) |
| 8.6.0, one call reverted | counterfactual | passes with only tryModifyDocument's call changed back to deleteQueue.getNextSequenceNumber() (shows causation; not a fix) |
| 8.0.0, 8.5.2 | not affected | pass: all three calls return (sequence numbers did not take the DocumentsWriter monitor) |
A finder, an independent reproducer, cross-model reviews (OpenAI GPT-5.6, executed), a devil's advocate, an adjudication, a credibility audit and a final repro check all reproduced the deadlock and called it real; the "works as designed" case was rated weak.
Checks and verdicts377 words
Each check ran independently; the table lists what each one executed.
| Check | Verdict | What it found |
|---|---|---|
| Finder (proof-obligation audit) | reproduced | Own test (pauses at InfoStream messages): deadlock on main (assertions on and off), 10.6.0, 9.12.3; passes with fix A; 41 tests of 3 upstream suites pass with it. |
| Refute review (OpenAI GPT-5.6, executed) | real (high) | Own test deadlocks with assertions on and off and a second seed; its sequential control passes; fix A passes. Rated "the schedule is forced through the internal InfoStream" weak: the pause takes no lock. |
| Independent reproducer | real | Own test, written from the claim only (one pause in keepFullyDeletedSegment): fails on main (5 of 5 iterations, assertions off), 10.6.0, 10.5.2, 9.12.3; passes with fix A. |
| Intended-lens review (OpenAI GPT-5.6, executed) | real (high); intended case weak | Own test deadlocks, control passes, fix A passes; re-run on 5adff78bb168 with assertions on and off. |
| Intended-behaviour devil's advocate | real (high) | A variant that pauses only the tryDeleteDocument thread deadlocks; a variant with addIndexes(CodecReader...) instead deadlocks (main, assertions off). No intended, documented, misuse or unreachable argument survived. |
| Adjudication | confirmed (high); priority Medium | Re-ran the reproducer's test on stock main, assertions off, fresh seed: 3 deadlocked threads. LFV-474 is a different bug at the same fix site. |
| Credibility audit and final repro check | confirmed | Re-runs on main fail with the same frames; the one-test repro fails with assertions on and off and in 5 of 5 separate JVMs, and passes with fix A. |
| This doc's brief | real; fix A tested | The minimal test fails on main with gradle and on the 10.6.0, 10.5.2 and 9.12.3 jars (assertions on and off); fix A passes it and 1,397 tests of the index package. A later probe shows which lock each later writer call waits for. |
| Fact checks of the brief and this doc | real; wording corrected | An independent check re-ran the minimal test (main with assertions on and off and with the plain Lucene104 codec; the 10.6.0 jars; an older main) and fix A (20 of 20 iterations); a reviewer from another model family (OpenAI GPT-5.6) read the recorded evidence. Both checked the file:line, quote, commit and count claims; their wording corrections are applied here. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 482 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,464 words
- Registry title
- IndexWriter.deleteAll can deadlock with a concurrent flush publish and a concurrent tryDeleteDocument or addIndexes, because DocumentsWriter.lockAndAbortAll waits for the flush-ticket purge lock while holding the DocumentsWriter monitor
- Area
- index / IndexWriter deleteAll (IndexWriter-DocumentsWriter lock order)
- Priority
- Medium, rank 109: When deleteAll() meets a concurrent flush publish and a tryDeleteDocument, tryUpdateDocValue or addIndexes call, the writer deadlocks for good with no exception and only a JVM restart recovers, on default settings in every release since 8.6.0. The three calls must meet in short windows and deleteAll during live indexing is uncommon, so the trigger is rare.
- Impact class
- hang
- Impact note
- Applications that call deleteAll() while other threads index and that also use tryDeleteDocument() or tryUpdateDocValue() (Expert) or the non-expert addIndexes() are hit when the three calls meet in their short windows: the three threads hang for good with no exception (-ea and -da alike), and the writer is then stuck, since the held IndexWriter and DocumentsWriter monitors and the purge lock block later commit(), close(), getReader() and flush publishes until the JVM restarts [read]; default codec and config suffice (the repro sets maxBufferedDocs=2 only to make the flush deterministic), no scale is needed, and every release from 8.6.0 through 10.6.0 and main are affected.
- Summary
- Three threads share one IndexWriter: one adds documents and its addDocument() flushes and publishes the new segment, one calls tryDeleteDocument() on an NRT reader (or addIndexes()), and one calls deleteAll(). You expect all three calls to return, because IndexWriter is documented as completely thread safe for concurrent calls of any of its methods. Instead they can deadlock for good: deleteAll() holds the DocumentsWriter monitor (in lockAndAbortAll) and waits for the flush-ticket purge lock, the publishing thread holds that purge lock and waits for the IndexWriter monitor (publishFlushedSegment), and tryDeleteDocument() (or addIndexes()) holds the IndexWriter monitor and waits for the DocumentsWriter monitor to get a sequence number (getNextSequenceNumber).
- Introduced
- 18af6325ed98b50c23a1f026ab9d05f17039c731 (master, first released in 9.0.0) and its branch_8x copy 47ed21324c41d4414cd259ceaba15f5d45863ac6 (first released in 8.6.0), both 'LUCENE-9304: Fix IW#getMaxCompletedSequenceNumber() (#1427)' (s1monw; apache/lucene-solr#1427; JIRA LUCENE-9304 = apache/lucene#10344): to fix a sequence-number visibility race it made DocumentsWriter.getNextSequenceNumber synchronized and switched tryModifyDocument (behind tryDeleteDocument / tryUpdateDocValue) and both addIndexes methods to it, inside the IndexWriter monitor, which added the IndexWriter-monitor -> DocumentsWriter-monitor edge that closes the cycle [exec: 8.5.2 passes and 8.6.0 deadlocks with the same test; on 8.6.0, reverting only the tryModifyDocument call makes it pass, which shows causation but is no fix, since it reopens the race that commit closed]. The other two edges are older [read]: DocumentsWriter monitor -> purge lock from 58105a203a19 (LUCENE-7868, 2017-06-21, first in 7.0.0), which added ticketQueue.forcePurge to the synchronized lockAndAbortAll; purge lock -> IndexWriter monitor by design since 4.0 (innerPurge publishes while holding the purge lock). (2020-04-14 (author date of 18af6325ed9 and 47ed21324c4))
- First affected version
- 8.6.0 [exec: the independent reproduction's one-test JUnit, an earlier and longer test of the same scenario as the minimal repro, deadlocks on 8.6.0 (-ea); 8.5.2 and 8.0.0 pass it; 8.5.2 shipped after the commit but from branch_8_5, which does not contain it]
- Affected versions
- Every release from 8.6.0 through 10.6.0 (released 2026-10-29, 4ae946e5a9b) and main 5adff78bb168 (12.0.0-SNAPSHOT). Executed and deadlocked with the independent reproduction's one-test JUnit, an earlier and longer test of the same scenario as the minimal repro (for 8.x and 9.0.0 the same file with the pre-9.1 LuceneTestCase import): 8.6.0, 8.11.4, 9.0.0, 9.12.3, 10.0.0, 10.6.0 and main (-ea, history runs), 10.5.2 (-ea, the independent reproduction), and main with -da (the independent reproduction, the adjudication and a separate re-run) [exec]; the minimal repro fails on main and on the 10.6.0, 10.5.2 and 9.12.3 release jars (-ea and -da) [exec]; the finder's test also deadlocks on main (-ea, -da), 10.6.0 and 9.12.3 [exec]; all three edges are present at every release tag from 8.6.0 to 8.8.1 and from 9.0.0 to 10.6.0 in the history clone [read]. Not affected: 8.0.0 and 8.5.2 [exec: pass]; 7.0.0 to 8.5.2 take sequence numbers without the DocumentsWriter monitor [read]. Not new in 10.6. Not fixed upstream: upstream main 1c2072f0284 (2026-11-02) has no commit to DocumentsWriter.java or DocumentsWriterFlushQueue.java since 2026-10-01 [read].
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal: proof-obligation audit
- Upstream and prior reports
- no report of this cycle found (2026-11-02, on top of the finder's, the independent reproduction's, the intended-behaviour check's and the adjudication's searches): two scripted prior-report searches (51 queries: 16 GitHub issue and PR, 3 commit, 2 downstream, 20 mailing-list over dev@, java-user@, issues@ and commits@, 4 JIRA-archive and 6 web, 193 distinct hits, the only highly relevant one #6070; and the finder's 43 queries, 1673 hits, none highly relevant), 30 manual GitHub searches (apache/lucene and downstream Elasticsearch, OpenSearch and Solr) plus 4 on apache/lucene-solr, the file lists of all 36 open apache/lucene PRs updated since 2026-10-29 (none touches DocumentsWriter.java or DocumentsWriterFlushQueue.java; #16571, #16847, #16948 and #16810 touch IndexWriter.java or DocumentsWriterFlushControl.java without changing deleteAll, lockAndAbortAll, forcePurge, publishFlushedSegment(s), tryModifyDocument or getNextSequenceNumber), JIRA JQL, list archives incl. builds@ (testDeleteAllNoDeadLock CI mails) and web searches. Related, class level only: open #6070 (LUCENE-5006, mikemccand, 2013) says 'there are no clear rules about lock order to avoid deadlocks like #6066' and asks upstream to 'define the allowed concurrent behavior eg when an app calls deleteAll while other threads are indexing'; it names no cycle and predates two of the three edges (7.0.0 and 8.6.0). Related, different: #6066 (LUCENE-5002, a deleteAll deadlock in DocumentsWriterFlushControl, fixed in 4.3.1), #10403, #10512 and #9829 (testDeleteAllNoDeadLock failures with assertion or exception modes, fixed, not hangs), #13571 and #13169 (sequence-number races), #12654 (another IndexWriter deadlock, tragic-exception handling, fixed by #12751), lucene-solr#1427 (the introducing change; its review never mentions lock order). Not fixed upstream: upstream main 1c2072f0284 (2026-11-02) has no commit to DocumentsWriter.java or DocumentsWriterFlushQueue.java since 2026-10-01, and the only IndexWriter.java commit since 2026-10-25 by author date is 5adff78bb16 (#16963, unrelated). Re-check (2026-11-02): GitHub search lockAndAbortAll 1 hit (#13127, unrelated), 'deleteAll deadlock' 3 (#6070, #6066, #11328 suggester thread safety), purgeLock 1 (#12801, Monitor), forcePurge 7 (#6070 and older items), 'tryDeleteDocument deadlock' 0, 'getNextSequenceNumber deadlock' 1 (#13571), deadlock items created since 2026-10-31 0, deleteAll items updated since 2026-10-31 0, deleteAll 0; none of the 32 apache/lucene PRs updated since 2026-10-31 is about deleteAll or IndexWriter locking.
- Lucene commit tested
- main
5adff78bb168(12.0.0-SNAPSHOT, 2026-11-02) with gradle and with a standalone JUnit runner, assertions on and off; the 10.6.0, 10.5.2 and 9.12.3 release jars, assertions on and off; an earlier test of the same scenario on 8.0.0 and 8.5.2 (pass) and 8.6.0, 8.11.4, 9.0.0, 9.12.3, 10.0.0 and 10.6.0 (deadlock). OpenJDK 25.0.3, Linux aarch64.
- Finding record
lucene/core/src/java/org/apache/lucene/index/DocumentsWriter.java:266-276 (at 5adff78bb168) (synchronized lockAndAbortAll calls ticketQueue.forcePurge first: DocumentsWriter monitor -> purge lock; fix site)lucene/core/src/java/org/apache/lucene/index/DocumentsWriterFlushQueue.java:36, (at 5adff78bb168) 82-133 (purgeLock; innerPurge runs the consumer while holding it, comment at :94 'if we block on publish -> lock IW'; forcePurge lock() at :118; tryPurge tryLock() at :128)lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:6265-6292 (at 5adff78bb168) (publishFlushedSegments: the purge consumer calls publishFlushedSegment) and :2953 (private synchronized void publishFlushedSegment): purge lock -> IndexWriter monitorlucene/core/src/java/org/apache/lucene/index/IndexWriter.java:1688 (at 5adff78bb168) (public synchronized long tryDeleteDocument), :1791 (private synchronized long tryModifyDocument), :1824 ('return docWriter.getNextSequenceNumber();'): IndexWriter monitor -> DocumentsWriter monitorlucene/core/src/java/org/apache/lucene/index/DocumentsWriter.java:583-586 (at 5adff78bb168) (synchronized long getNextSequenceNumber, 'this must be synced otherwise the delete queue might change concurrently')lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:3194-3201 (at 5adff78bb168) (addIndexes(Directory...): synchronized (this) -> docWriter.getNextSequenceNumber()) and :3350-3369 (addIndexes(CodecReader...): the same edge) [read; the CodecReader path executed by the intended-behaviour check]lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2728-2748 (at 5adff78bb168) (deleteAll javadoc: only MergeAbortedException for concurrent forceMerge, addIndexes(CodecReader[]) and forceMergeDeletes) and :2750-2766 (deleteAll: fullFlushLock, then docWriter.lockAndAbortAll())lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:167-170 (at 5adff78bb168) (class javadoc: 'completely thread safe, meaning multiple threads can call any of its methods, concurrently'; the only deadlock warning is about applications that synchronize on the writer)lucene/core/src/test/org/apache/lucene/index/TestIndexWriterDelete.java:326 (at 5adff78bb168) (testDeleteAllNoDeadLock: deleteAll concurrent with indexing is supported)
- The finding in 30 seconds
lucene/core/src/java/org/apache/lucene/index/DocumentsWriter.java:188-195, (at 5adff78bb168) :266-276, :583-586lucene/core/src/java/org/apache/lucene/index/DocumentsWriterFlushQueue.java:36, (at 5adff78bb168) :82-135lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:448-449, (at 5adff78bb168) :1688-1697, :1721, :1791-1827, :2750-2767, :2953, :3194-3201, :3350-3369, :6265-6292, :6428-6431
- Who is affected
lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2728-2748 (at 5adff78bb168) (deleteAll javadoc)lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:1374, (at 5adff78bb168) :1886-1890, :2764, :4349, :4399-4412, :5840
- Why it looks unintended
lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:167-170, (at 5adff78bb168) :2743-2745, :2931-2932lucene/core/src/java/org/apache/lucene/index/DocumentsWriterFlushQueue.java:93-98 (at 5adff78bb168)lucene/core/src/test/org/apache/lucene/index/TestIndexWriterDelete.java:326 (at 5adff78bb168)
- Suggested fix
lucene/core/src/java/org/apache/lucene/index/DocumentsWriter.java:266-276, (at 5adff78bb168) :535; DocumentsWriterFlushQueue.java:38; IndexWriter.java:2765lucene/core/src/java/org/apache/lucene/index/DocumentsWriter.java:153, (at 5adff78bb168) :583, :588 (other synchronized users of the DocumentsWriter monitor)
- TestDeleteAllDeadlock.java (one test):
repro/LFV-1137/TestDeleteAllDeadlock.java(86 lines)
Drawn from the code at 5adff78bb168 and the executed minimal test.
assets/LFV-1137/bug.png(482 KB)
