Since 9.4.0, addIndexes(CodecReader...) blocks indefinitely when the writer uses NoMergeScheduler

Up to 9.3.0, addIndexes(CodecReader...) merged the readers itself, in the calling thread, and its javadoc said it did not use the MergeScheduler. Since 9.4.0 (LUCENE-10216) it asks the merge policy for a merge of the readers, hands that merge to the merge scheduler and waits for it with an unbounded CompletableFuture.get(). NoMergeScheduler.merge does nothing, so the merge never starts and the call blocks until another thread interrupts it (ThreadInterruptedException) or aborts the writer's merges, for example with rollback(), close() or deleteAll() (MergeAbortedException). NoMergePolicy keeps that merge on purpose so that addIndexes keeps working with merges disabled, so the pair the NoMergeScheduler javadoc recommends hangs too. Every release from 9.4.0 to 10.6.0 and main is affected; nothing is corrupted and the readers are simply not added. The claim is narrow: the call should return or throw, not block.

• The hand-off and the wait (IndexWriter.java:3306-3313, all from LUCENE-10216):

MergePolicy.MergeSpecification spec = mergePolicy.findMerges(readers);
boolean mergeSuccess = false;
if (spec != null && spec.merges.size() > 0) {
  try {
    spec.merges.forEach(addIndexesMergeSource::registerMerge);
    mergeScheduler.merge(addIndexesMergeSource, MergeTrigger.ADD_INDEXES);
    spec.await();

• registerMerge only estimates the merge's size and queues it (IndexWriter.java:3408-3417); only a scheduler that calls getNextMerge() and merge(...) on that source runs it.
• No time limit: MergeSpecification.await() is "Waits, until interrupted, for all merges to complete", a plain CompletableFuture.get() (MergePolicy.java:533-544). A bounded await(long, TimeUnit) exists (:546-557) but is not used here.
• The scheduler does nothing: public void merge(MergeSource mergeSource, MergeTrigger trigger) {} (NoMergeScheduler.java:45), empty since the class was added in 2010. The MergeScheduler contract is "Run the merges provided by MergeSource#getNextMerge()" (MergeScheduler.java:43-49).
• The policy still asks for the merge: NoMergePolicy.findMerges(CodecReader...) returns the default single merge, "to enable addIndexes for consumers like IndexRearranger" (NoMergePolicy.java:42-50). The default TieredMergePolicy does not override that method and returns the same single merge.
• Only an abort or an interrupt releases it: everything that calls abortMerges() does, for example rollback() and close() (through rollbackInternal(), IndexWriter.java:2621) and deleteAll() (:2771; its javadoc warns that concurrent addIndexes(CodecReader[]) callers may receive MergeAbortedException, :2743-2745). abortMerges() aborts the queued addIndexes merges (:2821-2837, :3443-3454) and addIndexes then throws MergeAbortedException("merge was aborted.") (:3376-3378). An interrupt gives ThreadInterruptedException (MergePolicy.java:539-540) but leaves the unstarted merge queued until something aborts merges. The parked thread holds no writer lock: the call's synchronized block ends before the hand-off (IndexWriter.java:3298-3304).
• Not a race: it waits on a merge that nothing executes. Deterministic, seed-independent, with assertions on and off.

addIndexes(Directory...) with the same configuration returns and adds the documents (3 -> 8 docs in a verifier's control). The merge-running schedulers that were tested work: SerialMergeScheduler and the default ConcurrentMergeScheduler, each with NoMergePolicy.

• Who. Code that configures NoMergeScheduler (IndexWriterConfig.setMergeScheduler is documented as "Expert", IndexWriterConfig.java:271-279) and adds readers with addIndexes(CodecReader...). It hangs with NoMergePolicy and with the default TieredMergePolicy.
• Effect. The first call blocks, deterministically, with any number of documents (the repro adds 1), whenever the merge policy returns a non-empty reader-merge spec: NoMergePolicy and the default TieredMergePolicy both do; a custom policy returning null or an empty spec makes the call return early (IndexWriter.java:3325-3338). The thread parks without CPU use; a single-threaded caller never gets control back. The readers are not added and no merge output is written; nothing is corrupted. The flush at the start of the call (IndexWriter.java:3286) still writes the target's own buffered changes, if it has any.
• In-tree reach. The benchmark module accepts merge.scheduler=NoMergeScheduler (CreateIndexTask.java:110-113) and AddIndexes(false) calls addIndexes(CodecReader...) (AddIndexesTask.java:72, conf/addIndexes.alg:41): a verifier ran that combination, which hangs on main and returns on 9.3.0. PKIndexSplitter takes caller-supplied configs and calls addIndexes(CodecReader...), so a caller passing NoMergeScheduler would hit it (code reading). IndexRearranger builds its own configs with the default scheduler and is not affected.
• Not hit. addIndexes(Directory...); the merge-running schedulers that were tested, SerialMergeScheduler and ConcurrentMergeScheduler, each with NoMergePolicy.
• Likelihood: rare. The pairing is unusual; the recorded searches found no report since 9.4.0 (2022).

The class sits in org.apache.lucene.index so it can live next to TestAddIndexes upstream; it uses no package-private access. Smallest input: one empty document, one segment, ByteBuffersDirectory; the source writer uses a default IndexWriterConfig, the target writer sets only NoMergeScheduler and NoMergePolicy. The extra thread is only a watchdog, because a hang cannot be bounded on the calling thread. After waiting up to 10 s the test records whether the call is still running, calls rollback(), which releases a blocked call (so no thread leaks), waits for the thread, and fails if the call had been blocked. Any exception from the call counts as returning, so a fail-fast fix passes.

Regression test: this same class (or its one method added to TestAddIndexes); it passes with the recommended fix, well under a second. It accepts any exception, so with the fail-fast fix an expectThrows(IllegalStateException.class, ...) on the calling thread pins the fix more tightly.

cp $REPRO/TestAddIndexesNoMergeScheduler.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestAddIndexesNoMergeScheduler -Ptests.seed=DEADBEEF
# takes about 10 s to fail; assertions off fails the same way: -Ptests.asserts=false (the recorded runs added --offline)

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF (stack trimmed)
TestAddIndexesNoMergeScheduler > testAddIndexesCodecReaderReturns FAILED
    java.lang.AssertionError: addIndexes must return or throw, but was still blocked after 10 s
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:4385ADD63B8A706A]:0)
        at org.apache.lucene.index.TestAddIndexesNoMergeScheduler.testAddIndexesCodecReaderReturns(TestAddIndexesNoMergeScheduler.java:53)

# thread dump while blocked (main 645d4fe14a20, seed 1234ABCD, same failure)
"Thread-0" #41 [16675] prio=5 os_prio=0 cpu=3.02ms elapsed=4.17s ... waiting on condition
   java.lang.Thread.State: WAITING (parking)
	- parking to wait for  <0x00000000f5d23bb8> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.CompletableFuture.get(java.base@25.0.3/CompletableFuture.java:2093)
	at org.apache.lucene.index.MergePolicy$MergeSpecification.await(MergePolicy.java:537)
	at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3313)

# standalone runner, seed DEADBEEF: main 645d4fe14a20 fails with -ea and -da (about 11.4 s);
# releases 10.6.0 and 9.4.0 fail; 9.3.0 passes (whole run 1.3 s)

• Kept working on purpose: LUCENE-10216 added the NoMergePolicy.findMerges(CodecReader...) override that keeps the merge "to enable addIndexes for consumers like IndexRearranger" (NoMergePolicy.java:43-46); the PR author wrote on #633 that "There are consumers of NoMergePolicy that expect addIndexes(CodecReaders...) to work". NoMergeScheduler's javadoc: "it is recommended to use both if you want to disable merges from ever happening" (NoMergeScheduler.java:26-30).
• No blocking in the contract: the addIndexes(CodecReader...) javadoc (IndexWriter.java:3254-3274) describes merging through MergePolicy#findMerges(CodecReader...) and lists only CorruptIndexException, IOException and IllegalArgumentException.
• Where Lucene handles it explicitly, it throws: NoMergeScheduler.getIntraMergeExecutor throws UnsupportedOperationException("NoMergeScheduler does not support merges") (NoMergeScheduler.java:57-60). The upstream test scheduler that skips addIndexes merges, TestAddIndexes.PartialMergeScheduler, closes each skipped merge (TestAddIndexes.java:759-790), and its test expects addIndexes to throw (:807-808). These are single cases, not a general contract, and not every case is handled: in a sibling report, close(), forceMerge(1) and forceMergeDeletes() wait forever with NoMergeScheduler, known since #5316 (2014) and still unfixed.
• A regression: up to 9.3.0 the method merged inline and its javadoc said "this method does not call or make use of the MergeScheduler" (9.3.0 IndexWriter.java:3124-3125). LUCENE-10216's issue, PR and reviews never discuss a scheduler that runs no merges.
• Tests: no upstream test combines NoMergeScheduler with addIndexes.

The strongest case for "intended" (a read-only cross-model review rated it strong; the second cross-model review and the adjudication, moderate): the configuration contradicts itself. NoMergeScheduler "never executes any merges ... regardless of the MergePolicy used"; since 9.4.0 the CHANGES entry for LUCENE-10216 says the MergeScheduler triggers the merges "required by addIndexes(CodecReader[])", and MergePolicy.findMerges(CodecReader...) calls that merge required (MergePolicy.java:619). A user who disables every merge and asks for a merge-based operation gets no readers added.

Why it fails: that justifies at most not adding the readers, not a silent wait with no time limit that a single-threaded caller cannot leave. Where Lucene handles a scheduler that will not run a merge explicitly, it throws (getIntraMergeExecutor, PartialMergeScheduler), Lucene deliberately kept this method working for the "no merges" policy, and nothing in the javadoc of the recommended pair hints that it breaks this method. Hence the narrow claim: return or throw. Note that the 9.4.0 hand-off itself was announced in CHANGES; only its consequence for NoMergeScheduler was not.

One fix was validated through gradle in a scratch worktree of 645d4fe14a20; a second (run the merge inline) passed the hunt's own check; a javadoc note is a complement. A generic "merges still queued" check was considered and rejected.

Check the scheduler right after the merge policy returns a non-empty spec and before registerMerge, so nothing is queued and no merge output is written (the call's initial flush still runs first, as today).

if (spec != null && spec.merges.size() > 0) {
  if (mergeScheduler == NoMergeScheduler.INSTANCE) {
    // the readers are added through merges, which NoMergeScheduler never runs: fail fast
    // instead of waiting forever in spec.await()
    throw new IllegalStateException(
        "addIndexes(CodecReader...) adds the readers through merges, but the configured"
            + " NoMergeScheduler never runs merges");
  }
  try {
    spec.merges.forEach(addIndexesMergeSource::registerMerge);

Trade-offs

• Only calls that hang today change; a null or empty spec still returns early (IndexWriter.java:3325-3338). NoMergeScheduler is a final singleton with a private constructor, so the identity check is exact.
• The readers are still not added: users who want "no merges, but addIndexes works" switch to a merge-running scheduler with NoMergePolicy (works today) or to addIndexes(Directory...); the message could say so.
• A custom scheduler that silently drops merges still hangs; that scheduler breaks the MergeScheduler contract ("Run the merges provided by...").
• IllegalStateException vs UnsupportedOperationException (as getIntraMergeExecutor throws) is the maintainers' choice.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31): stock, the minimal test fails. With the fix: the minimal test passes (well under a second) with assertions on and off; TestAddIndexes 39/39, TestNoMergeScheduler 3/3 and TestNoMergePolicy 8/8 pass: 51 tests, 0 failures. Formatting was not checked.
• The standalone runner gave the same pass on 645d4fe14a20 with assertions on and off, and the hunt's check on a9779ef3c193 (the call throws within the bound, nothing added).

In NoMergeScheduler.merge, when the trigger is MergeTrigger.ADD_INDEXES, loop getNextMerge() / mergeSource.merge(merge); getIntraMergeExecutor returns Runnable::run instead of throwing.

Trade-offs

Restores the observed add-and-return outcome of 9.3.0 (the readers are added, through today's scheduler path rather than 9.3.0's inline merge), matching the intent of the NoMergePolicy override. But it breaks the class's documented promise ("never executes any merges") and needs a javadoc change, and TestNoMergeScheduler calls merge(null, random trigger), which would throw NullPointerException when the random trigger is ADD_INDEXES, so it needs a null guard.

What the checks validated

The hunt's check on a9779ef3c193 passed: the call returned with no exception and added the document. Not run through gradle or the module's tests.

A note on addIndexes(CodecReader...) and on NoMergeScheduler.

Trade-offs

Alone it leaves the hang in place; useful together with either fix.

Generic, so it would also catch custom schedulers.

Trade-offs

Wrong for legal schedulers: ConcurrentMergeScheduler.merge stops pulling merges when maybeStall returns false (ConcurrentMergeScheduler.java:564-568; subclasses may override maybeStall, :599-609) and picks queued merges up later when a merge thread finishes (:678-686). Not run.

Method: semi-formal (an assumption of a TLA+ liveness proof checked against the code, then a public-API JUnit).

• Model. ForceMergeWait.tla models callers waiting for forced merges and merge threads running them. Its property F3, "no caller waits forever" (<>(cpc[c] \in {"ok", "threw"}) for every caller), holds under a fairness assumption on merge threads (TLC: 73,524 and 916,793 states for one and two callers). The model does not cover addIndexes.
• Signal. Checking that fairness assumption against the shipped schedulers: NoMergeScheduler.merge takes no merge at all. Two code paths wait on such merges: close() / forceMerge (a separate, already known report) and, since 9.4.0, addIndexes(CodecReader...) (this bug).
• Confirmation. A JUnit on the stock main jar blocked with assertions on and off; a release bisection showed 9.3.0 returns and 9.4.0 hangs. An independent reproducer, a devil's-advocate verifier, read-only cross-model reviews (OpenAI GPT-5.6), an adjudication and a credibility audit checked it; a follow-up ran the version history, a prior-report search and a security screen.