forceMergeDeletes returns a MergeObserver that waits on merges IndexWriter refused, so it never completes

IndexWriter.forceMergeDeletes(boolean) asks the merge policy for forced-deletes merges, calls registerMerge on each and ignores its boolean result. registerMerge returns false, logging only "reject merge ... is already marked for merge", when one of the merge's segments is already part of a registered merge; such a merge is never queued, never run and never closed. The method then returns new MergePolicy.MergeObserver(spec) over the whole, unfiltered proposal. A merge's completion future is completed only when the merge is closed, so the observer never completes: numCompletedMerges() stays below numMerges(), await() blocks until interrupted (even after IndexWriter.close()), awaitAsync() never completes, and await(timeout) returns false, which its javadoc reads as a timeout or an error although nothing failed. The blocking forceMergeDeletes() does not hang itself (its wait loop keeps waiting only while a proposed merge is pending or running) but returns without waiting for the refused merge, with an incomplete observer, against its javadoc "merges will already be complete when it returns". It needs LogDocMergePolicy, LogByteSizeMergePolicy or a custom policy that proposes segments already being merged, plus any overlapping merge (an ordinary background merge is enough); the default TieredMergePolicy skips such segments. The observer is @lucene.experimental and new in 10.4.0 (#15378); every release since has the bug. The index is not damaged. Observing only the merges that registered fixes it.

1. Registration ignores the answer. Under the writer lock, IndexWriter.java:2358-2365: spec = mergePolicy.findForcedDeletesMerges(segmentInfos, cachingMergeContext); (:2359), then for (int i = 0; i < numMerges; i++) registerMerge(spec.merges.get(i)); (:2363).
2. registerMerge can refuse. Its javadoc: "Else (the merge conflicts) false is returned" (IndexWriter.java:5153-5157). For a segment in mergingSegments it logs "reject merge ... is already marked for merge" and returns false (:5171-5183), before pendingMerges.add(merge) (:5206). The other false branch, "does not exist in live infos" (:5184-5195), cannot fire for a policy that proposes segments of the infos it was given, because the proposal is computed under the same lock; a custom policy that proposed other segments would end up the same way.
3. Nobody completes a refused merge. OneMerge.mergeCompleted (MergePolicy.java:192) is completed only in OneMerge.close (:285-299, the one complete call at :289). A merge that never entered pendingMerges is never picked up by a merge thread, so nothing closes it.
4. The observer wraps the unfiltered proposal. return new MergePolicy.MergeObserver(spec); (IndexWriter.java:2405). numMerges() is the proposal's size (MergePolicy.java:885-887), numCompletedMerges() counts completed futures (:895-906), await() is CompletableFuture.allOf(...).get(), "Waits, until interrupted" (:528-544, :915-917; an interrupt ends it with ThreadInterruptedException), await(timeout) returns false on timeout (:547-557, :928-930), and awaitAsync() returns the allOf future (:938-942).
5. The blocking call does not wait for it. The wait loop (IndexWriter.java:2369-2400) throws on a tragic error or a merge exception and keeps waiting only while a proposed merge is in pendingMerges or runningMerges (:2386-2389). The refused merge is in neither, so the call never waits for it: in the repro, where it is the only proposed merge, the call returns at once; when other proposed merges register, it waits for those and still returns the refused one in the observer.
6. Which policies trigger it. LogMergePolicy.findForcedDeletesMerges (LogMergePolicy.java:409-465) groups adjacent segments with deletes and never consults getMergingSegments() (its findMerges does, :497). TieredMergePolicy.findForcedDeletesMerges skips merging segments (TieredMergePolicy.java:938, :947, :960). The abstract findForcedDeletesMerges javadoc (MergePolicy.java:654-662) sets no precondition about merging segments.
7. Why it was harmless before 10.4.0. The loop that ignores the result dates from 2008 (LUCENE-325). While forceMergeDeletes returned void, nothing held a reference to the refused merge.

Stock LogDocMergePolicy with mergeFactor 3 and three 10-document segments, two with one deleted document each: the third commit starts an ordinary natural merge of all three. forceMergeDeletes(false) then proposes the two segments with deletes; an InfoStream sees IndexWriter log "reject merge", and after waitForMerges() and close() the observer still reports 0 of 1. The latch the tests put in ConcurrentMergeScheduler#doMerge holds the natural merge open so the timing is deterministic; the code path only needs that merge to be still running when forceMergeDeletes is called (every recorded run of this variant used the latch).

  merge thread                                   caller
  merge M1 over segment _0 registered, running
                                                 forceMergeDeletes()
                                                   policy proposes M2 over _0
                                                   registerMerge(M2) -> false  "reject merge" (ignored)
                                                   wait loop: M2 not pending, not running -> return
                                                   return new MergeObserver([M2])
  M1 finishes
                                                 M2: never pending, never run, never closed
                                                 numCompletedMerges() = 0 of 1, forever

• Who is hit. Applications that (1) use LogDocMergePolicy, LogByteSizeMergePolicy or a custom policy whose findForcedDeletesMerges proposes a segment that is already being merged, (2) call forceMergeDeletes while an overlapping merge runs (a natural merge, a background forceMerge, or an earlier forceMergeDeletes(false)), and (3) then wait on or poll the returned observer. The default TieredMergePolicy and TemporalMergePolicy skip such segments; TieredMergePolicy controls pass.
• What breaks. await() without a timeout parks the caller until it is interrupted, also after the writer is closed; awaitAsync() callbacks never run; await(timeout) always returns false, which reads as a timeout or an error; progress reporting with numCompletedMerges() / numMerges(), the javadoc's own example, never reaches 1; the blocking call returns an observer that contradicts its javadoc.
• What does not break. An executed probe parked an unbounded await() and, from another thread, added and deleted documents, opened an NRT reader, committed, ran forceMergeDeletes again and closed the writer, all normally. The parked thread holds no writer lock and pins no reader or file; deleted documents stayed hidden and CheckIndex was clean. Thread.interrupt() releases it.
• Scale. None needed: one segment of 2 documents is enough. The window is as long as any overlapping merge runs.
• Reach. The API is @lucene.experimental and new in 10.4.0. Elasticsearch, OpenSearch and Solr ignore the returned observer (read-only code search, default branches); no downstream report was found. Likelihood: rare.
• Not part of this bug. The blocking call returning while the competing merge still runs is older behaviour (seen on 9.12.3 and 10.3.2 too); deletes applied after the competing merge opened its readers then stay in the index, hidden, until a later forceMergeDeletes, with TieredMergePolicy too.

Stock LogDocMergePolicy, ByteBuffersDirectory, two documents with one deleted, no sleeps, no reflection; the codec is not pinned (no codec code is involved). Concurrency is a precondition, not noise: a merge must be in flight on the segment, so one merge thread is held by one latch.

The test covers the blocking call. The non-blocking call has the same cause: with forceMergeDeletes(false), an independent test saw numMerges=1 numCompletedMerges=0 await(5s)=false awaitAsync().isDone()=false after IndexWriter.close(), and an unbounded await() had not returned after 5 s.

Regression test: this same test. It passes with the fix and can go into TestIndexWriterMerging next to testForceMergeDeletesBlockingWithObserver. With the fix the assertion holds as 0 == 0 (the refused merge is no longer observed), so on its own it does not show the conflict happened; if the maintainers want that, the test can also check for IndexWriter's "reject merge" InfoStream line, as the natural-merge variant does.

cp $REPRO/TestForceMergeDeletesRejectedMerge.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestForceMergeDeletesRejectedMerge -Ptests.seed=DEADBEEF
# the same result with -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF (assertions on; the same with -Ptests.asserts=false)
TestForceMergeDeletesRejectedMerge > testObserverCompleteWhenBlockingCallReturns FAILED
    java.lang.AssertionError: completed merges expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:FC4A538E24B1C2C4]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.index.TestForceMergeDeletesRejectedMerge.testObserverCompleteWhenBlockingCallReturns(TestForceMergeDeletesRejectedMerge.java:63)

# the same failure from the 10.6.0 and 10.4.0 release jars
# with the fix: 1 test, 0 failures (assertions on and off)

• The contract. forceMergeDeletes() @return (IndexWriter.java:2421-2422): "a MergeObserver to monitor merge progress. Since this method blocks until completion, merges will already be complete when it returns." forceMergeDeletes(boolean) @return (:2343): "a MergeObserver to monitor merge progress and wait for completion". MergeObserver.await() @return (MergePolicy.java:912-913): "true if all merges completed successfully or no merges were needed, false on error". numMerges() (:883): "number of merges, or 0 if no merges were scheduled".
• The tests. TestIndexWriterMerging.testForceMergeDeletesBlockingWithObserver uses newLogMergePolicy() and asserts "await should return true immediately" after forceMergeDeletes(true), on an idle writer. No upstream observer test calls forceMergeDeletes while another merge holds the same segments.
• History. #15378 (by Salvatore Campagna, merged by Vigya Sharma) added the observer for #14515. Its 23 review comments discuss the API's shape, numCompletedMerges, toString and test timeouts, and never mention refused merges. In that review Vigya Sharma asked for "an assert here that tests that numMerges == completedMerges" in the blocking-call test, and Salvatore Campagna replied "Since forceMergeDeletes(true) blocks until completion, all merges should be done when it returns": the assertion this bug breaks.

The strongest case: IndexWriter.getMergingSegments() is "to be used by a MergePolicy to avoid selecting merges for segments already being merged" (IndexWriter.java:2520-2522), a MergeSpecification is only a proposal, and MergeObserver is @lucene.experimental; so a policy that proposes merging segments misuses the API, and the observer may track proposals.

Why it fails: findForcedDeletesMerges states no such precondition (MergePolicy.java:654-662), while findFullFlushMerges states one explicitly and says what happens (:677-681); the policy that does it is Lucene's own public LogMergePolicy, which the upstream observer tests use; @lucene.experimental allows the API to change, not to contradict what it documents today; and a monitoring handle that can never complete for a legal configuration has no use. Every verifier and the refute review rejected the case; three of them call it weak. At most it moves the fix into LogMergePolicy, which leaves custom policies exposed. Verdict: not intended.

One fix was built and validated in a scratch worktree of 645d4fe14a20. The alternatives come from the adjudication and the devil's-advocate verifier; option C is the separate LogMergePolicy bug.

One file, +8/-1, in IndexWriter.forceMergeDeletes(boolean). Every merge left in the proposal is registered (registerMerge returned true), so a merge thread runs it and closing it completes its future; the blocking wait loop and the observer then look at the same set.

      if (newMergesFound) {
        // only merges that were actually registered will run and complete: observe only those
        final MergePolicy.MergeSpecification registered = new MergePolicy.MergeSpecification();
        final int numMerges = spec.merges.size();
        for (int i = 0; i < numMerges; i++) {
          if (registerMerge(spec.merges.get(i))) {
            registered.add(spec.merges.get(i));
          }
        }
        spec = registered;
      }

Trade-offs

When every proposed merge is refused, the observer has numMerges() 0 and await() returns true at once, which the class javadoc reads as "no merges are needed": the caller cannot tell "nothing to do" from "a conflicting merge already holds these segments". The blocking call still returns while that other merge runs (older behaviour, not this bug).

What the checks validated

• Stock: the minimal test fails with assertions on and off (gradle, seed DEADBEEF, scratch worktree of 645d4fe14a20).
• With the fix: the minimal test passes with assertions on and off.
• Closest suites with the fix (every core test class that calls forceMergeDeletes, plus the merge-policy and scheduler suites): TestIndexWriterMerging (incl. the observer tests), TestIndexWriter, TestIndexWriterReader, TestConsistentFieldNumbers, TestSoftDeletesRetentionMergePolicy, TestTieredMergePolicy, TestLogMergePolicy, TestIndexWriterForceMerge, TestConcurrentMergeScheduler: 9 classes, 230 tests, 0 failures, 5 skipped.
• :lucene:core:checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass.
• Earlier: every hunt repro passes with the fix (the finder's, the independent reproducer's 5 tests, the cross-model reviewers' tests, the natural-merge and custom-policy variants).
• Not run: nightly tests, benchmarks.

Complete or close each refused OneMerge, so the observer counts it as done and numMerges() keeps the policy's proposal. The future is private and completed only by OneMerge.close, which also runs the merge's mergeFinished hook and reader clean-up, so closing a merge that never started needs a design decision. Not tested.

Trade-offs

complete(true) reports a merge that never ran as completed successfully. complete(false) records the failure only in OneMerge.hasCompletedSuccessfully(): await() still returns true, because allOf(...).get() returns normally for a normally completed future (the same gap as the separate bug where await() returns true after a failed merge). Only completeExceptionally(...) makes await() return false, which the javadoc reads as an error.

The one-line LogMergePolicy.findForcedDeletesMerges fix of the sibling bug (LogMergePolicy proposes merging segments for forced deletes). With it, the stock-policy repros can no longer set up the conflict.

Trade-offs

Fixes only the two stock Log policies: a custom policy that proposes every segment with deletes still fails with it and passes only with the IndexWriter fix (executed). Worth doing as its own issue.

Say in the forceMergeDeletes and MergeObserver javadoc that merges refused because their segments are already merging are included and never complete.

Trade-offs

Cheapest, but await() without a timeout can still hang for a legal configuration.

Method: non-formal (a stress test outside the study's baseline gave the first signal). The hunt unit on the merge-scheduler lifecycle wrote a TLA+ model (CmsSched.tla, property TerminalAllClosed: every registered merge is eventually closed) and a seeded stress harness (concurrent indexing, forced merges and forceMergeDeletes(false), with every returned observer awaited after the writer closed). The harness failed in its first 3-iteration smoke run on stock main ("MergeObserver never completed after the writer closed", seed 1234ABCD). The model has no refused merges, so it could not show this.

The finder then found the root cause by reading the code, wrote a deterministic JUnit, and had an independent reviewer from another model family (OpenAI GPT-5.6) try to refute it before filing the candidate.