IndexWriter waits for and leaks a full-flush merge it refused to register, instead of throwing the documented IllegalStateException

With merge on commit or merge on refresh (on by default since 9.3.0), IndexWriter asks the merge policy's findFullFlushMerges for merges and waits up to maxFullFlushMergeWaitMillis (500 ms) for them. When a proposed merge includes a segment that is already in a registered merge, the javadoc says commit() and prepareCommit() throw IllegalStateException. They do not: updatePendingMerges ignores registerMerge's refusal, so the refused merge stays in the spec that preparePointInTimeMerge opens readers for and that commit and getReader wait on. A refused merge never runs, so each such commit or NRT reopen waits the whole timeout, the merge's readers and file references outlive IndexWriter.close() (199 files open after 40 commits with a naive policy, in a probe tuned for frequent natural merges), and with a long wait DirectoryReader.open(writer) keeps waiting after another thread's rollback(). Only a custom policy that returns overlapping merges (hit on every commit) or ignores MergeContext#getMergingSegments() (hit when a natural merge holds one of its segments) triggers it; Lucene's own policies do not. It has been in every release since 8.6.0.

1. Refusal ignored. IndexWriter.updatePendingMerges gets the spec from findFullFlushMerges for COMMIT and GET_READER (IndexWriter.java:2494-2497), calls registerMerge(spec.merges.get(i)) for every merge without looking at the boolean result, and returns the whole spec (:2511-2517).
2. What a refusal is. registerMerge returns false when one of the merge's segments is already in mergingSegments ("reject merge ... is already marked for merge", :5171-5182) or is not in the live infos (:5184-5194). Only an accepted merge goes into pendingMerges (:5206) and gets registerDone = true (:5247). For natural merges this is harmless: nobody uses the returned spec.
3. Readers opened anyway. preparePointInTimeMerge (:3950-4110) runs initMergeReaders on every merge of the returned spec, through getPooledInstance(sci, true) and getReaderForMerge(..., mr -> deleter.incRef(...)) (:4078-4089). A refused merge never runs and is never aborted, so these readers and IndexFileDeleter references are never released.
4. Awaited anyway. Commit: prepareCommitInternal calls preparePointInTimeMerge(..., COMMIT, ...) when the wait is above 0 (:3858-3865), then mergeScheduler.merge(mergeSource, COMMIT) and pointInTimeMerges.await(maxCommitMergeWaitMillis, ...) (:3886-3906), inside synchronized (commitLock) (:3775). getReader: preparePointInTimeMerge(..., GET_READER, ...) (:634-648) and the same await (:738-739). MergeSpecification.await waits for every merge's completion future (MergePolicy.java:546-557); the refused merge's future never completes, so the wait always runs to its timeout. That wait is bounded and interruptible. (With SerialMergeScheduler, mergeScheduler.merge first runs the accepted merges in the calling thread without the timeout, IndexWriterConfig.java:495-497; the extra wait for the refused merge is still timed.)
5. Rollback cannot help. abortMerges completes only pending and running merges (:2821-2844), so rollback() cannot release a waiting getReader. rollback() first claims the close (shouldClose, :1374-1394) and then takes commitLock (:2589): a waiting getReader holds no lock it needs, but a stalled commit does, so rollback waits behind it.
6. The authors assumed it cannot happen. The error path of preparePointInTimeMerge, which runs only if opening the merge readers fails, asserts "merge should be pending but isn't" for every merge of the spec (:4099).

The repro's input with the info stream on: "add merge to pendingMerges: _0 _1", "reject merge _1 _2: segment _1 is already marked for merge", "now run merges during commit" listing both merges. {_0, _1} merges into _3 in 25 ms, but "done waiting for merges during commit" comes 504 ms after the wait starts and "commit: took 527.9 msec". The writer then deletes _0's files but not _1's, which the refused merge still references. With MockDirectoryWrapper, close() leaves _1.cfs and _2.cfs open. A natural merge already running when the commit proposes an overlapping merge gives the same: no exception after a 2,045 ms wait (2,000 ms configured). A naive "merge all segments under 1 MB on commit" policy that ignores getMergingSegments(), over a TieredMergePolicy with setSegmentsPerTier(2.0), with setMaxBufferedDocs(2), the default scheduler and wait, 400 documents and a commit every 10, took at least 480 ms (the probe's threshold for about the full wait) on 40 of 40 commits (max 533 ms) and left 199 files open after close(); the same policy skipping getMergingSegments(): 0 of 40 slow commits (max 18 ms), clean close. The two non-default settings keep natural merges of the small segments running almost all the time: with both at their defaults the same probe gave 0 of 200 slow commits over 5 runs and a clean close every time. So a policy that only ignores getMergingSegments() is hit only when a natural merge holds one of its segments at commit time; one that returns overlapping merges is hit on every commit.

• Who. Only a custom findFullFlushMerges that proposes a segment already in a registered merge, or two overlapping merges in one spec. TieredMergePolicy and LogMergePolicy skip merging segments in findMerges (TieredMergePolicy.java:347-349, LogMergePolicy.java:604, :638), and the default findFullFlushMerges is a size-filtered subset of findMerges (MergePolicy.java:689-714). MockRandomMergePolicy and the sandbox MergeOnFlushMergePolicy filter too. A read-only check of the default branches of Elasticsearch, OpenSearch and Solr (2026-10-26, no revisions pinned) found no affected policy: OpenSearch's CriteriaBasedMergePolicy drops merging segments before it delegates, Solr's LatestVersionMergePolicy delegates on filtered infos, and in Elasticsearch only one test overrides findFullFlushMerges.
• Default vs opt-in. The policy is application code. A policy that returns overlapping merges needs nothing else since 9.3.0 (merge on full flush on by default, 500 ms). A policy that only ignores getMergingSegments() also needs a natural merge running over one of its segments at commit time. In 8.6.0-9.2.0 the path ran only with an explicit non-zero wait.
• How often. On every commit or NRT reopen for a policy that returns overlapping merges. For a policy that only ignores getMergingSegments(), whenever a natural merge holds one of the proposed segments: 40 of 40 commits in a probe tuned for frequent natural merges (setSegmentsPerTier(2.0), setMaxBufferedDocs(2)), 0 of 200 with default settings.
• Effect. Each affected commit or reopen takes the full timeout instead of milliseconds, holding commitLock meanwhile. Two related leaks accumulate with ongoing indexing (199 files open after 40 commits in the tuned probe): the refused merge's readers are never closed, so their file handles stay open even after close(); and its IndexFileDeleter references keep this writer from deleting the merged-away files (once the writer is closed, a later IndexWriter on the directory can delete them). With a long configured wait, DirectoryReader.open(writer) keeps waiting after another thread's rollback() (still waiting 20 s later). No wrong results or data loss: the merges that registered are applied. CheckIndex was not run after the leak.
• Likelihood: rare. It takes a policy that ignores documented guidance, and the public searches found no prior report.
• Workarounds. Skip MergeContext#getMergingSegments() in findFullFlushMerges and never return overlapping merges: the probe's control shows that is clean. Or set setMaxFullFlushMergeWaitMillis(0), which turns merge on commit and on getReader off altogether (IndexWriterConfig.java:493; commit and getReader run preparePointInTimeMerge only when the wait is above 0, IndexWriter.java:3858, :619).

The test sits in org.apache.lucene.index for locality only; every API it uses is public. ByteBuffersDirectory keeps the separate file leak from showing up as a suppressed failure at directory close, and setCommitOnClose(false) keeps close() from running the policy a second time (with the default, close() commits again and this policy, which assumes three segments, throws IndexOutOfBoundsException on the two that are left). It asserts only the documented exception; the stall (528 ms) is visible in the info stream, and the files left open after close() and the getReader wait after rollback() are shown by the other checks, not by extra tests.

Regression test: this same test, as its own class or as a method of TestIndexWriterMergePolicy. It passes core's checkGoogleJavaFormat, ecjLintTest and forbiddenApisTest as is.

TestRejectedFullFlushMerge > testCommitThrowsOnOverlappingFullFlushMerges FAILED
    junit.framework.AssertionFailedError: Expected exception IllegalStateException but no exception was thrown
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:342F1B8CD1AEAE62]:0)
        at app//org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1432)
        at app//org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
        at app//org.apache.lucene.index.TestRejectedFullFlushMerge.testCommitThrowsOnOverlappingFullFlushMerges(TestRejectedFullFlushMerge.java:48)
1 test completed, 1 failed

# info stream of the same test (abridged)
IW: add merge to pendingMerges: _0(11.0.0):c1 _1(11.0.0):c1 [total 1 pending]
IW: reject merge _1(11.0.0):c1 _2(11.0.0):c1: segment _1(11.0.0):c1 is already marked for merge
IW: now run merges during commit: MergeSpec:
  1: _0(11.0.0):c1 _1(11.0.0):c1
  2: _1(11.0.0):c1 _2(11.0.0):c1
IW: now apply merge during commit: _0(11.0.0):c1 _1(11.0.0):c1 into _3
IFD: now delete 3 files: [_0.cfe, _0.si, _0.cfs]
IW: done waiting for merges during commit
IW: commit: took 527.9 msec

# with the recommended fix: BUILD SUCCESSFUL (1 test, 0 failures), assertions on and off

• The contract. MergePolicy.java:677-681: "If a OneMerge in the returned MergeSpecification includes a segment already included in a registered merge, then IndexWriter#commit() or IndexWriter#prepareCommit() will throw a IllegalStateException. Use MergeContext#getMergingSegments() to determine which segments are currently registered to merge." In the minimal test the registered merge is the spec's own first merge, registered a moment earlier: IndexWriter registers the proposed merges in order (IndexWriter.java:2513-2514) and rejects the second through the same "already marked for merge" branch (:5171-5182) as any other conflict. #1155's check threw in exactly this case. For the narrower reading (a merge registered before the policy ran), the finder's running-merge test shows the same missing exception: no exception after a 2,045 ms wait.
• The design record. The first merge-on-commit implementation, lucene-solr #1155 (043c5dff6f4, 2020, reverted before any release), threw IllegalStateException("MergePolicy ... returned merging segments from findFullFlushMerges") when registerMerge returned false, and added that sentence. In review msfroh wrote "I suppose we could silently disregard the merge if we remember to count down the latch for the merge that fails to register. ... I lean slightly toward explosive failure", and mikemccand replied "OK I agree to explosive failure! Less trappy." The reverted #1552 was the first version without the check; the shipped #1617 re-landed the code without it. None of their comments or reviews discusses the change. Today's code does neither option: it neither throws nor disregards the merge.
• Other text. IndexWriterConfig.java:482-488: full-flush merges that time out "are not aborted, and will still run to completion ... like natural segment merges"; a refused merge never runs. IndexWriter.java:4099 asserts "merge should be pending but isn't" for every point-in-time merge. Lucene's tests check through MockDirectoryWrapper that closing a writer releases every file.
• getReader. No exception is promised there; that symptom rests on the leak and the waiter.

The strongest "not a bug" case is misuse: the javadoc tells policies to use MergeContext#getMergingSegments(), every shipped policy does, MockRandomMergePolicy says "Do not return any merges involving already-merging segments", the sandbox TestMergeOnFlushMergePolicy asserts "findFullFlushMerges must not return already merging segments", and MergePolicy is expert API.

Why it fails: the same javadoc defines the result of exactly this input, so the input is inside the contract, not undefined behaviour. And "it fails silently instead of loudly" would excuse at most the missing exception, not readers and files that outlive close() or a waiter that rollback() cannot release. Misuse argues for low priority, not for intent.

Two directions; maintainers may prefer either, and both change IndexWriter only. The recommended one was built and run through gradle in a scratch worktree of 645d4fe14a20; the alternative was checked with the patched class on the stock jars.

IndexWriter.java +36/-3. Right after updatePendingMerges(...) returns in preparePointInTimeMerge, before any merge reader is opened; plus a cleanup in prepareCommitInternal.

if (pointInTimeMerges != null
    && pointInTimeMerges.merges.stream().anyMatch(merge -> merge.registerDone == false)) {
  if (trigger == MergeTrigger.COMMIT) {
    for (MergePolicy.OneMerge merge : pointInTimeMerges.merges) {
      if (merge.registerDone) { // abandon the registered ones, as the error path below does
        pendingMerges.remove(merge);
        try { abortOneMerge(merge); } finally { mergeFinish(merge); }
      }
    }
    throw new IllegalStateException("MergePolicy " + config.getMergePolicy().getClass()
        + " returned merging segments from findFullFlushMerges");
  }
  pointInTimeMerges.merges.removeIf(merge -> merge.registerDone == false); // GET_READER
}

Trade-offs

• Matches the javadoc and the original #1155 behaviour and message. For getReader no exception is documented, so refused merges are dropped there.
• Needs the cleanup in prepareCommitInternal: wrap the preparePointInTimeMerge(..., COMMIT, ...) call and deleter.decRef(toCommit.files(false)) before rethrowing, because the commit's files were incRef'd just before (IndexWriter.java:3857). Without it, rollback() after the exception leaves 9 files (_0, _1, _2 .cfe/.cfs/.si); with it, none. That is the leak path of LFV-456 (a separate, known bug with more throw sites); if a general fix for it lands first, the cleanup is unnecessary.
• Exception safety (cross-model fact check, by reading): the loop that abandons the registered merges is a plain loop, while the existing error path uses IOUtils.applyToAll (IndexWriter.java:4093-4104), which goes on past an exception. Closing a merge runs the overridable OneMerge.mergeFinished; if that throws, the plain loop stops and later registered merges stay pending. A final patch should use the same applyToAll cleanup, and a test with a throwing mergeFinished would cover it. Not run.
• Behaviour change: an application running such a policy today gets an exception on each affected commit instead of a slow commit and a leak. That is what the javadoc promises and what the reviewers chose ("Less trappy"), but it is visible. After the exception (by reading): it is not tragic, so the writer stays usable once the policy is corrected; with the same policy every later commit throws again; and with the default commitOnClose(true), close() commits, throws again, rolls the writer back (IndexWriter.java:1328-1335) and rethrows.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle --no-daemon --offline, OpenJDK 25.0.3, seed DEADBEEF. Stock: the repro fails with assertions on and off.
• With the fix: the repro passes, assertions on and off; TestIndexWriter*, TestConcurrentMergeScheduler, TestTieredMergePolicy, TestLogMergePolicy and the repro: 25 suites, 393 tests, 0 failures, assertions on and off; all core org.apache.lucene.index.* tests: 215 suites, 1,396 tests, 0 failures (assertions on); sandbox TestMergeOnFlushMergePolicy: 6 tests, 0 failures; core's checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest pass. Other modules and the full core suite were not run.
• With the fixed class on the stock jars, the finder's 4 tests pass: both exception cases, no files open after close(), getReader returns after rollback(); the rollback probe leaves no files.

pointInTimeMerges.merges.removeIf(merge -> merge.registerDone == false) before the initMergeReaders loop, +6 lines; the findFullFlushMerges javadoc sentence then has to say the merge is skipped.

Trade-offs

Compatible and quiet: no stall, no leak, no new exception. But a policy bug stays silent (the merge just does not happen), the javadoc must change, and the repro here, which asserts the documented exception, fails with it by design; the regression test would have to assert the missing stall or leak instead.

What the checks validated

With the patched class on the stock jars (not through gradle): the naive-policy probe gives 0 of 40 slow commits (max 24 ms) and a clean close(); the finder's leak and getReader-rollback tests pass.

The same behaviour as the previous option, at the call site that ignores the result.

Trade-offs

Does not reach the sibling LFV-207, whose forceMergeDeletes has its own registration loop. Not run.

Make findFullFlushMerges skip MergeContext#getMergingSegments() and never return overlapping merges. Or set setMaxFullFlushMergeWaitMillis(0), which disables merge on commit and on getReader (IndexWriterConfig.java:493).

Trade-offs

The probe's control does the first: 0 of 40 slow commits and a clean close(). The second gives up merge on commit and on getReader altogether (by reading: commit and getReader call preparePointInTimeMerge only when the wait is above 0).

Method: non-formal (code reading while modelling, then dynamic checks).

• Signal. The hunt unit modelled IndexWriter's point-in-time merges (merge on commit and merge on refresh) in TLA+. Writing the model's file:line table, the finder saw that updatePendingMerges ignores registerMerge's result while preparePointInTimeMerge opens and awaits the whole spec. The model's merge action registers every proposed merge, so no model output was involved.
• Confirmation. A same-package JUnit on public and protected API (4 tests: two overlapping merges in one spec; a conflict with a running natural merge; files open after close(); getReader after rollback()) failed 4 of 4 on main with and without assertions and on 10.5.2, 9.12.3, 9.8.0, 9.0.0 and 8.11.4.
• Then a review and an adversarial review by a model from another family (OpenAI GPT-5.6, reading only), an independent reproducer with its own JUnit, an intended-behaviour devil's advocate with a hook-free naive-policy probe, an adjudication, a history and prior-report follow-up, registry re-runs, a credibility audit, a repro-minimization step and this doc followed.