Merge-on-refresh never releases the files of a merged segment whose documents were all deleted during the merge

Merge-on-refresh (LUCENE-8962, on by default since 9.3.0) makes a near-real-time refresh wait up to 500 ms for merges of small segments. When such a merge completes, the refresh opens a reader on the merged segment and takes a reference on its files (deleter.incRef), at the start of commitMerge and before the deletes made during the merge are carried over. If those deletes removed every document, commitMerge drops the merged segment, and its file deletion skips the referenced files. finishGetReaderMerge then closes the unused reader and clears mergedReaders without decRef, so the cleanup that pairs close with decRef finds nothing. The files keep their reference for the rest of the writer's life: deleteUnusedFiles(), commit(), forceMerge, close() and rollback() leave them, and only the next IndexWriter open deletes them. Searches, commits and CheckIndex stay correct; it is a disk-space and file-count leak, one small merged segment per hit. A one-test repro fails on main 645d4fe14a20, 10.6.0, 10.5.2, 9.12.3 and 9.3.0; deleting the two offending lines fixes it.

// getReader with merge-on-refresh (:619-648): the callback for each completed refresh merge
sci -> {
  SegmentReader apply = readerFactory.apply(sci);
  mergedReaders.put(sci.info.name, apply);   // :643
  deleter.incRef(sci.files());               // :647, the reference
}

// onGetReaderMergeResources (:649-666), run at the end of getReader (:723) or on error (:716):
// for each reader still in mergedReaders: deleter.decRef(its files), then close it (:658-664)

// finishGetReaderMerge (:727-753)
StandardDirectoryReader reader = maybeReopenMergedNRTReader(...);
IOUtils.close(mergedReaders.values());       // :749, closes the leftover readers ...
mergedReaders.clear();                       // :750, ... and forgets them: no decRef
return reader;

1. DirectoryReader.open(w) runs getReader. With maxFullFlushMergeWaitMillis > 0 (500 by default) it prepares point-in-time merges for MergeTrigger.GET_READER (:619-648), then finishGetReaderMerge runs the merge and waits for it (:738-739).
2. The merge writes _2. commitMerge starts with merge.onMergeComplete() (:4894-4896). The wrapper's onMergeComplete (:4031-4045) has three conditions (:4034-4037); its only check against a dropped segment is info.info.maxDoc() > 0 /* never do this if the segment is dropped / empty */ (:4036-4037), and the other two (collection not stopped, merge not aborted) hold here. _2 has maxDoc 2, so the callback runs: reader opened, put in mergedReaders, incRef of _2's files.
3. commitMerge carries over the deletes made during the merge (:4936-4937), finds the segment allDeleted (:4946-4949) and drops it with readerPool.drop and deleteNewFiles (:5002-5007). deleteNewFiles deletes only files without a reference (IndexFileDeleter.java:633-636, FileDeleter.java:154-156, 173-191), so _2's files survive.
4. The wrapper's mergeFinished applies a merge to the point-in-time SegmentInfos only when segmentDropped == false (:3978-3980). maybeReopenMergedNRTReader removes a reader from mergedReaders, and later decRefs its files, only when its segment is in those SegmentInfos (:772, :780, :787-789). _2 is not, so its reader stays in the map.
5. finishGetReaderMerge closes it and clears the map (:749-750). onGetReaderMergeResources.close() at the end of getReader (:723) finds nothing to release.
6. _2's files keep a reference count of 1 for the rest of the writer's life. IndexFileDeleter.refresh(), which rollback() uses, also skips referenced files (IndexFileDeleter.java:455-486, :467). The next IndexWriter starts with fresh counts and deletes them as unreferenced.

The point-in-time reader itself is correct: it keeps _0 and _1 as they were when the refresh started, because deletes made during the merge belong to the next refresh (:3940-3948, javadoc of preparePointInTimeMerge).

• Who. Applications on 9.3.0 or later with the default IndexWriterConfig that refresh NRT readers (DirectoryReader.openIfChanged, SearcherManager.maybeRefresh) while their own commits or flushes apply updates or deletes. 8.7.0 to 9.2.0: only with setMaxFullFlushMergeWaitMillis > 0 and a merge policy that returns full-flush merges.
• Settings. The default TieredMergePolicy proposes full-flush merges of segments each below its 16 MB floor (MergePolicy.java:689-714, TieredMergePolicy.java:79, 187-189); the default wait is 500 ms (IndexWriterConfig.java:108).
• Trigger. Every document of a refresh merge's input segments must be deleted, and the deletes applied, while that merge runs. That favours workloads that keep rewriting the same documents.
• Effect. One small merged segment's files per hit, kept through deleteUnusedFiles(), commit(), forceMerge, close() and rollback(); the next IndexWriter open deletes them. In one probe a leaked segment was 3 files, 2,378 bytes. The deleted documents' bytes stay in those files until then (raw file access only). No premature delete, no wrong result, no corruption, CheckIndex clean, no open file handle after close().
• How often. Timing-dependent and not measured on real workloads. A tight default-config loop (one thread updating 2 ids, one committing continuously, refreshes for 4 s) leaked 675 merged segments; a milder loop (a commit every 2 updates, 5 s) leaked 1 to 13 merged segments per run on 2026-10-26 and none in 5 runs on 2026-10-31.
• Likelihood: plausible. No fault is needed, but the window is narrow.
• Workaround until fixed (by reading, not run on main): setMaxFullFlushMergeWaitMillis(0) turns off full-flush merges ("Set to 0 to disable merging on full flush"; this also turns off merge-on-commit), or reopen the IndexWriter from time to time. 9.2.0, where the wait is 0 by default, passes the minimal test.

One test class, one method, single thread: under SerialMergeScheduler the refresh merge runs in the refreshing thread, so the order is fixed. Public API only, ByteBuffersDirectory (no mock filesystem, no lock file), about 1 s. Merge-on-refresh comes from the default config: the test sets only the merge policy and the scheduler, and the 500 ms wait is the default. It fails with any seed (10 of 10 random-seed iterations).

Do not use deleteDocuments(MatchAllDocsQuery) in the hook: IndexWriter turns it into deleteAll(), which waits for the running merge, so the test deadlocks itself (test-side only).

Formatting: google-java-format (./gradlew tidy) wraps the comment on line 69 into two lines; after that the test passes core's format, ECJ and forbidden-apis checks and still fails on stock main.

Regression test: this same test, as its own class or as a method of TestIndexWriterMergePolicy, which holds the upstream merge-on-getReader tests.

cp $REPRO/TestGetReaderMergeDroppedSegmentLeak.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestGetReaderMergeDroppedSegmentLeak -Ptests.seed=DEADBEEF -Ptests.asserts=true
# same failure with -Ptests.asserts=false and with other seeds

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true (identical with false)
TestGetReaderMergeDroppedSegmentLeak > testFullyDeletedGetReaderMergeLeavesNoFiles FAILED
    java.lang.AssertionError: expected:<[]> but was:<[_2.fdm, _2.fdt, _2.fdx, _2.fnm, _2.si, _2_Lucene104_0.doc, _2_Lucene104_0.psm, _2_Lucene104_0.tim, _2_Lucene104_0.tip, _2_Lucene104_0.tmd]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:CB15C06EE4D5EEF7]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.index.TestGetReaderMergeDroppedSegmentLeak.testFullyDeletedGetReaderMergeLeavesNoFiles(TestGetReaderMergeDroppedSegmentLeak.java:70)
  NOTE: test environment is: codec=Asserting(Lucene104): {}, ...
1 test completed, 1 failed

# releases, seed DEADBEEF: the same assertion (file names follow the randomized test codec)
10.6.0 -ea/-da, 10.5.2 -ea/-da: [_2.fdm, _2.fdt, _2.fdx, _2.fnm, _2.si, _2_Lucene104_0.doc, ..., _2_Lucene104_0.tmd]
9.12.3 -ea: [_2.fdm, _2.fdt, _2.fdx, _2.fnm, _2.si, _2_Lucene912_0.doc, _2_Lucene912_0.psm, _2_Lucene912_0.tim, _2_Lucene912_0.tip, _2_Lucene912_0.tmd]
9.3.0  -ea: [_2.fdm, _2.fdt, _2.fdx, _2.fnm, _2.si, _2_Lucene90_0.doc, _2_Lucene90_0.tim, _2_Lucene90_0.tip, _2_Lucene90_0.tmd]
9.2.0  -ea: PASS (no merge on refresh by default)

# with either fix: 1 test completed, 0 failed (assertions on and off)

# default config, no hook, 4 s timing loop (not the repro), assertions off
stock:          refreshes=2422 leakedFiles=2025 leakedSegments=675
fix v2:         refreshes=2269 leakedFiles=0 leakedSegments=0
recommended:    refreshes=2316 leakedFiles=0 leakedSegments=0

• The code's own cleanup releases the reference. onGetReaderMergeResources "closes the merged readers and decrements the files references ... for readers that haven't been removed from the mergedReaders and release elsewhere" and does decRef then close for each (IndexWriter.java:651-664); maybeReopenMergedNRTReader decRefs the readers it consumes (:787-789). Only finishGetReaderMerge closes without decRef, and by clearing the map it hides the leftovers from the cleanup.
• The guard shows the intent. The callback is skipped for segments that are "dropped / empty" (:4036-4037); it misses only a segment that becomes 100% deleted later in the same commitMerge.
• Public contracts. rollback(): "This removes any temporary files that had been created, after which the state of the index will be the same as it was when commit() was last called" (:2567-2570). deleteUnusedFiles(): "Expert: remove any index files that are no longer used" (:6171). Internal: "When all the commits referencing a certain file have been deleted, the refcount for that file becomes zero, and the file is deleted" (IndexFileDeleter.java:47-50).
• Lucene's tests treat leftover files as failures: TestIndexWriter.assertNoUnreferencedFiles fails when a fresh writer would delete anything.
• History. The feature author fixed the mirror-image problem on merge-on-commit before release (a premature delete of a "fully deleted and dropped" segment, #10005 comment of 2020-06-22). None of the 81 comments and reviews on lucene-solr#1623 discusses a dropped merged segment or leftover files.

The strongest case for "works as designed": the next IndexWriter open cleans up, and Lucene tolerates unreferenced files elsewhere (MockDirectoryWrapper's check is "off by default; see LUCENE-5574"); LUCENE-8324 called a dropped segment's leftover files "not a massive issue"; the refcount promise is in an internal javadoc; deleting from wrapForMerge is an unusual hook; and four years passed without a CI report, although the test framework turns merge-on-refresh on.

Why it fails: LUCENE-5574 covers a reader that outlives its writer, while the repro closes every reader first, and next-open cleanup is the crash-recovery path. LUCENE-8324 was fixed anyway, "for correctness", which is the right framing here: real and minor. The public rollback() and deleteUnusedFiles() javadocs promise the deletion. The hook only makes the timing deterministic: a hook-free default-config loop leaks too, and the fix removes that leak. CI is silent because the unreferenced-files check is off by default and the merge-on-getReader tests do not delete all documents of a refresh merge while it runs. Verdict: not intended, a minor defect.

Two fixes were built and validated through gradle in a scratch worktree of 645d4fe14a20; either is short. The recommended one reuses the cleanup the code already has; the alternative, the hunt's fix v2, releases the leftovers at once. The hunt's model checking covers releasing the leftovers in general, and its randomized runs used the hunt's first version (v1) of this fix, whose normal path is the same as v2's.

IndexWriter.finishGetReaderMerge, lines 749-750: delete the close-and-clear (-2/+3 lines with a comment).

// IndexWriter.finishGetReaderMerge, after maybeReopenMergedNRTReader(...):
-      IOUtils.close(mergedReaders.values());
-      mergedReaders.clear();
+      // Readers still in mergedReaders belong to merges that were not applied to the point-in-time
+      // SegmentInfos, e.g. a merged segment dropped as fully deleted. Leave them in the map:
+      // onGetReaderMergeResources closes them and releases their file references.
       return reader;

Trade-offs

• mergedReaders is local to the getReader call, and nothing adds to it once stopCollectingMergedReaders is set (:741). onGetReaderMergeResources.close() runs at the end of getReader (:723) or in the error path (:716) and does decRef plus close for each leftover (:658-664), exactly once. This matches that cleanup's own comment.
• The leftover readers stay open a little longer, through the maybeMerge(FULL_FLUSH) at :704-707 (with SerialMergeScheduler that may run merges synchronously); the files are still released before getReader returns.
• Not covered by the hunt's TLA+ model or its randomized multi-thread runs (those runs used the hunt's first version of the alternative).

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle --offline, OpenJDK 25.0.3, seed DEADBEEF: the repro passes; all core org.apache.lucene.index.* tests (215 suites, 1,396 tests, 56 skipped, 0 failures) pass with assertions on and off; core checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass.
• The 4 s default-config loop leaks 0 files (stock: 2,025).

Copy the leftovers out of mergedReaders, clear the map, then decRef each leftover's files and close it; if that throws, close the reopened reader too (+17/-1 lines).

List<SegmentReader> leftovers = new ArrayList<>(mergedReaders.values());
mergedReaders.clear();
try {
  IOUtils.close(
      leftovers.stream()
          .flatMap(
              sr ->
                  Stream.<Closeable>of(() -> deleter.decRef(sr.getSegmentInfo().files()), sr))
          .toList());
} catch (Throwable t) {
  IOUtils.closeWhileSuppressingExceptions(t, reader);
  throw t;
}

Trade-offs

Releases the files right away, but duplicates the cleanup logic. Its first version decRef'd before clearing the map, so an exception while closing could let onGetReaderMergeResources decRef the same files again (found by the cross-model review); v2 clears first.

What the checks validated

• Here: the repro, TestIndexWriterMergePolicy, TestIndexWriterReader and TestIndexFileDeleter (57 tests) and all core org.apache.lucene.index.* tests (215 suites, 1,396 tests) pass with assertions on and off; core's format, ECJ and forbidden-apis checks pass after ./gradlew tidy, which joins the Stream.of call onto one line; the 4 s loop leaks 0.
• In the hunt: with the fix modelled (the leftovers' references released; the model does not encode v2's code) the TLA+ model holds every invariant (up to 71,975,928 states with 5 segment names); the hunt's first version (v1) of this fix, whose normal path is the same as v2's, passed 600 randomized multi-thread runs with 0 leaked and 0 missing files (stock: 21 of 150 leaked); with v2 itself, 8 upstream test classes pass and the hook-free loops leak 0.

As merge-on-commit does, in mergeFinished when segmentDropped == false.

Trade-offs

The refresh must open the merged reader before the merge's own deletes are carried over, to keep point-in-time semantics (:3940-3948), so the reader and its files must be held from onMergeComplete on (by reading).

onGetReaderMergeResources (and v2) decRef first, so a released file is deleted while its reader is still open: 6 delete-while-open attempts in a probe with the fix.

Trade-offs

Harmless on POSIX; where the OS refuses to delete an open file, FSDirectory keeps a pending delete and retries later. Not run.

setMaxFullFlushMergeWaitMillis(0) turns off full-flush merges (on refresh and on commit), or reopen the IndexWriter from time to time to reclaim the files.

Trade-offs

By reading, not run on main; loses the merge-on-refresh benefit.

Method: semi-formal (a code-reading hypothesis during modelling, witnessed by a model checker, then executed).

• A hunt unit modelled the IndexFileDeleter reference-count protocol in TLA+ (IFDRefcount.tla): flushes, commits, merges, merge-on-commit, merge-on-refresh, NRT readers and rollback.
• While writing the merge-on-refresh part, the finder saw that the callback's incRef has no decRef on the dropped-segment path. TLC confirmed it: invariant RcExact is violated in an 8-state trace and NoLeakAtQuiescence in a 9-state trace (flush, flush, refresh with a refresh merge, the merge writes, all its documents are deleted, commitMerge drops it, the refresh finishes, rollback: the merged segment's files keep a reference). With one thread (no deletes during the merge) the model is clean (1,572,782 states); with the fix every invariant holds (up to 71,975,928 states). The stock model never deletes a referenced file (up to 5,705,664 states).
• The finder then replayed the trace as a public-API JUnit test on the real jars and measured a hook-free default-config loop. Independent reproducers, cross-model reviews (OpenAI GPT-5.6, reading only), an adjudication, an audit, the version-history runs, a prior-report search, a repro-minimization step and this doc followed.