IndexWriter opened from a reader's IndexCommit keeps the reader's segment files open when its constructor fails

IndexWriterConfig#setIndexCommit documents that a writer can be opened from an open reader by passing reader.getIndexCommit(). On that path the IndexWriter constructor first builds its ReaderPool, which pre-enrolls one SegmentReader per segment of the reader, each sharing the reader's segment core and taking a reference on it. Only then does it build the IndexFileDeleter, which reads every segments_N and calls IndexDeletionPolicy.onInit. If anything throws from then on, the constructor's catch (Throwable) closes the merge scheduler and the write lock but not the pool, and the caller never gets the writer. So after the caller closes the reader, each segment core still holds the pool's reference and its files stay open: with one one-document commit and a policy whose onInit throws, MockDirectoryWrapper reports {_0.cfs=1} still open, with assertions on and off, on main and on every release from 7.4.0 to 10.6.0. The same failing policy with a commit from listCommits, or with no commit, leaves nothing open. An I/O error while the deleter reads a commit leaks the same way with the default policy. The leak arrived as a side effect of LUCENE-8260, which moved the pre-enroll step ahead of the deleter. Results and index files are unaffected. Closing the pool in the catch, as #15675 did for the merge scheduler, fixes it.

• Segment core (SegmentCoreReaders): the per-segment open files and readers shared by every SegmentReader of that segment. Each sharing reader takes a reference (SegmentReader.java:151-152, core.incRef()); when the count reaches 0 the core closes its files, including the compound-file input (SegmentCoreReaders.java:167-179, opened at :83).
• Reader commit: DirectoryReader.getIndexCommit() on a standard reader returns a ReaderCommit that carries the reader (StandardDirectoryReader.java:571-574, :644-646); a commit from listCommits is a ReaderCommit built with a null reader (DirectoryReader.java:456, :481; field set at StandardDirectoryReader.java:595), so its getReader() returns null.
• Pre-enrollment: the ReaderPool constructor's loop that pools one SegmentReader per leaf of that reader, "so any in-memory NRT live docs are correctly carried over, and so NRT readers pulled from this IW share the same segment reader" (ReaderPool.java:90-115).

1. The constructor takes the write lock (IndexWriter.java:954), enters try (:956) and gets the commit's reader, reader = commit.getReader() (:982-990).
2. readerPool = new ReaderPool(..., reader, config) (:1148-1158) pre-enrolls one SegmentReader per segment, each adding a reference to the reader's core (ReaderPool.java:96-113).
3. Only then deleter = new IndexFileDeleter(...) (IndexWriter.java:1166-1182) reads every commit (IndexFileDeleter.java:161) and calls policy.onInit(commits) (:237).
4. If that throws, catch (Throwable t) (IndexWriter.java:1202-1209), "init: hit exception on init; releasing write lock", runs IOUtils.closeWhileSuppressingExceptions(t, config.getMergeScheduler(), writeLock) (:1206). The pool is not in that list, and it lives in a private final field (:403) of a writer the caller never receives.
5. r.close() removes the reader's own reference, but the pooled reader's reference stays: the core never reaches 0 and _0.cfs stays open.

The failed writer stays reachable from the config, which stores it through the package-private setIndexWriter in a private SetOnce field (IndexWriter.java:947, IndexWriterConfig.java:119, :126-132), so the exact wording is "cannot be released through any public API".

Other triggers of the same missing release (fault-injection matrix, assertions on and off): an NRT reader's commit with the throwing onInit; an I/O error while the deleter reads a commit, with the default policy; and two failures inside the pre-enroll loop, an I/O error while a later segment's pooled reader reads its field infos and LFV-1062's leaf-sorted unequal-size IllegalArgumentException, which leak the readers pooled so far.

• Who is hit. Writers opened with IndexWriterConfig#setIndexCommit(reader.getIndexCommit()), the Expert path LUCENE-6524 added in 5.3.0 ("IndexWriter can now be initialized from an already open near-real-time or non-NRT reader"), whose constructor fails after the pool is built. Lucene's own RandomIndexWriter takes this path at random in tests (RandomIndexWriter.java:96-104).
• Default vs opt-in. Opt-in: by default no commit is set ("The default is null which opens the latest commit point", IndexWriterConfig.java:234-241). A commit from listCommits, or none, is not affected.
• What triggers it. An application IndexDeletionPolicy whose onInit throws the IOException it declares (none of the stock policies throws there); an I/O error while IndexFileDeleter reads a segments_N, with the default policy; or a failure inside the pre-enroll loop (an I/O error on a later segment, or LFV-1062's leaf-sorted reader with segments of unequal size).
• How much. One segment-core reference per segment per failed attempt, growing with each attempt; a successful retry does not release the earlier one. Open file handles grow per attempt only when each attempt uses a fresh reader: three failed attempts with the same reader leave {_0.cfs=1}, three with a fresh reader each leave {_0.cfs=3}. The write lock and the merge scheduler are released, so a retry can open.
• At the OS level (one Linux probe, not part of the claim): after the failed open the caller dropped the reader, a normal writer deleted _0.cfs, and 5 System.gc() calls ran. With MMapDirectory, which FSDirectory.open returns on 64-bit JVMs (FSDirectory.java:155-161), the process still had 3 mappings of the deleted _0.cfs (control: 0); they belong to a shared arena that only an explicit close releases, so the file's disk space stays allocated as long as they exist (not measured beyond the probe). With NIOFSDirectory the one leaked file descriptor was reclaimed by garbage collection once the reader and the config were dropped.
• What does not happen. No wrong results, no index damage, no hang; with an injected I/O error while the deleter reads a commit, the committed file set is unchanged, with no unreferenced or missing files. The constructor fails loudly.

The smallest input: one segment, and a policy that always throws, so the failure comes right after the pool is built. The oracle is MockDirectoryWrapper's own open-file check on close(). The test sits in org.apache.lucene.index next to TestIndexWriterFromReader but uses no package-private API. The same test is the regression test. Assertions on and off, and other seeds (1065C0DE, C0FFEE12), give the same failure. All runs were on Linux aarch64 with OpenJDK 25.0.3. The missing release is plain Java control flow, so it does not depend on the platform; the OS-level effects above are Linux observations.

# gradle, stock 645d4fe14a20, seed DEADBEEF, assertions on
TestIndexWriterFromReaderFailedOpen > testFailedOpenFromReaderCommitReleasesSegmentReaders FAILED
    java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
        at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:884)
        at org.apache.lucene.index.TestIndexWriterFromReaderFailedOpen.testFailedOpenFromReaderCommitReleasesSegmentReaders(TestIndexWriterFromReaderFailedOpen.java:51)
        Caused by:
        java.lang.RuntimeException: unclosed IndexInput: _0.cfs
            at org.apache.lucene.tests.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:782)
            at org.apache.lucene.tests.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:843)
            at org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:87)
            at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:91)
            at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:83)

# assertions off (-Ptests.asserts=false), seeds DEADBEEF and 1065C0DE: the same failure
# 10.6.0, 10.5.2 and 9.12.3 release jars: the same failure

# with the fix (either option), assertions on and off
classes=1 tests=1 failures=0 errors=0 skipped=0

# closest suites + the repro, seed DEADBEEF: TestIndexWriter (119), TestIndexWriterExceptions (40),
# TestConcurrentMergeScheduler (21), TestIndexWriterFromReader (12), TestIndexFileDeleter (11), TestDeletionPolicy (8),
# TestReaderPool (6), TestIndexWriterOnDiskFull (4), TestIndexWriterExceptions2 (1)
stock:          classes=10 tests=223 failures=1 errors=0 skipped=5   (the one failure is the repro)
with fix A:     classes=10 tests=223 failures=0 errors=0 skipped=5
with fix A + B: classes=10 tests=223 failures=0 errors=0 skipped=5

• The catch is cleanup. Its log line reads "init: hit exception on init; releasing write lock" (IndexWriter.java:1204). LUCENE-2476 (#3550, 2010) added the write-lock release on any constructor exception; #15675 (GITHUB#14971, 2026) added the merge scheduler plus TestConcurrentMergeScheduler#testMergeSchedulerClosedOnIndexWriterInitFailure (:965-987), whose trigger is itself a user error (OpenMode.APPEND on an empty directory). The pool is the one resource holding open files that the catch still misses (the DocumentsWriter and a half-built IndexFileDeleter are not closed either, but hold no open files).
• The trigger is in contract. public abstract void onInit(List<? extends IndexCommit> commits) throws IOException (IndexDeletionPolicy.java:62), with no "must not throw" note (:46-61). The I/O-error trigger uses the default policy.
• Lucene's tests assume closing the reader is enough. RandomIndexWriter opens a writer from reader.getIndexCommit() at random and, if the constructor fails, closes only the reader (RandomIndexWriter.java:96-125). TestIndexWriterFromReader#testWithNoFirstCommit (:87) and #testOnClosedReader (:380) close their directory after a failed constructor; their failures come before the pool has enrolled any reader (testOnClosedReader fails at reader.leaves() inside the ReaderPool constructor, ReaderPool.java:94). No test asserts or tolerates the leak.
• History: a side effect of a refactor. Through 7.3.1 the constructor built the IndexFileDeleter first and pre-enrolled after it, so an onInit or deleter failure found an empty pool (7.3.1 passes the test). LUCENE-8260 (#9307, "Extract ReaderPool from IndexWriter", 2018) moved the loop into the ReaderPool constructor, ahead of the deleter, and kept the handler as it was; the commit message describes only the extraction.

The strongest case for "intended" (the devil's-advocate verifier and the intended-lens reviewer): setIndexCommit is "Expert" and getIndexCommit is "Expert" and @lucene.experimental (DirectoryReader.java:644-649); no javadoc promises that a failed constructor releases everything (its @throws covers I/O errors only, IndexWriter.java:941-943); and the main trigger is a user policy that throws, which could be called user error.

Why it fails: the catch exists for cleanup, and maintainers already fixed a leak in this exact catch whose trigger was a user error (#15675); onInit declares IOException; the I/O-error trigger needs no user code; Lucene's own test framework closes only the reader after a failed constructor on this path; and the leak came in by accident with a refactor. The argument explains the Low priority, not the leak. All reviewers rated it weak.

The hunt's two-hunk diff, unchanged, and its first hunk alone, each run through gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF) with the minimal repro and the closest suites; the fault-injection matrix shows which hunk fixes which trigger.

Hunk A, IndexWriter: keep the pool in a local, assign the field from it, and add it to the catch's close list (+4/-2). ReaderPool is Closeable and close() drops every pooled reader (ReaderPool.java:45, :233-237, :314-341); the helper ignores null (IOUtils.java:97-106). The snippet above is condensed.

Hunk B, ReaderPool constructor (+30/-18, mostly re-indentation): wrap the pre-enroll loop so a failure inside it releases what was pooled so far, and decRef the reader whose ReadersAndUpdates could not be built. Without it, a failure inside the loop leaves pool null in the writer's catch.

      try {
        for (int i = 0; i < leaves.size(); i++) {
          // ... SegmentReader newReader = new SegmentReader(...) ...
          ReadersAndUpdates rld;
          try {
            rld = new ReadersAndUpdates(/* ... */, newReader, /* ... */, config);
          } catch (Throwable t) {
            IOUtils.closeWhileSuppressingExceptions(t, newReader::decRef);
            throw t;
          }
          readerMap.put(newReader.getOriginalSegmentInfo(), rld);
        }
      } catch (Throwable t) {
        // release the readers pre-enrolled so far: the caller never gets this pool
        IOUtils.closeWhileSuppressingExceptions(t, this::dropAll);
        throw t;
      }

// IndexWriter(Directory, IndexWriterConfig)
+    ReaderPool pool = null; // closed if this constructor fails after creating it
     try {
-      readerPool = new ReaderPool(/* ... */, reader, config);
+      pool = new ReaderPool(/* ... */, reader, config);
+      readerPool = pool;
       // ... IndexFileDeleter, onInit ...
     } catch (Throwable t) {
-      IOUtils.closeWhileSuppressingExceptions(t, config.getMergeScheduler(), writeLock);
+      IOUtils.closeWhileSuppressingExceptions(t, pool, config.getMergeScheduler(), writeLock);

Trade-offs

• No change on success: one local variable and one try.
• Coverage. Hunk A alone fixes the minimal repro, the onInit trigger (NRT and non-NRT) and the deleter-read trigger; hunk B alone fixes only the two in-loop triggers; both together pass all 8 tests of the fault-injection matrix.
• Test. The regression test (the minimal test) exercises hunk A only; hunk B is backed by the fault-injection matrix, which is evidence, not a proposed test.
• Overlap. LFV-1062's name-pairing fix edits the same loop: the diffs overlap in text but are logically independent (a combined patch was not executed).
• One edge left in hunk B (the cross-model fact check): if readerMap.put itself throws, in practice only a VM error such as OutOfMemoryError, after the current reader's ReadersAndUpdates was built, that reader is not yet in the map and dropAll() misses it; an "enrolled" flag around the put would close it. Not material in practice.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, 2026-11-02: the minimal test fails on stock with -Ptests.asserts=true and =false (and a second seed) and passes with the fix both ways; the closest suites (9 classes, 222 tests, 5 skipped) pass on stock and with the fix.
• google-java-format 1.37.0, run standalone: no line of the fix changes. spotlessCheck was not run, and the gradle runs had no eclint configured.
• Earlier: the hunt's shadow check (assertions on and off), the matrix 8 of 8 with both hunks, 78 upstream tests on stock and with the fix, and the refute reviewer's run (19 upstream tests including the #15675 test).

The IndexWriter hunk alone (+4/-2), the snippet of the recommended option.

// IndexWriter(Directory, IndexWriterConfig)
+    ReaderPool pool = null; // closed if this constructor fails after creating it
     try {
-      readerPool = new ReaderPool(/* ... */, reader, config);
+      pool = new ReaderPool(/* ... */, reader, config);
+      readerPool = pool;
       // ... IndexFileDeleter, onInit ...
     } catch (Throwable t) {
-      IOUtils.closeWhileSuppressingExceptions(t, config.getMergeScheduler(), writeLock);
+      IOUtils.closeWhileSuppressingExceptions(t, pool, config.getMergeScheduler(), writeLock);

Trade-offs

The smallest change, and fully covered by the regression test. Failures inside the pre-enroll loop (an I/O error while a later segment is pre-enrolled, LFV-1062's IllegalArgumentException) still leak the readers pooled so far.

What the checks validated

Gradle, same worktree and seed: the minimal test passes with assertions on and off; the closest suites (222 tests) pass. In the matrix it fixes testA, testA2 and testB and leaves testC and testD failing.

Restore the pre-7.4.0 order, so an onInit or deleter failure finds an empty pool.

Trade-offs

Changes initialization order in a constructor that LUCENE-8260 deliberately restructured; whether the deleter or later steps rely on the pool existing was not checked; failures after the pool is built and inside the loop would still leak. Hunk A is smaller and covers strictly more.

Method: non-formal. A code-reading probe read the constructor's catch and saw that the pool built before the deleter is never closed (2026-10-30). Its executed main() probe leaked with a throwing onInit and in an I/O-error mode, and was clean in a no-failure control; a separate re-run confirmed it.

• A hunt unit (2026-11-01) wrote an exception-edge map of the reader-commit branch (file:line mapped) and ran it as a fault-injection matrix on the stock jar: a throwing onInit (non-NRT and NRT reader), MockDirectoryWrapper I/O failures scoped to IndexFileDeleter.<init> and ReaderPool.<init>, LFV-1062's leaf-sorted unequal-size reader, two controls and a deleter file-set check. 5 of 8 fail on stock, 8 of 8 pass with the fix, and each hunk alone fixes only its own triggers.
• It reduced the lead to the one-test repro, traced the history to LUCENE-8260 and ran the test on releases back to 7.3.1.