Closing a FilterCodecReader closes the wrapped segment's files behind its reference count

FilterCodecReader.doClose() calls in.doClose(), the wrapped reader's internal close hook, instead of in.close(), which releases one reference. So closing the wrapper closes the wrapped segment's files even while another reader, typically the DirectoryReader the segment came from, still holds it. With the default codec and MMapDirectory, that reader then silently finds 0 hits for a term that matched 1 document, its stored fields throw AlreadyClosedException, and its own close() later releases the segment a second time and, if the segment has doc values, throws. FilterLeafReader calls in.close() and is correct in the same sequence; FilterDirectoryReader also forwards to in.close() (the LUCENE-6696 fix). The call came in 6.4.2 (LUCENE-7676, 2017); every release from 6.4.2 to 10.6.0 (10.6.0 by reading) and main is affected. Lucene's production code never closes these wrappers, so only applications that close one while its segment is still in use are hit.

• Reference count. Every IndexReader counts its users: incRef() adds one, close() and decRef() remove one, and the reader closes its files only when the count reaches 0. doClose() is the hook decRef() runs at that moment (IndexReader.java:215-232, "Implements close." at :358-359).
• Segment reader (leaf). A DirectoryReader holds one SegmentReader per segment; each is a CodecReader.
• FilterCodecReader. A public abstract wrapper around a CodecReader, mainly used to feed IndexWriter.addIndexes(CodecReader...) and merges. SortingCodecReader is a subclass.

1. FilterCodecReader.doClose() is in.doClose(); (FilterCodecReader.java:123-126). It compiles only because the class is in IndexReader's package.
2. Closing the wrapper takes the wrapper's own count to 0 and runs this hook, which runs SegmentReader.doClose() on the wrapped segment: core.decRef() and segDocValues.decRef(...) (SegmentReader.java:213-225). The segment's own count is not touched, so it stays above 0, its closed flag stays false and ensureOpen() keeps passing (IndexReader.java:238-248: it checks the count and the never-set closedByChild flag, not closed).
3. The shared core closes postings, stored fields, points, vectors and norms (SegmentCoreReaders.java:167-180). The terms reader clears its field map on close (Lucene103BlockTreeTermsReader.java:272-280) and the points reader its readers map (Lucene90PointsReader.java:153-157), so with the default codec term and point queries see an empty segment, with no exception (randomized test codecs and directories can throw instead). Stored fields, doc values and vectors throw AlreadyClosedException. In the example the segment reader is the core's only user; readers of one segment may share the core and the doc-values producers (SegmentReader.java:42-43), and then the files close at a later release, possibly while another reader still uses them (read, not run).
4. When the owner's release takes the segment's count to 0 (StandardDirectoryReader.java:566), SegmentReader.doClose() runs a second time. SegmentDocValues.decRef finds the producer already gone: assert dvp != null : "gen=" + gen, or a NullPointerException without assertions (SegmentDocValues.java:83-84). A segment without doc values closes silently. While another holder keeps a reference the owner's close can be clean: in the near-real-time probe it was, because the IndexWriter's reader pool still held the segment reader.
5. FilterLeafReader.doClose() and FilterDirectoryReader.doClose() call in.close() (FilterLeafReader.java:416-419, FilterDirectoryReader.java:162-164). A bytecode sweep over the core, test-framework and module jars (all but benchmark, demo, luke and distribution) finds no other call of another reader's doClose().

Default codec, MMapDirectory, 10 documents in one segment, id:3 has 1 hit. With an extra reference taken for the wrapper, the segment's count is 2 before and after the wrapper's close; after the leftover reference is released, count(id:3) is 0, stored fields throw AlreadyClosedException: this FieldsReader is closed, and DirectoryReader.close() throws AssertionError: gen=1 (or NullPointerException ... "dvp" is null), on main and 10.5.2, with assertions on and off. Re-sorting with SortingCodecReader.wrap and addIndexes, then closing the wrappers without an extra reference: the copy has all 10 documents, the still-open source reader finds 0 hits for id:3, and its close() throws gen=-1.

• Who is hit. Code that closes a FilterCodecReader (its own subclass, or the wrapper SortingCodecReader.wrap returns: a SortingCodecReader, or an anonymous FilterCodecReader when the reader is already sorted) whose wrapped reader is a leaf of an open DirectoryReader or a near-real-time reader. The natural trigger is copying, filtering, re-sorting or reordering an index through addIndexes(CodecReader...) and then closing the wrappers, with or without an extra reference: addIndexes does not close the readers it is given ("The provided IndexReaders are not closed"), and FilterCodecReader does not document whether closing it releases the wrapped reader.
• Who is not. Lucene's production code (no harmful call site found): merges release the pooled segment reader, not the merge wrapper; SoftDeletesDirectoryReaderWrapper closes only its wrapped DirectoryReader; the misc splitters and the BP reorderer's documented usage never close their wrappers. Two downstream wrappers that were read (Solr's index splitter, Elasticsearch's lazy soft-deletes reader) are not closed directly. No affected downstream is known. The test framework closes such a wrapper in two norms-format tests, harmlessly (nothing uses the reader afterwards).
• What the damage looks like. On the damaged segment, with the default codec and MMapDirectory, term and point queries silently match nothing (randomized test codecs and directories can throw instead); stored fields, doc values and kNN throw; other segments answer correctly. The owner's close() throws when it takes the segment's count to 0 and the segment has doc values. With a near-real-time reader (no deletes; with deletes not run) the damage carries into readers reopened from the same IndexWriter, and the next merge of that segment is a tragic event that closes the writer and loses uncommitted documents (5 of 15 in a probe); the committed index stays intact and CheckIndex is clean.
• Settings. Opt-in usage pattern of an expert API; otherwise defaults: default IndexWriterConfig, default codec, assertions on or off. 10 documents are enough. The root cause does not depend on the directory; the runs on record used MMapDirectory and randomized test directories on JDK 25 / Linux aarch64.
• How likely. Rare: it needs code that closes such a wrapper and then keeps using the wrapped reader's owner. A one-off migration often closes the source reader right after addIndexes, which then only shows as a throwing close. No report since 2017.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: id:1 hits in the open DirectoryReader after closing a FilterCodecReader expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.index.TestFilterCodecReaderCloseKeepsReferencedReader.testClosingWrapperKeepsReferencedSegmentSearchable(TestFilterCodecReaderCloseKeepsReferencedReader.java:55)

• Contracts. IndexReader#incRef (IndexReader.java:164-170, repeated for tryIncRef at :188-192): "RefCounts are used to determine when a reader can be closed safely ... the IndexReader will not really be closed until decRef has been called for all outstanding references." decRef (:207-209): "If the refCount drops to 0, then this reader is closed." doClose (:358-359): "Implements close." FilterLeafReader (FilterLeafReader.java:328): "Note that base reader is closed if this FilterLeafReader is closed", implemented as in.close(). addIndexes(CodecReader...) (IndexWriter.java:3257): "The provided IndexReaders are not closed": IndexWriter leaves their lifecycle to the caller. FilterCodecReader itself documents no ownership: its constructor only stores in and takes no reference (FilterCodecReader.java:54-61).
• Sibling code. FilterLeafReader and FilterDirectoryReader call in.close(); MultiReader calls close() or decRef(); DirectoryReader releases its leaves with decRef(). A bytecode sweep over the core, test-framework and module jars (all but benchmark, demo, luke and distribution) finds no other call of another reader's doClose().
• Tests. No upstream test asserts the current behaviour (TestFilterCodecReader checks only that every method is overridden, and getDelegate). Two norms-format tests close a test-framework MergingCodecReader leaf instead of its DirectoryReader and rely on that releasing the segment: they fail with an empty doClose() and pass with in.close(). That is compatibility evidence against option B, not a documented ownership contract.
• History. in.doClose() came with LUCENE-7676 (#8727, ae68e6cebc3, 2017-02-08, @cpoerschke), "Fixed FilterCodecReader to override more super-class methods", together with the reflective test that requires an override of every CodecReader method. The issue body is empty; @rmuir wrote "looks good" and @jpountz "+1"; nobody discussed close behaviour. Before it, closing a FilterCodecReader did not touch the wrapped reader (5.0.0 to 6.4.1, where the inherited CodecReader.doClose() closed only the wrapper's own thread-local state; executed on 5.5.5 and 6.4.1).
• Precedent. LUCENE-6696 (#7754, 2015, @jpountz, fixed in 5.3.0): the same in.doClose() in FilterDirectoryReader, fixed with in.close(): "This is wrong because if you call close() on both the underlying reader and a wrapper around it, then doClose() will have been called several times, which will break ref counting."

The strongest "works as designed" case (built by the cross-model refute reviewer, the intended-behaviour reviewer and the devil's advocate; the cross-model reviews rated it weak, and the devil's advocate found that none of its points holds up): FilterCodecReader is a merge and addIndexes view that Lucene's production code never closes; SortingCodecReader "should only be used for merging"; its constructor takes no reference and no javadoc says whether the wrapper owns its delegate (the strongest point), so closing a wrapper around someone else's segment without taking a reference could be called a caller error; and the line is old and two committers approved it.

Why it fails: "never closed by Lucene" limits reach, not validity, since close() is public and addIndexes does not close the readers it is given; "only for merging" is about cost. No ownership model makes in.doClose() right: owning one reference means in.close(), owning none means an empty doClose() (as before 6.4.2, the wrapped reader is left alone), and even exclusive ownership would leave the wrapped reader looking open. Even the "caller error" case (no extra reference) gives a silent 0 hits where FilterLeafReader throws AlreadyClosedException. The 2017 review discussed only the test, so the line's age is no evidence of acceptance. What the case gets right: ownership is undocumented, so closing a wrapper without a reference can reasonably be called misuse; under that reading option A turns the silent 0 hits into a loud AlreadyClosedException rather than making the call work, and A releases only one reference even when several wrappers each took one. That bears on the fix and the framing; with a reference taken and released, the result still breaks the documented incRef/decRef contract. Verdict: not intended, a defect.

Three code options were built and run in a scratch worktree at ea57360912ed. How each behaves when a caller closes a wrapper: with an extra reference taken (the test), A and C are correct and B leaks that reference; without one (the re-sort flow), B is correct and A and C fail loudly with AlreadyClosedException instead of today's silent 0 hits; with two wrappers over one segment, each with its own reference, only C releases both (A releases one, as FilterLeafReader does today).

doClose() calls in.close(); the constructor javadoc gains "Note that the underlying reader is closed if this FilterCodecReader is closed: closing this reader releases one reference to it". +4/-1 lines; ./gradlew tidy leaves it unchanged.

  @Override
  protected void doClose() throws IOException {
-    in.doClose();
+    in.close();
  }

Trade-offs

• Matches FilterLeafReader, FilterDirectoryReader and the LUCENE-6696 fix.
• Callers who close a wrapper without taking a reference and keep using the owner now get AlreadyClosedException instead of a silent 0 hits and a throwing close: loud, but a behaviour change (the re-sort probe).
• Two wrappers over one segment, each with its own reference, release only one between them, because IndexReader.close() is idempotent per instance; FilterLeafReader has the same property today.
• The same idempotency means that after the wrapper's in.close() the wrapped reader's own close() does nothing, so an owner that releases it with close() rather than decRef() leaks one reference. DirectoryReader and the IndexWriter's reader pool release leaves with decRef(), so the repro and the validated suites are unaffected; FilterLeafReader and FilterDirectoryReader behave the same today (executed with FilterDirectoryReader by the fact check).

What the checks validated

• Both repro tests: 6 of 6 pass with -ea and -da; 15 of 15 with -Ptests.iters=5.
• The two repro classes plus 18 related core suites (TestFilterCodecReader, TestIndexReaderClose, TestSoftDeletesDirectoryReaderWrapper, TestSortingCodecReader, TestIndexSorting, TestAddIndexes, TestIndexWriter, TestIndexWriterReader, ...): 372 tests, 0 failures.
• The whole core module: 749 classes, 8,871 tests, 0 failures, 322 skipped. The whole codecs module: 693 tests, 0 failures. The whole misc module: 315 tests, 0 failures. These module runs, the 18 suites and the static checks ran with assertions on (seed DEADBEEF); with assertions off, the repro tests and the standalone program.
• ./gradlew :lucene:core:check -x test (formatting, ecj lint, forbidden APIs): BUILD SUCCESSFUL.
• The standalone program on the fixed jar: 1 hit, clean close, with -ea and -da; the security probe: 0 violations.

Closing a wrapper never affects the wrapped reader, as in 5.0.0 to 6.4.1 (where the inherited CodecReader.doClose() closed only the wrapper's own thread-local state), and like the view SlowCodecReaderWrapper.wrap builds for a reader that is not a CodecReader (a CodecReader is returned unchanged).

Trade-offs

• The try-with-resources idiom around addIndexes just works.
• A caller who takes a reference for the wrapper, as FilterLeafReader needs, silently leaks it, and the segment is never closed.
• Breaks TestLucene90NormsFormatMergeInstance.testIndependentIterators and testIndependentSparseIterators ("MockDirectoryWrapper: cannot close: there are still 7 open files"), which close a MergingCodecReader leaf instead of its DirectoryReader; they would have to change (compatibility pressure, not an ownership contract).

What the checks validated

Both repro tests and the 18 related suites pass with -ea and -da (372 tests, 0 failures); the whole core module: 8,871 tests, 2 failures (the two norms tests).

Releases exactly one reference per wrapper close.

Trade-offs

Two wrappers over one segment are correct too; it departs from the in.close() convention of the sibling classes, and behaves like in.close() when no reference was taken.

What the checks validated

Both repro tests and the 18 related suites pass with -ea and -da (372 tests, 0 failures); the whole module was not run.

Method: code reading (the hunt's "non-formal" tier), a hunt unit (delegation in Lucene's filter readers, 2026-10-26).

• The finder (an AI tool) wrote a bytecode scanner (JDK class-file API, over the core, test-framework and module jars, all but benchmark, demo, luke and distribution) that classifies every method of every wrapper class as a forward, a super call or custom code, and runs checks for missed overrides, partial views and cache-helper misuse. It listed FilterCodecReader#doClose as a plain forward.
• Reading the code next to FilterLeafReader's showed that forwarding this particular hook skips the delegate's reference count. A JUnit on the stock jars confirmed it.
• Check F8 (any IndexReader#doClose() called on an object other than this/super) was then added to the scanner; it finds only this call. No formal or bounded-exhaustive tool produced the first signal.
• Then an independent reviewer from another model family (OpenAI GPT-5.6) tried to refute it by running its own tests, and the hunt's checks followed (see the record).