After one failed doc-values close, a reopened reader is handed the closed producer

All readers of one segment core share one SegmentDocValues, a cache with one reference-counted doc-values producer per doc-values update generation. When the last reader of a generation lets go, release() closes the producer and only then removes the generation from the cache (SegmentDocValues.java:54-59 on main). If the close throws, the removal is skipped and RefCount.decRef puts the count back to 1 ("Put reference back on failure", RefCount.java:51-54). Nobody owns that count: the closing reader is already marked closed (IndexReader.java:224), so the release is never retried. The next reader of the same core that needs the generation, for example one returned by openIfChanged(reader, commit) onto a kept commit, gets the closed producer, and its doc-values reads that touch the closed files throw AlreadyClosedException (in the repro, the first read). A fresh DirectoryReader.open of the same commit reads correct values. It needs an input close that throws after closing its file, which the stock MMapDirectory input close never does, plus commit-point navigation with kept commits; it fails loudly and changes no data. The order dates from LUCENE-5189 (2013): every release from 4.6.0 to 10.6.0 and main are affected. The fix removes the cache entry before closing.

• Doc-values generation: each commit that carries doc-values updates for a segment writes them as a new generation of doc-values files (for example _0_1_Lucene90_0.dvd, the file whose close failed in the independent reproducer's diagnostic run). A reader of that commit reads each field from the generations that hold its values: with main's default doc-values overlays (8, IndexWriterConfig.java:115), n is read from generation 1 (the sparse update) layered over generation -1, the segment's original doc values (SegmentDocValuesProducer.java:77-81, :182-186); with overlays off, as in 10.6.0, only from the generation that last rewrote it.
• SegmentDocValues: one per segment core, shared by every SegmentReader of that core (SegmentReader.java:153, this.segDocValues = sr.segDocValues;). It maps each generation to a RefCount<DocValuesProducer> (SegmentDocValues.java:35-36).
• Reopen across a doc-values-only change: openIfChanged(reader, commit) builds the new segment reader from the old one (StandardDirectoryReader.java:328-337), so both draw from the same cache.

// SegmentDocValues.java:51-60
return new RefCount<>(dvFormat.fieldsProducer(srs)) {
  protected void release() throws IOException {
    object.close();                       // :55  throws here...
    synchronized (SegmentDocValues.this) {
      genDVProducers.remove(gen);         // :57  ...so this never runs
    }
  }
};

// RefCount.java:46-55 (condensed)
if (refCount.decrementAndGet() == 0) {
  try { release(); }
  catch (Throwable t) { refCount.incrementAndGet(); throw t; }   // "Put reference back on failure"
}

// SegmentDocValues.java:64-75: a cached entry is incRef'd and returned without any check

1. r1.close() reaches SegmentReader.doClose (SegmentReader.java:214-225), which decRefs generation 1 once. The count hits 0, release() runs, and Lucene90DocValuesProducer.close (:510-512, IOUtils.close(data, ...)) throws from the data input, which the repro's wrapper has already closed (IOUtils.close still tries the other inputs).
2. The removal at :57 is skipped and RefCount.decRef restores the count to 1. IndexReader.decRef set closed = true before doClose (IndexReader.java:222-227), so r1 can never retry: the count has no owner.
3. openIfChanged(r2, commit 1) asks the shared cache for generation 1, finds the entry, incRefs it to 2 and returns the closed producer. The reader opens normally; its first doc-values read touches the closed data input and throws (a dense constant column would still read: it is served from metadata, Lucene90DocValuesProducer.java:895-902).

With the repro's plain directory the read fails in ByteBuffersIndexInput.ensureOpen with AlreadyClosedException: Already closed.; under the test framework's MockDirectoryWrapper the same bug shows as its use-after-close check, "Abusing closed IndexInput!".

• Who is hit. After one failed close, every later reader of that segment core that needs the released generation is handed the closed producer, and its doc-values reads that touch the closed files fail (in the repro, the first read), until all readers of the core are closed.
• What triggers the failed close. A doc-values input close() that throws while a reader closes, after it has really closed its file (the repro's wrapper closes its delegate, then throws). The stock MMapDirectory input close propagates no exception (MemorySegmentIndexInput.java:707-730 retries until the arena closes); FSDirectory.open returns MMapDirectory on 64-bit JREs and NIOFSDirectory otherwise (FSDirectory.java:155-161). NIOFSDirectory (FileChannel.close, NIOFSDirectory.java:122-126) or a custom or filter Directory can throw there. A close that throws before closing its file may leave the producer usable or only partly closed (not tested).
• Default vs opt-in. Every call is public, documented API; the deletion policy that keeps older commits and the commit navigation are the application's choice. No scale is needed: 2 documents and 1 update.
• Loud, contained. A clean AlreadyClosedException, no data changed, no wrong results; a fresh DirectoryReader.open of the same commit reads correct values. Before 6.6.0 the failed close() itself returned normally, so the first sign was the later read failure.
• Not claimed. Wrong results, corruption, and the NRT (IndexWriter reader pool) path were not tested.
• Likelihood: contrived. Two rare events in one process; no public report of it was found since 4.6.0 (2013).

Input: two docs, id = 0, 1 with n = 0, 1; commit 0; updateNumericDocValue(id:0, n, 10); commit 1. The test draws no random values. It uses two docs with distinct values: per the minimization notes, with one doc the generation-1 column is a constant and the read never touches the closed file. The fault wrapper's close() closes its delegate and then throws (lines 45-49); that order is what leaves the cached producer unusable. It belongs in lucene/core/src/test/org/apache/lucene/index/, next to TestDirectoryReaderReopen, and passes :lucene:core:check -x test.

The exception is thrown by getNumericDocValues("n") on r3, the first doc-values call, so the two assertions after it are never reached on stock code; with the fix they pass.

Regression test: this same test.

TestReopenAfterFailedDocValuesClose > testReopenAfterFailedDocValuesClose FAILED
    org.apache.lucene.store.AlreadyClosedException: Already closed.
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:365A404F094011A9]:0)
        at org.apache.lucene.store.ByteBuffersIndexInput.ensureOpen(ByteBuffersIndexInput.java:223)
        at org.apache.lucene.store.ByteBuffersIndexInput.slice(ByteBuffersIndexInput.java:62)
        at org.apache.lucene.store.ByteBuffersIndexInput.slice(ByteBuffersIndexInput.java:28)
        at org.apache.lucene.store.FilterIndexInput.slice(FilterIndexInput.java:97)
        at org.apache.lucene.codecs.lucene90.IndexedDISI.createBlockSlice(IndexedDISI.java:376)
        at org.apache.lucene.codecs.lucene90.IndexedDISI.<init>(IndexedDISI.java:304)
        at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.newIndexedDISI(Lucene90DocValuesProducer.java:277)
        at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.getNumeric(Lucene90DocValuesProducer.java:1084)
        at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.getNumeric(Lucene90DocValuesProducer.java:781)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.getNumeric(PerFieldDocValuesFormat.java:318)
        at org.apache.lucene.index.SegmentDocValuesProducer.getNumeric(SegmentDocValuesProducer.java:184)
        at org.apache.lucene.index.CodecReader.getNumericDocValues(CodecReader.java:156)
        at org.apache.lucene.index.TestReopenAfterFailedDocValuesClose.testReopenAfterFailedDocValuesClose(TestReopenAfterFailedDocValuesClose.java:81)

• The contract. openIfChanged(DirectoryReader, IndexCommit): "If the IndexCommit differs from what the provided reader is searching, open and return a new reader" (DirectoryReader.java:276-287); resources are shared "when possible" (:233-234). Nothing documents that sibling readers break after a failed close; IndexReader.close only says no other methods should be called on the closed reader (IndexReader.java:345-346), and the repro never touches r1 again.
• The cache's design. A design comment in the LUCENE-5189 discussion (#6253) plans for SegmentDocValues to drop a generation once no reader uses it: "If it also removes a gen from the map when it's no longer referenced by any SR, we don't need to take care of clearing the genDVP map when all SRs were closed"; release() does so, but only when the close succeeds.
• Lucene's close policy. LUCENE-5553 (#6615, 4.7.1) made a failing reader close still close everything and dropped the reference put-back from IndexReader.decRef, with maintainers agreeing on "close really closes and throws first exc" behaviour. RefCount kept the same put-back one level down.
• Precedents and tests. LUCENE-7775 (#8826: decRef dropped every close exception because of an inverted null check; fixed to throw the first one) and #13436 (later producers were not closed after a failing one) fixed other close failures in SegmentDocValues.decRef as bugs. TestDirectoryReaderReopen.testOverDecRefDuringReopen asserts that the old reader keeps working after its reopen fails on an I/O error (an open failure there, not a close failure).

The strongest case (the devil's-advocate verifier's steelman): RefCount.decRef puts the reference back on purpose ("Put reference back on failure"), which reads as retry-later semantics, so keeping the entry after a failed release is by design. TaxonomyReader.decRef (lucene/facet) still has the same put-back.

Why it fails: nothing can retry here. The idiom mirrors the IndexReader.decRef of 2013, and LUCENE-5553 removed it there on purpose; the closing reader is marked closed before doClose, so it can never retry (TaxonomyReader.decRef sets closed only after doClose succeeds, so a retry is possible there); SegmentDocValues is the only Lucene user of this RefCount; and no javadoc or test documents the put-back (RefCount.java:22-45). The cross-model review (OpenAI GPT-5.6) rated the intended-behaviour case weak. What survives is a severity argument (rare trigger, loud failure), hence priority Low.

Validated with gradle in a clean scratch worktree of 5adff78bb168 (OpenJDK 25.0.3, seed DEADBEEF, 2026-11-03). Workaround until fixed: after a reader close fails, open the commits you need with DirectoryReader.open(commit) instead of openIfChanged from a reader of the same segment core, or close all readers of that core first.

A failed close leaves no entry behind, so the next request for the generation opens a fresh producer. This matches the LUCENE-5189 intent that a generation leaves the map once no reader references it.

       protected void release() throws IOException {
-        object.close();
+        // drop the cache entry first, so that a failed close cannot leave a closed producer cached
         synchronized (SegmentDocValues.this) {
           genDVProducers.remove(gen);
         }
+        object.close();
       }

Trade-offs

The success path is unchanged: release() runs only from SegmentDocValues.decRef, which holds the same lock as getDocValuesProducer (SegmentDocValues.java:64, :78), so no thread sees the in-between state. The half-closed producer is dropped rather than retried, which is no worse than today: nothing retries it now either. Its RefCount keeps the restored count of 1 but is no longer in the cache. Not explored: partly closed inputs, repeated close failures, file handles a failed close leaves open.

What the checks validated

• Stock 5adff78bb168: the repro fails through gradle with assertions on and off; with the fix it passes, on and off.
• With the fix, 11 suites (the repro, TestDirectoryReaderReopen, TestDirectoryReader, TestSegmentReader, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestMixedDocValuesUpdates, TestSortedNumericDocValuesUpdates, TestIncrementalDocValuesUpdates, TestIndexWriterOnError, TestIndexWriterExceptions): 226 tests, 0 failures, 6 skipped.
• :lucene:core:check -x test (format, lint, forbidden APIs) with the fix and the test: passes.
• Earlier, with the same reorder shadowed in, the finder's, the independent reproducer's and the cross-model reviewer's tests all passed, and the adjudication's re-run of the independent reproducer's test passed.

try { object.close(); } finally { synchronized (SegmentDocValues.this) { genDVProducers.remove(gen); } }. Same effect as the reorder.

Trade-offs

Slightly longer; not run.

What the checks validated

Not run (equivalent by reading).

Mirrors LUCENE-5553 for IndexReader. On its own it does not fix the bug: the entry would stay cached with count 0, and getDocValuesProducer would incRef it to 1 and hand out the closed producer.

Trade-offs

RefCount is a public class in org.apache.lucene.util; only worth doing together with the reorder, for consistency.

What the checks validated

Not run (insufficient by reading).

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

• The obligation: a producer whose release fails must not stay in the per-generation cache that the other readers of the segment core draw from. release() closes before it removes (SegmentDocValues.java:55-57), and RefCount.decRef restores the count on failure (RefCount.java:51-54), so the obligation is not met.
• A JUnit test using a test-framework fault at the generation-1 input close witnessed it on main (assertions on and off, three seeds) and on 10.6.0 and 9.12.3. An independent reproducer, an executed review by another model family (OpenAI GPT-5.6) and a devil's-advocate verifier wrote their own tests from the claim alone; the one-test plain-directory repro was cut from the independent reproducer's test on 2026-11-03.