openIfChanged returns null and isCurrent() stays true after a flush when the only changes are doc-values updates

Suppose the only changes since an NRT reader was opened are doc-values updates, or soft deletes made with updateDocValues that leave the segment at least one live document. If flush(), a forceMerge that has nothing to merge, or a writeAllDeletes reader writes those changes to disk, the reader is then reported current and DirectoryReader.openIfChanged(reader, writer) returns null. A fresh reader shows the change, yet SearcherManager keeps serving the old values.

It is deterministic and needs only the default config and a single document. The filed case fails on every tested release from 7.4.0 through 10.5.2, on branch_10_6 d04ee73177 (before 10.6.0) and on main; the 10.6.0 release (4ae946e5a9b) was not run. The same gap makes hasUncommittedChanges() return false for an update that the writer has applied but not yet written, since 7.0.0. It was filed as #16900. A three-file fix that records the change in IndexWriter.finishApply passes the repro and the full core test suite (one seed).

You keep a near-real-time (NRT) reader open on an IndexWriter and refresh it with DirectoryReader.openIfChanged(reader, writer). SearcherManager.maybeRefresh() ends in the same code: it calls openIfChanged(reader, commit) with a null commit by default, and for an NRT reader that also goes to the writer (StandardDirectoryReader.java:412-413). You change documents in place with doc-values updates, for example updateNumericDocValue, and at some point the writer is flushed.

1. Set up. Index two documents, id 1 and id 2, each with a numeric doc-values field val = 1. Open r = DirectoryReader.open(w). The reader records the writer's version number: 3.
2. Update. Call w.updateNumericDocValue(new Term("id", "1"), "val", 2).
3. Without a flush, it works. r.isCurrent() is false, and openIfChanged(r, w) returns a new reader that shows val = 2, at version 4.
4. With a flush, it fails. Do the same, but call w.flush() before reopening:
   • r.isCurrent() says true.
   • openIfChanged(r, w) returns null, which means "nothing changed".
   • A fresh DirectoryReader.open(w) shows val = 2 and also reports version 3, the same number as the stale reader r.

The same happens in these cases:

• Soft delete, then flush. The soft delete is made with updateDocValues on the soft-deletes field. The deleted document stays visible: the old reader counts 2 documents, a fresh one 1. This needs the segment to keep a live document (here id 2). A soft delete of a segment's last live document makes the writer drop the empty segment with a version bump, and the reopen works.
• A merge that has nothing to merge: forceMerge(1) on an index that already has one segment.
• A reader opened with writeAllDeletes = true, that is DirectoryReader.open(writer, true, true). No flush is needed; that reopen itself returns null.
• SearcherManager. maybeRefresh() returns true, which its javadoc defines as "either refreshed or that there were no changes to refresh"; it returns true in the working case too. The symptom is that acquire() keeps returning the stale searcher.

• The version is the reader's change stamp. isCurrent() and openIfChanged answer "nothing changed" when the writer's version equals the reader's and nothing is pending in the writer (IndexWriter.nrtIsCurrent, IndexWriter.java:6140-6146).
• A doc-values update goes through three stages. It is buffered, then applied (resolved to document numbers and parked in the writer's per-segment state, the reader pool), then written to a new doc-values file on disk.
• Applying it does not bump the version. finishApply bumps it only when documents were deleted, or when it drops segments whose documents are all deleted (IndexWriter.java:6721-6724, :6726-6734). A doc-values-only packet does neither. While the update is parked, it counts as pending (readerPool.anyDocValuesChanges()), so the reader is correctly not current.
• The normal reopen bumps it. It writes the parked update and calls checkpoint(), which raises the version (IndexWriter.java:608, :4123-4126). That is step 3.
• Three other paths write it without a bump. flush(), a forceMerge with nothing to merge, and a writeAllDeletes=true reader all write the parked update through ReaderPool.commit and then checkpointNoSIS(), which "does NOT bump segmentInfos.version" (IndexWriter.java:4119-4122, :4424, :2913-2916). The comment in ReaderPool.commit says the bump "was done previously (after we invoked BDS.applyDeletes)" (ReaderPool.java:363-369). That is true for deletes, not for doc-values updates.
• After such a write, nothing is pending and the version is still 3. So the writer reports the reader as current, and doOpenFromWriter returns null (StandardDirectoryReader.java:462, :469-472). That is step 4.
• Second symptom. The missing step also leaves the writer's change counter untouched. Commit, set a tiny RAM buffer so the update is applied at once, and update: hasUncommittedChanges() returns false, yet the next commit() writes val = 2 (IndexWriter.java:4225-4232).
• A fourth path, with no flush at all. Under reader-pool RAM pressure, writeSomeDocValuesUpdates writes parked updates with checkpointNoSIS() too (IndexWriter.java:854-859). It needs at least two update packets: the probe committed 200,000 documents, then, with a 0.5 MB buffer, sent one update matching all of them and 100,000 updates matching nothing. There isCurrent() is false, yet openIfChanged returns null, because the reader it opens has the same version as the old one and is discarded (StandardDirectoryReader.java:469-472), as in the writeAllDeletes case.

• Silent wrong results. There is no exception. Searches keep reading old doc-values, so anything built on them sees old values, such as sort keys, scoring signals, facets and doc-values filters (inferred from the stale values, not run feature by feature). Documents soft-deleted with updateDocValues stay visible. The stale state lasts until something bumps the version: an add, a hard delete that actually removes a document, a real merge or a commit. A delete that matches nothing does not end it, and repeated reopens keep returning null.
• Default configuration. The filed steps exactly (the default IndexWriterConfig, one document, no threads) return null on every release from 7.4.0 to 10.5.2, on branch_10_6 d04ee73177 (before 10.6.0; the 10.6.0 release was not run) and on main. The hunt test fails 20 of 20 derived seeds for each of its 5 bug tests. The reopen forms were run only on one- and two-document indexes and in a 200,000-document RAM-pressure probe; nothing in the mechanism depends on size, but that is read from the code. The soft-delete form uses the opt-in soft-deletes field. The writeAllDeletes form uses the expert open(writer, true, true).
• Who is exposed. Applications that make doc-values-only changes and then do one of these:
  • call flush();
  • call forceMerge(1) or forceMergeDeletes() when there is nothing to merge;
  • open, anywhere on the same writer, a reader with writeAllDeletes=true. This makes even a different, plain NRT reader stale;
  • send updates that hit reader-pool RAM pressure. That takes at least two update packets; the probe used one update matching 200,000 documents plus 100,000 that matched nothing, with a 0.5 MB buffer.
• NRT helpers inherit it.
  • SearcherManager and ReaderManager keep the stale reader.
  • ControlledRealTimeReopenThread.waitForGeneration returns true although the update is not visible.
  • The replicator's PrimaryNode opens its manager with writeAllDeletes=true (PrimaryNode.java:125). In the study, its flushAndRefresh() did not publish a doc-values-only change ("version 3 -> 3").
• "Commit only if dirty" can lose updates. The study's sequence: tiny buffer, update, if (hasUncommittedChanges()) commit(), then rollback(). The commit was skipped and the update was lost. With the default 16 MB buffer, updating 1.5 million distinct documents once each made hasUncommittedChanges() false after update 1,470,089.
• Downstream, from reading code only. Elasticsearch and OpenSearch are not exposed on their normal paths. They do not call IndexWriter.flush() on the indexing path, they open readers with writeAllDeletes=false, and every soft delete also adds a document. Applications built on PrimaryNode that use updateDocValues are possibly exposed. Solr in-place updates followed by an optimize or expungeDeletes that merges nothing were only simulated at the Lucene level.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected a new reader after a doc-values update, but got null
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertNotNull(Assert.java:713)
       org.apache.lucene.index.TestOpenIfChangedAfterDocValuesUpdateAndFlush.testNumericUpdateThenFlush(TestOpenIfChangedAfterDocValuesUpdateAndFlush.java:42)

• The contracts say otherwise (DirectoryReader.java at ea57360912ed):
  • openIfChanged(reader, writer) returns a reader "searching both committed and uncommitted changes from the writer" (:304-306), and null only "if there are no new changes" (:334-335).
  • For an NRT reader, isCurrent() "checks if either a new commit has occurred, or any new uncommitted changes have taken place via the writer" (:633-634).
  • getVersion() "is advanced every time a change is made with IndexWriter" (:620-621).
  • IndexWriter.hasUncommittedChanges() "Returns true if there may be changes that have not been committed". Its javadoc documents false positives and names no case where it returns false with uncommitted changes, though it does not promise that in so many words (IndexWriter.java:4219-4224). Here the next commit() does write the update, and LUCENE-5289 (#6353) fixed a false negative of this method as a bug.
• The code's own comment assumes the bump happened. ReaderPool.commit says the version bump "was done previously (after we invoked BDS.applyDeletes)" (ReaderPool.java:363-369). That holds for deletes only.
• An extra call makes a change less visible. Without flush() the update is seen. A hard delete followed by flush() is detected (version 3 to 4). So is tryUpdateDocValue followed by flush(), because that method calls changed() (IndexWriter.java:1780-1782).
• It used to work. On 7.3.1 the same steps return a new reader with val = 2 at version 4. The behaviour flips exactly at LUCENE-8290, a change about soft deletes whose issue never discusses isCurrent, openIfChanged or the version.
• No test or document covers it. The two randomized stress tests that reopen readers accept a null reopen without comparing it with a fresh reader (TestNumericDocValuesUpdates.java:1370-1374, TestMixedDocValuesUpdates.java:263-267), and they never flush after a doc-values update. Other tests do flush after doc-values updates (TestMixedDocValuesUpdates.java:531-546, :658-662), but then open a fresh DirectoryReader.open(writer) (:560, :675) instead of reopening an existing reader, so they cannot see the staleness. No javadoc, CHANGES or MIGRATE entry describes it.
• Precedents were fixed as bugs. #6050 (LUCENE-4986): an NRT reader missed tryDeleteDocument. #6353 (LUCENE-5289): hasUncommittedChanges() returned false for a pending delete-by-term.

The strongest case for "intended" was built by the independent reviewer:

• Writer-based NRT is @lucene.experimental, with "no hard guarantee on how quickly you can get a new reader" (DirectoryReader.java:313-314).
• The IndexWriter class javadoc says flushed changes "are not visible to IndexReader until either commit() or close is called" (IndexWriter.java:131-133).
• getVersion() first says it "returns the version recorded in the commit that the reader opened" (DirectoryReader.java:620), and only then that it "is advanced every time a change is made".
• Lucene's own randomized tests tolerate a null reopen.

Why it fails:

• The latency caveat is about how fast a reopen happens, not about a reopen that reports "no changes" and never sees them.
• The flush sentence is unqualified, so it does conflict with the NRT contracts. It predates them: it was added in 2008 (LUCENE-1194, 5b113c8af63), a year before writer-based NRT readers (LUCENE-1516, 3c91517add2, 2009), and its wording is the same today apart from formatting. The NRT javadocs are the more specific promise, and Lucene itself does not follow the flush sentence here: a fresh NRT DirectoryReader.open(writer) does see the change, and a hard delete across the same flush() is detected. Only isCurrent() and openIfChanged disagree with the fresh reader.
• @lucene.experimental weakens API stability, not the stated meaning of null.

The reviewer rated the argument "moderate" and the bug "real" with high confidence. One caution for the issue: the same javadoc still says "(though, the current implementation never returns null)" (DirectoryReader.java:306-307). That line is stale, so do not rely on it.

One fix, Fix B, was validated in a scratch worktree at ea57360912ed: it passes the repro, the closest suites and the full core suite. Fix A comes from the study and is known to be incomplete. The other options are untested sketches.

What it does:

• FrozenBufferedUpdates sets a flag on the segment state when it adds a resolved update (:355).
• closeSegmentStates returns that flag in ApplyDeletesResult.
• finishApply calls checkpoint() when doc-values updates were resolved and nothing was deleted.

Three files, +16/-2, all package-private.

if (result.anyDeletes()) {
  maybeMerge.set(true);
  checkpoint();
} else if (result.anyDocValuesUpdates()) {
  // Resolved doc values updates are a new NRT-visible change, just like deletes: bump the SIS
  // version now, because the flush/commit/RAM-pressure paths that later write them to disk
  // (ReaderPool#commit, writeSomeDocValuesUpdates) only call checkpointNoSIS():
  checkpoint();
}

Trade-offs

• Reach. It fixes every trigger (flush(), no-op forceMerge, writeAllDeletes readers, RAM pressure) and the hasUncommittedChanges() symptom, because it fixes the step they share.
• Cost. One extra checkpoint() (an IndexFileDeleter checkpoint) per applied packet that resolved doc-values updates, the same cost deletes already pay. Not benchmarked.
• Version timing. The version now also moves when updates are applied. Reopen frequency should not change, because a parked update already makes nrtIsCurrent false. That is reasoning from the code, not a measurement.
• Two bumps per update on the normal reopen path. The version moves once when the packet is applied and again when writeReaderPool(false) writes the update and calls checkpoint() (IndexWriter.java:4123-4126). A probe on main shows it: an update and a reopen without flush give version 4 unpatched and version 5 with Fix B (with -ea and -da). The reader-side checks compare versions for equality (IndexWriter.java:6143, StandardDirectoryReader.java:469), so the gap is probably harmless; no other user of the version was checked.

What the checks validated

At ea57360912ed, 2026-10-25, Gradle with JDK 25:

• Repro. 6 of 6 pass (seed DEADBEEF). With assertions off and -Dtests.iters=10 (seed 1234ABCD), 60 of 60 pass. Unpatched, the seed DEADBEEF command fails 5 of 6 with and without assertions; the 10-iteration command was not run unpatched.
• Closest suites. 17 classes, 341 tests, 0 failures, 12 skipped (seed 5EED0001): TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestMixedDocValuesUpdates, TestIndexWriterReader, TestDirectoryReaderReopen, TestSoftDeletesRetentionMergePolicy, TestSoftDeletesDirectoryReaderWrapper, TestIndexWriter, TestIndexWriterDelete, TestReaderPool, TestPendingSoftDeletes, TestStressNRT, TestIndexWriterThreadsToSegments, TestDocValuesFieldUpdates, TestSearcherManager, TestControlledRealTimeReopenThread and the repro (so the count includes its 6 tests).
• Full core suite. ./gradlew :lucene:core:test: 8,871 tests, 0 failures, 323 skipped (seed 5EED0002; the repro file was in the tree, so this includes its 6 tests).
• Nightly replicator. :lucene:replicator:test -Dtests.nightly=true (seed 5EED0003): 15 tests, 0 failures, 1 skipped, including TestNRTReplication 13 of 13 and TestStressNRTReplication 1 of 1.
• Format and lint. ecjLintMain and ecjLintTest pass. checkGoogleJavaFormat passes for the three fixed files and flags only line wrapping in the repro test class; run ./gradlew tidy before using it in a PR.
• Earlier, at the study commit 7cecf79523. The study report counts 32 new tests passing, including the RAM-pressure, PrimaryNode and hasUncommittedChanges tests (its logs show two runs, of 30 + 1 and of 31 tests). The full core suite: 8,895 tests, 0 failures. Nightly TestNRTReplication: 13 of 13. A TLA+ model of the version protocol: clean at 221,551 and 2,835,596 states.
• Independent reviewer. Their own two test classes pass with it, and so do TestNumericDocValuesUpdates.testUpdatesAreFlushed, testReopen and TestIndexWriterReader.testIsCurrent.

Six lines, which restore what getReader did before 7.4. Before readerPool.commit, call readerPool.writeAllDocValuesUpdates() and then checkpoint().

if (writeDeletes) {
  if (readerPool.writeAllDocValuesUpdates()) {
    checkpoint();
  }
  if (readerPool.commit(segmentInfos)) {
    checkpointNoSIS();
  }
}

Trade-offs

• Fixes flush(), no-op forceMerge and writeAllDeletes.
• Leaves the RAM-pressure path (writeSomeDocValuesUpdates) and hasUncommittedChanges() broken. At the study commit, 2 of the study's 30 new tests failed with it: the RAM-pressure test and a repeated-reopen probe. A TLA+ model found the hole first.
• Not re-run at ea57360912ed.

Let ReaderPool.commit report "doc-values updates written" separately from "live docs written", and call checkpoint() for the first. Do the same in writeSomeDocValuesUpdates. Untested sketch.

Trade-offs

Same reach as Fix A plus the RAM path. hasUncommittedChanges() stays wrong, because an applied but unwritten update is still not counted.

Untested sketch.

Trade-offs

Fixes only the second symptom, so it would have to come together with Fix A or the write-site option.

Compare per-segment doc-values and delete generations, instead of relying on the SegmentInfos version alone. Untested sketch.

Trade-offs

More invasive: it changes what "same version" means for readers and for doOpenFromWriter's equal-version shortcut.

Found by AI-assisted randomized model-based testing, without formal tools. A randomized test of a hash-based postings format under development, written and run with AI, failed on some seeds; the failures traced back to this reopen behaviour, and it was filed as #16900 on 2026-10-23.

Re-found by the 2026-10-22 study with 14 methods. None of these finds was blind: the task texts named the bug-note file pattern or listed the classes, so the area was hinted.

• Fastest. A bug-agnostic stateful fuzzer found it in under a second. A TLA+ model of the version protocol (NrtVersion.tla) gave a 2-step trace for flush() and another for writeAllDeletes. The same model later found the hole in Fix A.
• Missed. Five static analyzers: none of about 2,170 index-package reports pointed at it.
• What every success had in common. An oracle that states the freshness contract ("null or isCurrent() implies the same content as a fresh reader"). An operation set with a doc-values-only change plus a flush, a no-op forceMerge or a writeAllDeletes reader. And, in the models, checkpoint and checkpointNoSIS kept as separate steps.
• Second symptom. The hasUncommittedChanges() form was reached first by Alloy and TLC model checking in sessions that had no hint of this symptom (the area itself was hinted). In the study's votes Alloy called it a new bug and TLA+ the same root cause; a follow-up adjudicated it as a second symptom of the same root cause: one fix at finishApply removes both.

This hunt (2026-10-25):

• turned the study's tests into one minimal JUnit (5 bug tests and a control) and re-verified it on main ea57360912ed, 10.5.2 and 9.12.3;
• had it reviewed by an independent reviewer from another model family (OpenAI GPT-5.6), who wrote and ran their own repro;
• re-bisected the history and repeated the prior-report search;
• validated Fix B with Gradle.