IndexWriter opened from an older reader's IndexCommit can reuse a newer commit's version, so readers on that commit report isCurrent() == true

IndexWriterConfig#setIndexCommit can open a writer on an older commit that a deletion policy kept, and its javadoc invites passing an open reader's DirectoryReader#getIndexCommit. Since LUCENE-6524 (5.3.0) such a reader-backed commit takes its own branch of the IndexWriter constructor, which clones the reader's SegmentInfos, version included, and never reads the newest commit on disk; the branch taken for the same commit from DirectoryReader.listCommits keeps the newest commit's version "to preserve write-once. This is important if readers are open against the future commit points." So the new writer's versions restart below commits already in the directory, and one of them can equal the version of a newer commit with different content: in the repro segments_3 (3 documents) and segments_5 (1 document) are both version 10. A reader opened on segments_3 compares only versions, so it reports isCurrent() == true, openIfChanged returns null, a refreshed SearcherManager keeps serving the superseded commit (a deleted and committed document stays findable) and SearcherLifetimeManager.record throws IllegalArgumentException, until a later commit moves the version on. The index on disk is correct. It needs the Expert rollback through a reader's commit point while another reader stays on a newer commit, and an exact version match that depends on operation counts. An 11-line fix that raises the version to the newest commit's passes the repro and the whole core test suite.

// IndexWriter.java:982-990: a commit from DirectoryReader#getIndexCommit carries its reader
IndexCommit commit = config.getIndexCommit();
StandardDirectoryReader reader = commit == null ? null : commit.getReader();   // (condensed)
...
} else if (reader != null) {                                   // :1021  the reader branch
  ...
  segmentInfos = reader.segmentInfos.clone();                  // :1050  version = the reader's (4)
  ...
  if (reader.writer != null) {                                 // :1062  only for an NRT reader:
    // In case the old writer wrote further segments (which we are now dropping),
    // update SIS metadata so we remain write-once:            // :1067-1068
    segmentInfos.updateGenerationVersionAndCounter(reader.writer.segmentInfos);
    ...
  }
  rollbackSegments = lastCommit.createBackupSegmentInfos();    // :1073  no look at the newest commit
} else {                                                       // :1074  the commit-point branch
  String lastSegmentsFile = SegmentInfos.getLastCommitSegmentsFileName(files);
  segmentInfos = SegmentInfos.readCommit(directoryOrig, lastSegmentsFile);   // :1085  newest commit
  if (commit != null) {
    // Swap out all segments, but keep metadata in
    // SegmentInfos, like version & generation, to
    // preserve write-once. This is important if
    // readers are open against the future commit
    // points.                                                 // :1088-1092
    ...
    segmentInfos.replace(oldInfos);                            // :1103  keeps the newest version (10)
    changed();

1. Which branch runs. DirectoryReader#getIndexCommit returns a commit that carries the reader (StandardDirectoryReader.java:570-574, getReader() at :644-646). DirectoryReader.listCommits builds the same kind of commit with no reader (DirectoryReader.java:456, :481), so its getReader() returns null; a SnapshotDeletionPolicy.snapshot() commit inherits getReader() == null from IndexCommit (IndexCommit.java:112-118). Both take the commit-point branch.
2. The reader branch copies the old version. IndexWriter.java:1050 clones the reader's SegmentInfos, so the writer starts at segments_1's version 4. It raises the version only from the reader's own writer (:1062-1071), and only for an NRT reader; a plain reader has none. It never reads segments_3 (version 10).
3. The writer counts up from there. The constructor adds 2 at open (:1192-1196, "we always assume we are carrying over incoming changes"), and each commit-data commit adds 2 more in the repro:

1. File names never collide, versions do. IndexFileDeleter.inflateGens (IndexFileDeleter.java:257) raises the generation from the files in the directory, so the new commits are segments_4 and segments_5; nothing repairs the version.
2. isCurrent() compares only versions. For a reader without a live writer, StandardDirectoryReader.java:529-540 reads the newest commit and returns sis.getVersion() == segmentInfos.getVersion(): 10 == 10. openIfChanged returns null when isCurrent() (:477-482), so SearcherManager.refreshIfNeeded (SearcherManager.java:200) keeps the old searcher, and SearcherLifetimeManager.record, which keys searchers by version, throws IllegalArgumentException for a searcher on segments_5 (SearcherLifetimeManager.java:146-161).

Not every history collides: adding one document instead of commit data gives version 9 and no stale reader. The version restarts from the older reader's on every such open; whether a commit then lands below, exactly on, or above a newer commit's version depends on the operation counts, and only the exact match makes a reader stale.

• Who is hit. Applications that go back to an older commit through a reader's commit point, new IndexWriter(dir, new IndexWriterConfig().setIndexCommit(reader.getIndexCommit())), an Expert path the setIndexCommit javadoc documents, while a reader or SearcherManager opened on the directory is still on a newer commit. The older commit must still be on disk, so a deletion policy has to keep it: NoDeletionPolicy, a custom policy, or a SnapshotDeletionPolicy snapshot. The reader whose commit is passed may be a plain reader or an NRT reader whose writer closed before a later writer session committed.
• A realistic flow. A writer with SnapshotDeletionPolicy snapshots commit 1 (document a) and opens a reader on the snapshot, then commits b and c; a SearcherManager(dir, null) serves segments_3. A rollback writer on the default deletion policy, opened with setIndexCommit(snapReader.getIndexCommit()), adds d and commits. After maybeRefreshBlocking() the SearcherManager still serves segments_3 (version 10, 3 documents) although the only commit on disk is segments_4 (version 10, 2 documents). Run on main 645d4fe14a20 and, over 20 seeds, on a9779ef3c193.
• How often. It needs an exact version match, which depends on operation counts. An independent enumeration of 96 single-commit rollback histories (snapshot, SearcherManager, default policy on the rollback writer) left the SearcherManager stale in 14 through reader.getIndexCommit() and in 0 through the snapshot commit itself. No report was found in the 11 years since 5.3.0.
• What goes wrong. Silent and transient: isCurrent() true, openIfChanged null, a refreshed SearcherManager keeps serving the superseded commit, so a document that was deleted and committed is still found, until a later commit moves the version on (the follow-up's stale-searcher probe, main a9779ef3c193, 10.5.2 and 9.12.3). SearcherLifetimeManager.record throws IllegalArgumentException when two commits share a version.
• What does not go wrong. The index on disk is correct: CheckIndex is clean and a freshly opened reader sees the newest commit. Not the default configuration. Assertions on or off make no difference.
• Impact class: silent wrong results; priority low (Expert path, rare, transient).

The class sits in org.apache.lucene.index by Lucene test convention but uses nothing package-private. Use this one test as the regression test too.

The same failure, by the repro harness: main 645d4fe14a20 and the 10.6.0 jars, assertions on and off. The hunt's longer class (an NRT-reader variant, a SearcherLifetimeManager test and a listCommits control) fails 3 of 4 on main, 10.5.2 and 9.12.3; its control passes.

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on
TestWriterFromReaderCommitVersion > testWriterFromCommitReader FAILED
    java.lang.AssertionError: isCurrent() expected false but was true; reader version 10, latest commit version 10
        at org.apache.lucene.index.TestWriterFromReaderCommitVersion.testWriterFromCommitReader(TestWriterFromReaderCommitVersion.java:50)

# repro harness, the same message: main -ea, main -da, 10.6.0 -ea, 10.6.0 -da (1 run, 1 failed each)
# with fix A: the test passes (gradle, with 15 related suites, assertions on and off)

• Contract. DirectoryReader.java:628-629 (isCurrent): "If this reader was created by calling #open, then this method checks if any further commits (see IndexWriter#commit) have occurred in the directory." :637-638: "if this returns false, you should call #openIfChanged to get a new reader that sees the changes." :620-621 (getVersion): "This version is advanced every time a change is made with IndexWriter."
• The API invites the call. IndexWriterConfig.java:235-237: "Expert: allows to open a certain commit point. ... This can also be used to open IndexWriter from a near-real-time reader, if you pass the reader's DirectoryReader#getIndexCommit." Upstream TestIndexWriterFromReader#testAfterCommitThenIndexKeepCommits opens a writer from a reader that is behind the latest commit; that test class never checks isCurrent() of a reader left on the newer commit.
• Both branches keep the version on purpose. The commit-point branch keeps the version "to preserve write-once" (IndexWriter.java:1088-1092; SegmentInfos.java:1084-1086: replace "keeps generation, version, counter so that future commits remain write once"). The reader branch itself updates the metadata "so we remain write-once" (IndexWriter.java:1067-1068), but only from the reader's own writer.
• Same commit, two outcomes. The same commit from listCommits gives correct versions, and the same user call did in 5.2.1. Every other open path (create over an existing index, append, a listCommits commit) keeps the version at or above the newest commit's.
• Precedent. CHANGES.txt:18287-18288, LUCENE-2513 (2010): "when opening writable IndexReader on a not-current commit, do not overwrite \"future\" commits." There a writable IndexReader on an older commit overwrote a future commit's segments_N file (#3587): the file-level form of the same rule; here the file names are safe and the version is reused. Michael McCandless in LUCENE-3607 (#4681): "the version is still comparable across IRs opened at different times against the same index".
• History: a side effect. LUCENE-6524 (#7582) added the reader branch so the existing call could reuse the open reader; its 16 comments never mention the version, write-once or isCurrent.

Strongest case for "intended": setIndexCommit is Expert and getIndexCommit is @lucene.experimental, and Lucene never documents commit versions as strictly increasing or unique (LUCENE-2805 only says the version moves on every change, not on every commit). Going back to an old commit starts a new branch of history, and versions across branches are not comparable.

Why it fails:

1. On main every commit that is written gets a higher version than the writer's previous one (IndexWriter.java:3828-3835, from LUCENE-6523, 2015), and a commit without changes is not written at all (:5934-5936). Here two commits with different content, segments_3 (3 documents) and segments_5 (1 document), share one version, and a reader that the javadoc says detects "any further commits" does not.
2. Lucene already treats going back as one history in this directory: the commit-point branch keeps the newest version for exactly this reason, and the result must not depend on whether the IndexCommit object came from listCommits or from a reader.
3. Expert lowers the reach, not the correctness bar; no javadoc, CHANGES or MIGRATE entry, test or issue found documents a weaker guarantee for a reader's commit.

One code fix was run; A is recommended. Workaround for users today: pass the matching commit from DirectoryReader.listCommits(dir) (same segments file name as reader.getIndexCommit()) instead of the reader's commit; it takes the commit-point branch, which keeps the versions right. This works for plain readers of a commit; an NRT reader's uncommitted changes are in no listed commit. The writer then opens its own segment readers instead of sharing the held reader's (ReaderPool.java:90-112 pre-enrolls them only for a reader-backed commit).

Read the newest segments_N (the directory listing is taken under the write lock) and raise segmentInfos.version to its version when it is higher. The writer still reuses the reader's segments and open segment readers; the constructor then adds 2 as before, so its first commit is above the newest commit's version. Cost: one extra read of the newest commit file at open, as the commit-point branch already does.

--- a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
@@ -1070,6 +1070,17 @@ public class IndexWriter
           lastCommit.updateGenerationVersionAndCounter(reader.writer.segmentInfos);
         }
 
+        // Like the commit-point branch below, never go below the newest commit's version, to
+        // preserve write-once: readers may be open against commits newer than the reader's:
+        String latestSegmentsFile = SegmentInfos.getLastCommitSegmentsFileName(files);
+        if (latestSegmentsFile != null) {
+          long latestVersion =
+              SegmentInfos.readCommit(directoryOrig, latestSegmentsFile).getVersion();
+          if (latestVersion > segmentInfos.version) {
+            segmentInfos.version = latestVersion;
+          }
+        }
+
         rollbackSegments = lastCommit.createBackupSegmentInfos();
       } else {
         // Init from either the latest commit point, or an explicit prior commit point:

Trade-offs

It fixes new opens only; commits that already share a version keep it. It compares with the newest commit only, as the commit-point branch does. On an index where an affected release already wrote a newest commit with a lower version than an older one (segments_4 version 8 after segments_3 version 10), a later collision with an older commit's version is not ruled out: through the commit-point branch, which A does not change (a plain commit() right after opening segments_1 from listCommits writes segments_5 version 10, and a reader on segments_3 reports isCurrent() == true, stock and with A), or through the reader branch when the older commit's version is more than 2 above the newest's. In this example the reader branch with A commits at 11 or higher. B, applied to both branches, closes that.

What the checks validated

• Scratch worktree of main 645d4fe14a20 (removed afterwards), seed DEADBEEF: stock, the test fails through gradle; with A it passes.
• The repro and 15 related suites with A, -Ptests.asserts=true and false: TestIndexWriterFromReader, TestDeletionPolicy, TestSnapshotDeletionPolicy, TestPersistentSnapshotDeletionPolicy, TestDirectoryReader, TestDirectoryReaderReopen, TestTransactionRollback, TestRollback, TestIndexWriterCommit, TestIndexCommit, TestIndexFileDeleter, TestIndexWriterReader, TestCrash, TestIndexWriter, TestSearcherManager: 294 tests, 5 skipped, 0 failures each time.
• All of :lucene:core:test with A (assertions on): 8,898 tests in 749 classes, 322 skipped, 0 failures.
• With A shadowed into the main jars: the hunt's 4 tests (incl. SearcherLifetimeManager and the NRT-reader variant) and the snapshot-rollback probe pass, 5 of 5 (stock: 4 of 5 fail); the version probe gives 14, 16, 18.

Read every segments_N in the listing (or reuse the commits the file deleter loads at init) and take the maximum. Raised by the cross-model root-cause reviewer.

Trade-offs

Also covers indexes already made non-monotonic by an affected release. Costs one read per retained commit at open (many with NoDeletionPolicy), and the commit-point branch would need the same change to stay consistent. Not run.

For example also the commit generation.

Trade-offs

Hides the isCurrent/openIfChanged symptom but leaves two commits with different content and one version on disk, which SearcherLifetimeManager (keyed by version) still trips on. The hunt's validated isCurrent patch for the sibling LFV-012 leaves all 3 failures of the hunt's class. Not recommended alone.

• Method (semi-formal): bounded-exhaustive translation validation in a hunt unit on reopen equivalence. Every sequence of up to k operations over a 22-28-operation alphabet (adds, deletes, doc-values and soft updates, addIndexes, flushes, merges, commits, rollbacks, writer reopens, writers from a reader's commit, NRT reopens) ran on the stock jars in 32 configurations, with oracles for reopen equivalence, isCurrent soundness and write-once commit versions.
• Signal. The hypothesis came from reading the constructor while modelling the operation "writer from a reader's commit"; the harness then witnessed it independently at k=4 (308 sequences per configuration group; shortest: force-merge, commit, writer from reader commit, commit, giving segments_2 and segments_3 both version 9).
• Attribution. With this fix sketch shadowed in, together with other bugs' sketches, the "commit versions not increasing" signature disappeared in all 32 configurations; with only the other sketches it remained.
• Reduction. The finder wrote a public-API JUnit (4 tests, one control); it was later minimized to the one test above.