A late message from the old primary makes an NRT replica delete the new primary's pre-copied merged segment

In lucene-replicator's NRT segment replication every promoted primary gets a higher primary generation. ReplicaNode.newNRTPoint calls maybeNewPrimary before any version check, and maybeNewPrimary treats any generation different from lastPrimaryGen as a new primary: it deletes the merged segment files pre-copied from the current primary (pendingMergeFiles), and only then asserts that the generation went up. So after a clean fail-over (the old primary closed, then a replica promoted), one delayed notification or merge pre-copy request from the old primary deletes the new primary's pre-copied merged segment, with assertions on and off; with assertions off the replica also goes back to the old generation. The replica's next cut-over to the merged point then fails with a misleading "may be missing or corrupt" exception, or the merged segment is copied again. It recovers at the primary's next point, and nothing wrong is searched. Present since 6.0.0 (LUCENE-5438); ignoring an older generation before anything is touched fixes the repro and passes the module's nightly suites.

// ReplicaNode.newNRTPoint(), abridged
public synchronized CopyJob newNRTPoint(long newPrimaryGen, long version) throws IOException {
  maybeNewPrimary(newPrimaryGen);                    // :545  before any check
  if (version == curVersion) { return null; }        // :558  same version: skip
  if (version < curVersion) { return null; }         // :564  older version: skip ("This can happen")
  ...

// ReplicaNode.maybeNewPrimary()
protected synchronized void maybeNewPrimary(long newPrimaryGen) throws IOException {
  if (newPrimaryGen != lastPrimaryGen) {             // :707  any change, older too
    for (String fileName : pendingMergeFiles) {
      deleter.deleteIfNoRef(fileName);               // :718  1. pre-copied files deleted
    }
    assert newPrimaryGen > lastPrimaryGen ...;       // :721  2. the only ordering check
    lastPrimaryGen = newPrimaryGen;                  // :723  3. assertions off: back to 1
    pendingMergeFiles.clear();                       // :724
  } ...

launchPreCopyMerge (a merge pre-copy request) calls maybeNewPrimary the same way (:736).

1. Primary and replica at generation 2. The primary starts with generation 2 and forcePrimaryVersion 11, past every version (up to 10) the generation-1 primary published, as a promotion does; the replica starts following it.
2. Two segments, one sync. Two empty documents in two segments (_0, _1), flushAndRefresh(), and the replica copies that point.
3. Pre-copy. forceMerge(1) makes _2; the primary's merged-segment warmer pre-copies [_2.cfe, _2.cfs, _2.si] to the replica through launchPreCopyMerge, which records them in pendingMergeFiles. They have no reference yet: one is taken only when a point that uses them is installed (:446, :450).
4. The late call. newNRTPoint(1, 10) arrives. maybeNewPrimary(1) sees 1 != 2 and runs deleteIfNoRef on the three files, which deletes them.
5. Assertions on: the assert throws newPrimaryGen=1 vs lastPrimaryGen=2, after the deletion, and also skips pendingMergeFiles.clear(), so close() later tries to delete the files again (NoSuchFileException: _2.si, suppressed). Assertions off: lastPrimaryGen becomes 1 and the call returns at the older-version skip.

• Late call while B copies P2's merged point: the copy listed the pre-copied files as identical (CopyJob.java:85), so it fetches nothing; the cut-over reads the new SegmentInfos (:433-434), which references the deleted _3.si, and throws RuntimeException: CorruptSegmentInfoException: segment info file: _3.si cannot be read - it may be missing or corrupt before the searcher is switched (:440). B keeps serving version 11.
• Late call before that point: the merged point's copy fetches all three files again, 2167 bytes in the probe.
• Assertions off: a commit in the window records __primaryGen=1 (:368); by reading, a restart from it does a full, safe re-sync (:189-191).
• In every case a commit right after succeeds, and B recovers at P2's next point (version 15, generation 2).

• Who is hit. Applications that use lucene-replicator's NRT segment replication (PrimaryNode / ReplicaNode / CopyJob subclasses, all marked @lucene.experimental). Every release since 6.0.0, any index size, assertions on or off.
• What it takes. A fail-over during which one notification (or merge pre-copy request) from the old primary reaches a replica after it started following the new primary, plus a merged segment pre-copied from the new primary that no published point uses yet. Lucene's reference transport does not rule this out: the primary sends each notification on its own connection (SimplePrimaryNode.java:358-366, test code), the server runs one handler thread per connection (TestSimpleServer.java:344-356), and the replica hands the received generation to newNRTPoint unchecked (SimpleReplicaNode.java:205-218; :331-339 for pre-copy). Only timing keeps it rare. Likelihood: rare; nothing has been reported since 2016.
• What goes wrong. (1) The cut-over to the new primary's merged point fails with a misleading "may be missing or corrupt" exception and the replica keeps serving its previous point until the primary publishes again; the reference job runner logs and ignores it (Jobs.java:105-111, test code), so it shows as a failed refresh. (2) Or the merged segment is copied again, the cost the pre-copy exists to avoid ("NRT reopen time on replicas is only in proportion to flushed segment sizes, not merged segments", PreCopyMergedSegmentWarmer.java:33-36); for a large merge that is a full extra transfer and a slow refresh. (3) With assertions off the replica reports the old generation, and a commit in that window is tagged with it.
• What does not go wrong. No wrong documents are searched: the replica serves its previous point or the new primary's, the failure comes before the searcher switch, a commit right after succeeds, and nothing hangs.
• Impact class: crash (an unexpected exception on a state the API accepts); priority low (opt-in experimental module, rare trigger, recovers at the next point).

The ~120 lines below the test method are the minimum PrimaryNode, ReplicaNode and CopyJob subclasses that run the protocol on one thread, copying files straight from the primary's directory. The merged segment is pre-copied through the real path (PreCopyMergedSegmentWarmer -> preCopyMergedSegmentFiles -> ReplicaNode.launchPreCopyMerge). It lives in the module's package because launchPreCopyMerge is protected and pendingMergeFiles package-private, like the module's own tests. assertFalse(preCopied.isEmpty()) guards against a vacuous pass. The late call carries (1, 10): 10 is below the replica's version, so the replica never contacts the old primary and the only work the call does is maybeNewPrimary.

The seed only drives the test framework's randomization: 10 of 10 iterations under another seed fail with the same message, with assertions on and off. The consequences (the failed cut-over and the re-copy) are shown by a longer probe, not by this test; see the full explanation above. Use this one test as the regression test too. It uses an unnamed catch variable (_, Java 22+): in a 10.x checkout, whose source level is Java 21, rename it to e.

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on (assertions off: identical message)
TestReplicaLateOldPrimaryNotification > testLateNotificationFromOlderPrimaryGenKeepsPreCopiedMergedFiles FAILED
    java.lang.AssertionError: pre-copied [_2.cfe, _2.cfs, _2.si] but after the late call the replica has [_0.cfe, _0.cfs, _0.si, _1.cfe, _1.cfs, _1.si]
        at org.apache.lucene.replicator.nrt.TestReplicaLateOldPrimaryNotification.testLateNotificationFromOlderPrimaryGenKeepsPreCopiedMergedFiles(TestReplicaLateOldPrimaryNotification.java:59)

# repro harness, assertions on: the same message, plus (suppressed, from ReplicaNode.close at ReplicaNode.java:694)
#   java.nio.file.NoSuchFileException: _2.si
# 10.5.2 (no compound file): pre-copied [_2.fdm, _2.fdt, _2.fdx, _2.fnm, _2.si] but after the late call the replica has [_0.cfe, _0.cfs, _0.si, _1.cfe, _1.cfs, _1.si]
# control (the late call carries generation 2): passes, assertions on and off
# with fix A: passes, assertions on and off

# consequence probe (main 645d4fe14a20, assertions off; on: the same plus AssertionError newPrimaryGen=1 vs lastPrimaryGen=2)
late=DURING_COPY: B followed P2 at version 11, pre-copied merged files [_3.cfe, _3.cfs, _3.si]; late P1 notification: ...
  lastPrimaryGen after it=1 pre-copied files then missing=[_3.cfe, _3.cfs, _3.si]; P2's merged point (version 13): files to copy [] ...;
  after that copy B serves version 11 ...; copy failure=java.lang.RuntimeException: org.apache.lucene.index.CorruptSegmentInfoException:
  segment info file: _3.si cannot be read - it may be missing or corrupt ...; commit right after: failure=null;
  P2's next point (version 15 ...): B then serves version 15 ..., lastPrimaryGen=2
late=BEFORE_MERGED_POINT: ... files to copy [_3.si, _3.cfs, _3.cfe] (pre-copied files copied again: [_3.cfe, _3.cfs, _3.si], bytes copied 2167) ...
late=NONE (control): ... pre-copied files then missing=[] ... bytes copied 0 ...

• The generation exists for this comparison. Node.java:50-54: the primary gen "increments every time we promote a new primary, so replicas can detect when the primary they were talking to is changed"; PrimaryNode.java:136-139: "incremented each time a new primary is started for this index"; ReplicaNode.java:85: lastPrimaryGen is the "Primary gen last time we successfully replicated".
• The discard rationale covers only a new primary. The comment above the call (ReplicaNode.java:536-544) explains discarding pre-copied segments for "the same primary (just e.g. rebooted), or a promoted replica that had a newer NRT point". Nothing about an older generation.
• Reordering is already expected. An older version "can happen, if two syncs happen close together, and due to thread scheduling, the incoming older version runs after the newer version" (:564-566), and is skipped.
• The pre-copied files are the current primary's. :82-83: "Merged segment files that we pre-copied, but have not yet made visible in a new NRT point".
• No caller precondition is documented in the newNRTPoint javadoc (:525-529), maybeNewPrimary (:705), the package docs, CHANGES.txt or MIGRATE.md. The module author's 2017 design post: "once you pick the new primary, Lucene will take care of having all replicas cutover to it, removing stale partially copied files from the old primary"; it leaves leader election, failure detection, load balancing, cluster state and "the low level mechanics of how to copy bytes from primary to replica" to the application, and says nothing about late or reordered messages.
• History. Before 9afa560054aa (2016-02-08) maybeNewPrimary ran after the copy job was created, with the generation of the primary actually serving the copy, and deleted nothing. That commit moved the call to the top and added the deletion ahead of the older assert-only check, with a comment about new primaries only. No module test sends an older generation (the three explicit newNRTPoint calls in TestNRTReplication carry the generation-0 primary's own generation).
• The control. The same test with generation 2 in the late call passes, with assertions on and off: only the older-generation path deletes the files.

Strongest case for "intended" (an independent reviewer from another model family, OpenAI GPT-5.6, refute lens, reading only): the newNRTPoint javadoc speaks of a new infos "on the primary", and a deposed generation-1 sender is no longer the primary; the assert states that callers present generations in increasing order; the module is @lucene.experimental; the design post leaves tracking which node is the primary to the application, so filtering stale senders is its job.

Why it fails:

1. No javadoc states the precondition; an assert that fires after the damage is no contract, and with assertions off it is not checked at all.
2. The trigger is a clean fail-over: the old primary is already closed, so there is no live sender to fence, only a message in flight. Nothing documents that the caller must filter it: the application's ReplicaNode subclass could compare against the protected lastPrimaryGen (ReplicaNode.java:85-86), but no javadoc asks it to, the module's own test transport does not (SimpleReplicaNode.java:205-218 passes the received generation on unchecked), and the replica already holds that value and drops older versions the same way.
3. @lucene.experimental is about API stability, not about which inputs may damage the replica's state.
4. The reviewer agreed with the root cause, could not run Java, and rated its argument moderate; the tie-breaker, which ran the test, the control and the fix, confirmed the bug. The split-brain reading applies only to the study's dropped first test (a deposed primary that keeps publishing), which is not claimed.

One fix was run through gradle (A): the study's patch without its optional third hunk in start(). A check placed where the assert is now does not work, because the files are already deleted by then.

A notification from an older generation is logged and ignored before anything is touched, exactly like an older version (return null is what the version skips already return). maybeNewPrimary refuses an older generation before its deletion loop, which also covers a late launchPreCopyMerge from the old primary. The existing assert stays and is now always true.

--- a/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/ReplicaNode.java
+++ b/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/ReplicaNode.java
@@ -533,6 +533,17 @@ public abstract class ReplicaNode extends Node {
       throw new AlreadyClosedException("this replica is closed: state=" + state);
     }
 
+    if (newPrimaryGen < lastPrimaryGen) {
+      // A notification from a primary that has since been replaced: ignore it
+      message(
+          "top: ignoring NRT point from stale primaryGen="
+              + newPrimaryGen
+              + " (current primaryGen="
+              + lastPrimaryGen
+              + ")");
+      return null;
+    }
+
     // Cutover (possibly) to new primary first, so we discard any pre-copied merged segments up
     // front, before checking for which files need
     // copying.  While it's possible the pre-copied merged segments could still be useful to us, in
@@ -704,6 +715,10 @@ public abstract class ReplicaNode extends Node {
 
   /** Called when the primary changed */
   protected synchronized void maybeNewPrimary(long newPrimaryGen) throws IOException {
+    if (newPrimaryGen < lastPrimaryGen) {
+      throw new IllegalStateException(
+          "primaryGen=" + newPrimaryGen + " is older than current primaryGen=" + lastPrimaryGen);
+    }
     if (newPrimaryGen != lastPrimaryGen) {
       message(
           "top: now change lastPrimaryGen from "

Trade-offs

• A stale launchPreCopyMerge caller now gets an exception instead of a started job. Returning silently there instead would leave the reference transport's handler waiting on its finished flag until it notices the dead connection (SimpleReplicaNode.java:339-362), so the exception is the cleaner signal (read, not run).
• The ignore in newNRTPoint is silent apart from the log line; a maintainer may prefer to throw there too and let the transport decide. The repro accepts either.
• A check at the assert's current position does not fix it: the study's tie-breaker ran such a patch and its probe still failed 2 of 3 with assertions off.

What the checks validated

• Scratch worktree of main 645d4fe14a20 (removed afterwards), seed DEADBEEF: stock, the test fails through gradle with assertions on and off; with A it passes with both.
• With A, the whole :lucene:replicator:test task passes (16 tests: the repro ran; the 14 nightly tests and TestSimpleServer, which runs only as a child process, were skipped).
• With A and nightly tests enabled: TestNRTReplication 13 tests, 0 failures (51 s); TestStressNRTReplication 1 test, 0 failures (one run of 326 s, one seed only).
• Earlier, with the study's three-hunk patch shadowed into the stock jars: the repro passes with assertions on and off (also 10 of 10 iterations), the consequence probe passes 3 of 3, and the two hunks backported to 6.0.0 pass there.

Throw for newPrimaryGen < lastPrimaryGen at the top of maybeNewPrimary and leave newNRTPoint alone.

Trade-offs

Smaller, and it protects the pre-copied files on both paths, but a late notification then throws to the transport instead of being skipped like an older version.

ReplicaNode.java:191, assert myPrimaryGen < curPrimaryGen, guards a replica restarting from a commit tagged with a newer generation than the primary it is given; the study's three-hunk patch throws IllegalStateException there.

Trade-offs

That value comes from the caller of start(), not from a late message; a separate hardening, not needed for this bug.

• Method (semi-formal). The hunt plan for the replicator listed primary-generation fencing ("no copy accepted from a deposed primary") as a property to model. The finder wrote a TLA+ model of the NRT replication protocol with maybeNewPrimary in both assertion modes. TLC found the fencing property violated with assertions off (2076 distinct states); with assertions on the "no assert fires" property is violated instead (23 states). That trace needs a deposed primary that keeps publishing, and the property holds under the module's controller discipline (220 states), so the finder's first JUnit (a split brain) was dropped from the claim.
• Narrowing (non-formal). The devil's-advocate verifier ran a probe with the old primary closed before the promotion: with assertions off the generation went back to 1 and the next commit recorded it. The tie-breaker's probe added the merge pre-copy and showed the deletion, the failed cut-over and the re-copy, with assertions on and off.
• Reduction. The one-test repro used here was then written from the narrowed scenario.