ReplicaNode.commit() captures the current segments under the node lock, fsyncs their files holding only its separate commitLock, and only then reads VERSION_KEY from the searcher that is current. A point installed during the fsync therefore labels a commit that does not contain it.
- In
lucene-replicator's NRT segment replication (opt-in, experimental) aPrimaryNodeindexes and publishes NRT points, each with a version number; eachReplicaNodecopies the new files and switches its searcher to that point. - Any node can
commit(). The commit's user data storesVERSION_KEY, the version of the committed content; a node later started as a primary on that commit returns it fromgetLastCommitVersion(). - An application that keeps an operation log uses that number after a failover or a full restart: it replays the log from the position of that version, as the module's stress test does.
VERSION_KEY = 5. Select the still to open it at full size. Generated with AI, checked against the code.- A replica of an NRT-replicated index searches the primary's point version 3 (1 document). The primary adds a second document and publishes version 5 (2 documents).
- You call
replica.commit(). It captures the version-3 segments, then fsyncs their files without holding the node lock; meanwhile the replica installs version 5. - You expect the commit's
VERSION_KEYto be 3, the version it holds, or at worst lower:PrimaryNode.commit()accepts a lagging value because a replay then only repeats operations. - Instead it is 5: after the fsync,
commit()reads the version of whatever searcher is current. The commit holds 1 document but claims version 5. - A primary started on this commit reports
getLastCommitVersion()= 5 with only the first document. Replaying the operation log from version 5, as Lucene's own stress test does, skips the second document. No error anywhere.
Users of the opt-in, experimental NRT segment replication that commit replicas while the primary keeps refreshing, since 6.0.0. The label is wrong whenever a point is installed between the commit's capture and its version read, mostly the fsync (natural rate not measured); harm needs a primary started on that commit without re-committing and a log replay from getLastCommitVersion().
Mechanism, terms and numbers549 words
In lucene-replicator's NRT segment replication, ReplicaNode.commit() captures the current SegmentInfos under the node monitor, fsyncs their files while holding only its separate commitLock, and only then writes VERSION_KEY into the commit data from getCurrentSearchingVersion(), the version of whatever searcher is current at that moment. Installing a new NRT point takes only the node monitor, so a point that arrives during the fsync becomes the label of an older commit: in the repro the commit holds version 3 (1 document) and says 5 (2 documents). A PrimaryNode started on that commit returns 5 from getLastCommitVersion(), and an application that replays its operation log from that version after a failover or a cold start, as the module's own stress test does, skips the operations in between: adds are lost and deletes and updates undone. The committed index itself is consistent and Lucene never reads the value on a replica. The code dates from the feature's first commit (LUCENE-5438) and ships in every release from 6.0.0 to 10.6.0; the one-line fix VERSION_KEY = infos.getVersion() passes the repro and the module's nightly suites.
// ReplicaNode.commit(), abridged
final Object commitLock = new Object(); // :341
public void commit() throws IOException {
synchronized (commitLock) { // :346
synchronized (this) { // :350 node monitor
infos = ((SegmentInfosSearcherManager) mgr).getCurrentInfos(); // :351 version 3
indexFiles = infos.files(false);
deleter.incRef(indexFiles);
} // :354 monitor released
dir.sync(indexFiles); // :365 only commitLock held
commitData.put(PRIMARY_GEN_KEY, Long.toString(lastPrimaryGen)); // :368
commitData.put(VERSION_KEY, Long.toString(getCurrentSearchingVersion())); // :369 5 if installed meanwhile
infos.setUserData(commitData, false); // :370
infos.commit(dir); // :373 writes the version-3 infos
// Notify current infos (which may have changed while we were doing dir.sync above) what
// generation we are up to; ... // :375-377- Capture. Under the node monitor
commit()takes the currentSegmentInfos(version 3, 1 document) and increfs its files (ReplicaNode.java:350-354). That object is not changed later: installing a point builds a newSegmentInfosand swaps it in (:433-440;SegmentInfosSearcherManager.java:92-101updates the new object's generation, not the old one's). - Unlocked fsync.
dir.sync(indexFiles)runs holding onlycommitLock(:365).newNRTPointissynchronizedon the node (:530) andfinishNRTCopyinstalls the new point under the node monitor (:412,setCurrentInfosat:440); neither takescommitLock.launch(CopyJob)is documented as "Runs this job async'd" (:516-517), so the install normally runs on another thread while the commit fsyncs. - Late read. After the fsync,
VERSION_KEYcomes fromgetCurrentSearchingVersion()(:369), which returns the current searcher's reader version (Node.java:167-174): 5, the point installed meanwhile. - Write.
infos.commit(dir)writes the captured version-3 infos with that user data (:373). The next comment (:375-377) says the current infos "may have changed while we were doing dir.sync above" and handles that for the generation, not for the version. - Read back. A
PrimaryNodeopened on this commit keeps itsVERSION_KEY("keep current commitData version",PrimaryNode.java:99-114) andgetLastCommitVersion()returns it (:187-198): 5, while the index holds 1 document.ReplicaNodeitself reads onlyPRIMARY_GEN_KEYfrom a commit (:177).
When no point is installed between the capture and the late read, both give the same number, which is why idle commits are right (verifiers' idle-commit controls pass on stock).
Exposure and likelihood435 words
- Who is hit. Applications that use
lucene-replicator's NRT segment replication (PrimaryNode,ReplicaNodeandCopyJobsubclasses; the package is@lucene.experimental) and commit a replica while the primary keeps publishing points. The code is in every release since 6.0.0; it was reproduced on the releases listed under history with indexes of 1 and 2 documents, and nothing in the code path depends on the index size. - How often the label is wrong. Whenever a point is installed between a replica commit's capture, when it releases the node monitor (
ReplicaNode.java:354), and its late version read (:369); nearly all of that window is the fsync of the index files (:365). Every repro forces the interleaving (the minimal one from insideDirectory.sync, an instrumented schedule); the natural rate was not measured. - When it does harm. Only if a primary is later started on that replica commit without a fresh re-commit and the application replays an operation log from
getLastCommitVersion(). Lucene's own stress test does that replay: it maps each published version to the log position saved before its flush (TestStressNRTReplication.java:276-294), reads the new primary's commit version (:702), looks up that version's log position (:466-474) and replays from it (:520). It re-commits a replica before a live promotion (:425-434), which normally replaces a skewed commit, but its random commits while replication runs (:322-338) and its cold start after the whole cluster is down (:970-979) do not (read, not executed). - What goes wrong then. The operations between the committed version and the stored one are skipped: adds are lost, deletes and updates undone. In an executed probe a delete and an update of a per-document access-control field made inside the window were both undone after promotion and replay from
getLastCommitVersion()(7, while the commit held version 3); replaying from version 3 gave the right result. On stock main, 10.5.2 and 9.12.3 the probe's replay test fails and its control passes; with the fix both pass (run on main only). The probe is a custom in-process test. - What does not go wrong. The committed index is consistent: its files and
SegmentInfosare right, only the label is ahead. No exception; assertions on or off make no difference. - Downstream. A public code search (2026-10-26, not exhaustive) found no use of the NRT replicator in Elasticsearch, OpenSearch or Solr, and two open-source
ReplicaNodesubclasses that never readgetLastCommitVersionorVERSION_KEY. - Impact class: silent wrong results; priority low (opt-in experimental module, rare harm path, no report since 2016).
One JUnit test in the module's test package: an in-memory primary and replica, empty documents, single thread, deterministic. The replica directory's sync() installs the newer point inside the commit's fsync, the interleaving a copy thread produces. Put it in lucene/replicator/src/test/org/apache/lucene/replicator/nrt/ and run it with gradle.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.replicator.nrt;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.Collection;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Collectors;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.SegmentCommitInfo;
import org.apache.lucene.index.SegmentInfos;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FilterDirectory;
import org.apache.lucene.store.IOContext;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.util.IORunnable;
public class TestReplicaCommitVersionKey extends LuceneTestCase {
public void testCommitVersionKeyIsVersionOfCommittedInfos() throws Exception {
IndexWriter w = new IndexWriter(new ByteBuffersDirectory(), new IndexWriterConfig());
PrimaryNode primary =
new PrimaryNode(w, 0, 1, -1, null, null) {
@Override
protected void preCopyMergedSegmentFiles(
SegmentCommitInfo info, Map<String, FileMetaData> files) {}
};
AtomicReference<IORunnable> duringFsync = new AtomicReference<>();
Directory replicaDir =
new FilterDirectory(new ByteBuffersDirectory()) {
@Override
public void sync(Collection<String> names) throws IOException {
IORunnable r = duringFsync.getAndSet(null);
if (r != null) {
r.run();
}
super.sync(names);
}
};
Replica replica = new Replica(replicaDir, primary);
w.addDocument(new Document());
primary.flushAndRefresh();
replica.newNRTPoint(1, primary.getCopyStateVersion());
w.addDocument(new Document());
primary.flushAndRefresh();
duringFsync.set(() -> replica.newNRTPoint(1, primary.getCopyStateVersion()));
replica.commit();
SegmentInfos committed = SegmentInfos.readLatestCommit(replicaDir);
replica.close();
primary.close();
// expected: VERSION_KEY = version of the committed SegmentInfos; actual: the newer point's
assertEquals(
"VERSION_KEY of the replica commit",
committed.getVersion(),
Long.parseLong(committed.getUserData().get(Node.VERSION_KEY)));
}
private static class Replica extends ReplicaNode {
private final PrimaryNode primary;
Replica(Directory dir, PrimaryNode primary) throws IOException {
super(1, dir, null, null);
this.primary = primary;
start(1);
}
@Override
protected CopyJob newCopyJob(
String reason,
Map<String, FileMetaData> files,
Map<String, FileMetaData> prevFiles,
boolean highPriority,
CopyJob.OnceDone onceDone)
throws IOException {
CopyState state = primary.getCopyState();
return new CopyJob(reason, state.files(), this, highPriority, onceDone) {
@Override
public void finish() throws IOException {
for (Map.Entry<String, FileMetaData> e : toCopy) {
dest.getDirectory()
.copyFrom(primary.getDirectory(), e.getKey(), e.getKey(), IOContext.DEFAULT);
}
primary.releaseCopyState(state);
}
@Override
protected CopyOneFile newCopyOneFile(CopyOneFile current) {
throw new UnsupportedOperationException();
}
@Override
public void start() {}
@Override
public void runBlocking() {}
@Override
public boolean conflicts(CopyJob other) {
return false;
}
@Override
public boolean getFailed() {
return false;
}
@Override
public Set<String> getFileNamesToCopy() {
return toCopy.stream().map(Map.Entry::getKey).collect(Collectors.toSet());
}
@Override
public Set<String> getFileNames() {
return files.keySet();
}
@Override
public CopyState getCopyState() {
return state;
}
@Override
public long getTotalBytesCopied() {
return 0;
}
@Override
public int compareTo(CopyJob other) {
return Long.compare(ord, other.ord);
}
};
}
@Override
protected void launch(CopyJob job) {
try {
job.onceDone.run(job);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
@Override
protected void sendNewReplica() {}
}
}
# $REPRO holds TestReplicaCommitVersionKey.java; $LUCENE is a Lucene checkout
cp $REPRO/TestReplicaCommitVersionKey.java $LUCENE/lucene/replicator/src/test/org/apache/lucene/replicator/nrt/
cd $LUCENE
./gradlew :lucene:replicator:test --tests TestReplicaCommitVersionKey -Ptests.seed=DEADBEEF
- Expected
VERSION_KEYof the replica commit is 3, the version of theSegmentInfosit wrote (1 document).- Observed
VERSION_KEYis 5, the version of the point installed during the fsync:expected:<3> but was:<5>.
More files, environment, full output311 words
Only public and protected replicator API: an anonymous PrimaryNode and a small ReplicaNode subclass whose CopyJob copies the missing files with Directory.copyFrom and whose launch() runs the job at once; most of the file is that boilerplate, the scenario is 7 lines. Use this one test as the regression test too.
Why one thread is faithful: during dir.sync the commit holds only commitLock, while newNRTPoint and finishNRTCopy take only the node monitor, so a copy thread can install the point at exactly that moment. Running the install inside the first sync() after commit() starts makes that deterministic without a second thread, latches or sleeps. This is an instrumented, forced schedule: it shows that the locks allow the interleaving, not how often it happens naturally. The hunt's original two-thread test (a committer thread held in sync() by a latch) gives the same numbers and also starts a primary on the commit: getLastCommitVersion() = 5 with documents [d1].
Seeds DEADBEEF and 5EED1234 (3 iterations) fail the same way; the seed only drives the test framework's randomization.
| lucene | main 645d4fe14a20 and 10.6.0: reproduced, assertions on and off; 6.0.0 to 10.5.2: reproduced by the hunt's test; 5.5.5: no NRT replication API |
|---|---|
| jdk | OpenJDK 25.0.3 (build 25.0.3+9-LTS) |
| os | Linux aarch64 |
# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on (assertions off: identical)
TestReplicaCommitVersionKey > testCommitVersionKeyIsVersionOfCommittedInfos FAILED
java.lang.AssertionError: VERSION_KEY of the replica commit expected:<3> but was:<5>
at org.apache.lucene.replicator.nrt.TestReplicaCommitVersionKey.testCommitVersionKeyIsVersionOfCommittedInfos(TestReplicaCommitVersionKey.java:74)
# repro harness, the same message: main -ea, main -da, 10.6.0 -ea, 10.6.0 -da (1 run, 1 failed each)
# with the one-line fix: the test passes (gradle, assertions on and off)
# the hunt's two-thread test (also promotes a primary on the commit), main, 10.5.2, 9.12.3:
# AssertionError: replica commit: VERSION_KEY=5 (getLastCommitVersion()=5 after promotion), but the committed
# SegmentInfos is version 3 with documents [d1]; point 5 has documents [d1, d2] expected:<3> but was:<5>PrimaryNode.commit()accepts a lagging version because replay then means "replaying more ops than necessary"; a version ahead skips operations.- The module's stress test replays from the commit version a node "was guaranteed to already have indexed through".
commit()'s own comment says the infos "may have changed while we were doing dir.sync above".
The full argument281 words
- The accepted error is a lag.
PrimaryNode.java:222-224: "TODO (opto): it's a bit wasteful that we put "last refresh" version here, not the actual version we are committing, because it means on xlog replay we are replaying more ops than necessary." - The module's controller states the use.
TestStressNRTReplication.java:462-465: "Get xlog location that this node was guaranteed to already have indexed through; this may replay some ops already indexed but it's OK because the ops are idempotent".:486-488: "the userData in its latest commit point tells us which version it had indexed up to, so we know where to replay from in the xlog." - The method knows the infos can change.
ReplicaNode.java:375-377: "Notify current infos (which may have changed while we were doing dir.sync above) what generation we are up to". - Concurrent commits are designed for. The separate
commitLock(:341) lets the fsync run outside the node monitor, and the stress test commits a random node while replication runs (:322-338).TestNRTReplicationcommits replicas only after they caught up, and the stress test's related assert (NodeProcess.java:77-78) cannot detect this skew, because the new primary's version is force-advanced past every published refresh. - Design intent. On LUCENE-5438 (#6501, 2015-03-02) the author, mikemccand, wrote that the test's transaction log shows how NRT points are "used to replay indexing events on primary crash so no indexed documents are lost".
- Contract wording.
Node.java:56-60: "Key to store the version in the commit data, which increments every time we open a new NRT reader." Thin, but it names the commit's version. The "never ahead" rule itself is inferred fromPrimaryNode.java:222-224and the stress controller; the javadoc does not state it.
The devil's-advocate check135 words
Strongest case for "intended" (the devil's-advocate verifier): the package is @lucene.experimental; the VERSION_KEY javadoc never says the value equals the committed version; the primary's own value is approximate, so the replica's getCurrentSearchingVersion() could be read as a "latest point searched" watermark; and the reference controller re-commits a replica before a live promotion, so a skewed commit is replaced before use.
Why it fails:
- The direction of the error: the only documented approximation is a lag, justified by replay repeating operations; a lead has no use and makes replay skip operations.
- The controller's own cold start and random commits use a replica commit without re-committing it (read).
- Experimental lowers the reach, not the correctness bar; no javadoc, test or issue documents a value ahead of the content. The adjudication found no works-as-designed argument that survives.
Recommended A. Store the version of the captured infos (one line)
Store the version of the captured SegmentInfos: commitData.put(VERSION_KEY, Long.toString(infos.getVersion())) in ReplicaNode.commit(), one line. The repro passes, and so do the module's nightly suites with it: TestNRTReplication (13 tests) and TestStressNRTReplication (one run with a 304 s stress phase). Those suites are compatibility coverage; they do not force the interleaving.
--- a/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/ReplicaNode.java
+++ b/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/ReplicaNode.java
@@ -366,7 +366,7 @@ public abstract class ReplicaNode extends Node {
Map<String, String> commitData = new HashMap<>();
commitData.put(PRIMARY_GEN_KEY, Long.toString(lastPrimaryGen));
- commitData.put(VERSION_KEY, Long.toString(getCurrentSearchingVersion()));
+ commitData.put(VERSION_KEY, Long.toString(infos.getVersion()));
infos.setUserData(commitData, false);
// write and fsync a new segments_NAll options and trade-offs3 options · 448 words
One fix was run (A). Commits with no point installed between the capture and the late read store the same value as today; under the race the stored version is the older, correct one. A maintainer may also want the VERSION_KEY javadoc to say what the value means for replica and primary commits.
infos.getVersion() is exactly the version infos.commit(dir) writes: the captured object is not changed by a concurrent install, and setUserData(..., false) does not bump it (SegmentInfos.java:1073-1082).
--- a/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/ReplicaNode.java
+++ b/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/ReplicaNode.java
@@ -366,7 +366,7 @@ public abstract class ReplicaNode extends Node {
Map<String, String> commitData = new HashMap<>();
commitData.put(PRIMARY_GEN_KEY, Long.toString(lastPrimaryGen));
- commitData.put(VERSION_KEY, Long.toString(getCurrentSearchingVersion()));
+ commitData.put(VERSION_KEY, Long.toString(infos.getVersion()));
infos.setUserData(commitData, false);
// write and fsync a new segments_NTrade-offs
No adverse effect observed in the recorded tests. A replay from the stored version now repeats nothing and skips nothing. The stress test, which asserts that a new primary's commit version is in its version-to-log map (TestStressNRTReplication.java:475-484), passed with it; like the other suites it does not force the interleaving.
What the checks validated
- Scratch worktree of main
645d4fe14a20(removed afterwards), seed DEADBEEF: stock, the test fails through gradle with assertions on and off; with A it passes with both. - With A, the whole
:lucene:replicator:testtask passes (16 tests; the 14 nightly ones andTestSimpleServer, a child-process entry point that runs only nested, are skipped by default). - With A and nightly tests enabled:
TestNRTReplication13 tests, 0 failures;TestStressNRTReplication1 test, 0 failures (one run with a 304 s stress phase, 359 s for the test, with 29 random commits, 4 promotions and 4 cold starts; one seed only). - Earlier, with A shadowed into the stock jars: the hunt's and the verifiers' tests and the downstream probe (main) pass; on 6.0.0 the hunt's test passes with A. A separate run's race-versus-control test for the cross-model review passes with A on
645d4fe14a20, 2 of 2, assertions on and off (fact check).
long version = infos.getVersion(); next to getCurrentInfos() (ReplicaNode.java:351), stored later. Same effect as A, more explicit about the snapshot; a maintainer who also wants PRIMARY_GEN_KEY captured with the infos would put both there.
Trade-offs
Equivalent to A. Not run.
Makes commit and point install mutually exclusive, so the late read cannot see a newer point.
Trade-offs
Blocks every NRT install on that replica for the whole fsync of the index files, which the current code avoids by fsyncing outside the node monitor under its own commitLock. Not run; not recommended.
Downstream, an application that replays an operation log from getLastCommitVersion() loses the skipped operations, including deletes and access-control updates (an executed probe shows a deleted document searchable again), but no use of that replay pattern was found outside Lucene's own stress test.
TLA+ model checking (TLC) of the NRT replication protocol, with commit() modelled as its two halves (capture under the lock, write after the unlocked fsync): the invariant "a commit's VERSION_KEY is never ahead of its content" failed in a 6-state trace. A JUnit then drove the same interleaving on the stock jars and was reduced to one test.
The method in full144 words
- Method (formal): TLA+ model checking with TLC in a hunt unit on the NRT replication protocol. The finder's model splits
commit()into two actions as in the Java,CommitCaptureunder the node lock andCommitWriteafter the unlocked fsync, and checksCommitVersionSound == commit.versionKey <= commit.ver. - Signal. TLC reported the violation with a 6-state trace (294 distinct states, with assertions modelled off and on), also under the stress controller's discipline (202 states). With the fix modelled the invariant holds (7,836 distinct states for the commit model, 82,805 for the full model).
- Caveat. The finder's record does not say whether a code-reading hypothesis came before the TLC run.
- Reduction. A JUnit on the stock jars drove the same interleaving (a committer thread held in
Directory.syncby a latch while the replica installed the newer point); it was later minimized to the single-threaded test above.
No prior report found in GitHub issues and PRs, the file lists of all 413 open PRs, the git history, four mailing lists, the JIRA archive, CI failure mails or downstream repositories (2026-10-26; GitHub refreshed 2026-10-31). Every query naming VERSION_KEY or getCurrentSearchingVersion returned nothing. Open PR #13045 edits the same method without changing this line.
Searches4 searches
Nearest items, none this bug: open PR #13045 (issue #13044) adds a fillCommitData hook two lines above the line and keeps getCurrentSearchingVersion(); open PR #16838 edits getFilesToCopy and adds a TestReplicaNode class. Context, not reports: #6501 / LUCENE-5438 (the feature; replay "so no indexed documents are lost"), the java-user thread of 2025-02-21 on resetting the NRT version (replicas rejecting a lower version, the intended check), and the feature author's 2017 blog post on NRT segment replication (after a full-cluster crash, start from the latest commit and replay later operations). Limits: GitHub issue search does not index code (covered by the history check and the open-PR file lists); private trackers were not searched; no search proves absence.
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits (apache/lucene), downstream repositories | 2026-10-26 | two runs of the scripted prior-report search (VERSION_KEY, getCurrentSearchingVersion, getLastCommitVersion, finishNRTCopy, commitLock, ReplicaNode, xlog replay, symptoms) plus 41 manual GitHub searches, including nrtsearch, luceneserver, Elasticsearch, OpenSearch and Solr | no report; every query naming VERSION_KEY or getCurrentSearchingVersion returned 0 results |
| Open PRs and git history | 2026-10-26 | the changed files of all 413 open PRs; git log -S / -L / --grep over the full history | #13045, #16838, #13142 and #16847 touch ReplicaNode.java; none changes the version source |
| Mailing lists (dev@, java-user@, issues@, commits@, builds@), JIRA archive, web, CI failure mails | 2026-10-26 | 24 list, 9 JIRA and 12 web searches; 50 CI failure mails naming the NRT replication tests | no report; the CI failures are start-up, classpath and suite-timeout problems |
| GitHub refresh | 2026-10-31 | 8 issue/PR queries; the 17 open PRs created since 2026-10-25 with their files; upstream commits under lucene/replicator | no report; no new PR or commit touches lucene/replicator |
Present since the first version of ReplicaNode (LUCENE-5438, 1ae7291429ba, 2016) and first released in 6.0.0. Every release run from 6.0.0 to 10.6.0, and main, fails the same way; 6.0.0 passes with the one-line fix. Unchanged on upstream main 2f424fd428da.
Versions tested6 rows
Introduced by 1ae7291429ba ("current patch", mikemccand, 2016-01-24), the first commit of the LUCENE-5438 feature branch and the first version of ReplicaNode.java, merged into master by 12b8721a44db (2016-02-11); LUCENE-5438 "add near-real-time replication" is now #6501, with no PR and no CHANGES entry. Since then commit() changed only by the LUCENE-9570 reformat (2cbf261032d) and #378 (7d5df2d6fee). All 83 release tags from 6.0.0 to 10.5.2 in the history clone contain the line (no 8.x tags after 8.8.1 there; 8.11.4 was executed). On 6.0.0 and 6.1.0 a separate constructor-time close() bug (fixed in 6.2.0 by the backport 012fb76f7d0 of master commit de6485ea6a3) means only a ReplicaNode subclass that survives it reaches this code. Still present in 10.6.0, on branch_10_6 and on upstream main 2f424fd428da.
| Version or commit | Verdict | Result |
|---|---|---|
main 645d4fe14a20 | affected | the one test fails (gradle and harness, -ea and -da); passes with the fix |
| 10.6.0 | affected | the one test fails (-ea and -da) |
main ea57360912ed, a9779ef3c193, 10.5.2, 9.12.3 | affected | the hunt's test: VERSION_KEY 5 on a version-3 commit, getLastCommitVersion() 5 after promotion |
| 6.2.0, 6.6.6, 7.7.3, 8.11.4, 9.0.0, 10.0.0 | affected | the hunt's test fails with the same numbers |
| 6.0.0 | first release | fails with -ea and -da; passes with only the one-line fix |
| 5.5.5 | no API | no NRT replication package; the test does not compile |
An independent reproducer, a devil's advocate on intended behaviour, an adjudication (confirmed, high) and a credibility audit each re-ran tests and found it real; the devil's advocate's weakest oracle ("never ahead") fails too. A cross-model reviewer (OpenAI GPT-5.6, reading only) agreed on the root cause; one verdict was "not real" on procedure only, as it could not run code.
Checks and verdicts248 words
Each check had its own lens. The reproducer, the devil's advocate, the adjudication and the audit ran tests themselves; for the read-only cross-model reviews a separate run re-ran the hunt's test (it failed each time).
| Check | Verdict | What it found |
|---|---|---|
| independent reproducer | real | Own JUnit, written from the claim, fails 1 of 2 on main, 10.5.2 and 9.12.3 (-ea and -da) with the same numbers; its idle-commit control passes; with the fix 2 of 2 pass. |
| cross-model review, intended lens | real, medium | Read-only; root cause agrees, inputs in contract, intended argument moderate; its point that the controller re-commits before a live promotion was added to the reach. |
| cross-model review, refute lens | not real (procedural, low) | Could not run Java, so it voted not real on procedure only; on substance it agreed with the root cause. A separate run's race-versus-control test fails (control passes), and the hunt's test fails with two seeds and on two releases. |
| intended-behaviour devil's advocate | real | Built the strongest works-as-designed case and showed why it fails; own test with the weakest oracle ("never ahead of the committed version") fails 1 of 3, its two controls pass, 3 of 3 pass with the fix. |
| adjudication | confirmed, high | Re-ran the reproducer's test at a new seed (fails 1 of 2, -ea and -da); class silent wrong results. |
| credibility audit | keep | Re-ran the hunt's test at another seed (fails -ea and -da; passes with the fix). |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 474 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,498 words
- Registry title
- An NRT replica's commit() can store in VERSION_KEY the version of a newer NRT point installed during its fsync, so the commit claims a version ahead of the SegmentInfos it holds
- Area
- replicator / NRT segment replication (ReplicaNode.commit commit user data, PrimaryNode.getLastCommitVersion)
- Priority
- Low, rank 942: An experimental NRT replica commit can label itself with a newer version than it holds, and that only hurts if a primary is later cold-started on that commit, so it follows LFV-312.
- Impact class
- silent wrong results
- Impact note
- Applications that use lucene-replicator's NRT segment replication (PrimaryNode/ReplicaNode/CopyJob subclasses, opt-in and @lucene.experimental), in every release since 6.0.0 and at any index size, get a replica commit whose VERSION_KEY is ahead of its content whenever a newer NRT point is installed during that commit's fsync of the index files (the window is the whole fsync; the natural hit rate was not measured), and are harmed only if they later start a primary on that commit without re-committing it and replay their operation log from getLastCommitVersion(), as Lucene's own stress-test controller does on a cold start: the operations between the committed version and the stored one are then skipped (adds lost, deletes and updates undone, executed in a probe), while the committed index itself stays consistent and Lucene never reads the value on a replica.
- Summary
- You run Lucene's NRT segment replication and call commit() on a replica while the primary keeps publishing NRT points, and during the commit's fsync the replica installs a newer point (version 5, documents [d1, d2]). You expect the commit's VERSION_KEY to name the version whose content it holds (version 3, [d1]), or at worst an older one, because the module uses that value as the place to resume replaying operations and PrimaryNode.commit accepts only a lagging value ('replaying more ops than necessary'). Instead commit() writes the SegmentInfos it captured before the fsync (version 3) but reads VERSION_KEY from the searcher that is current after it (5), so a primary started on that commit reports getLastCommitVersion()=5 while it holds only d1, and a replay from version 5 skips d2.
- Introduced
- 1ae7291429bad742715344f86cfa5200229b3698 ('current patch' by mikemccand, 2016-01-24; the first commit of the LUCENE-5438 feature branch and the first version of ReplicaNode.java, whose commit() already captured infos under synchronized (this), ran dir.sync(indexFiles) holding only commitLock and then stored VERSION_KEY = getCurrentSearchingVersion()), merged into master by 12b8721a44db (2016-02-11; named by the author as the LUCENE-5438 commit); LUCENE-5438 'add near-real-time replication' (now #6501), no pull request, no CHANGES entry [read: git blame, git log -S, git log -L and git show in the full-history clone; exec: on 6.0.0 the repro fails and passes with only the one-line fix] (2016-01-24 (feature-branch commit); 2016-02-11 (merged into master))
- First affected version
- 6.0.0 [exec] (release commit 48c80f91b8e5, dated 2016-04-01; the first release with org.apache.lucene.replicator.nrt): the 6.0 port of the repro fails under -ea and -da and passes when only the VERSION_KEY line is changed to infos.getVersion(); on 6.0.0 and 6.1.0 the bug is reachable only through a ReplicaNode subclass that survives a separate constructor-time close() bug (success = true missing, fixed in 6.2.0 by de6485ea6a3), as the module's own SimpleReplicaNode does by accident, which is not part of this bug; 5.5.5 has no nrt package (the test does not compile) [exec]
- Affected versions
- every release from 6.0.0 through 10.6.0 and main ea57360912ed. [exec] 10.6.0: the one-test repro fails on the 10.6.0 release jars with -ea and -da (2026-10-31, also in the fact check). [exec] 6.0.0 (-ea and -da), 6.2.0, 6.6.6, 7.7.3, 8.11.4, 9.0.0, 9.12.3 (-ea and -da), 10.0.0, 10.5.2 (-ea and -da) and main (-ea and -da) all fail with the same assertion and numbers (VERSION_KEY=5 and getLastCommitVersion()=5, committed SegmentInfos version 3 with [d1]); the independent reproduction's independent test also fails on 10.5.2 and 9.12.3 (-ea and -da). [read] all 83 release tags from 6.0.0 to 10.5.2 in the history clone contain the introducing commit (the clone has no 8.x tags after 8.8.1; 8.11.4 executed), and commit() changed since only by the LUCENE-9570 reformat (2cbf261032d) and #378 (7d5df2d6fee, a null-check removal around updateGeneration). Not fixed on main (ReplicaNode.java:369) or branch_10x (line 375) [read, 2026-10-26]; no open PR changes the line. Not affected: 5.x (API absent) [exec]. All runs on aarch64 with JDK 25.0.3.
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its low-confidence “not real” is procedural), an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- formal: TLA+/TLC
- Upstream and prior reports
- none found (2026-10-26, read-only; the prior-report record, 191 recorded queries: scripted prior-report search with 16 GitHub issue/PR, 3 commit, 2 downstream, 20 mailing-list (dev@, java-user@, issues@, commits@), 4 JIRA-archive and 6 web queries, 46 hits, 0 high and 4 medium overlap, all read and unrelated; a second run with 29 lists, JIRA and web queries and 16 GitHub queries (13 by the tool before GitHub's search rate limit ran out, 3 by GraphQL); 41 manual GitHub GraphQL searches over apache/lucene and downstream users of the replicator; 24 lists.apache.org, 9 JIRA and 12 web searches; the file lists of all 413 open apache/lucene PRs; 50 CI failure mails naming TestStressNRTReplication or TestNRTReplication; the full local git history; every query naming getCurrentSearchingVersion or VERSION_KEY returned 0 results; the intended-behaviour check found nothing either); not fixed upstream (main ea57360912ed line 369 and branch_10x line 375 still read getCurrentSearchingVersion()). Related context, not reports of this bug: #6501 / LUCENE-5438 (the original feature; the author's comments tie NRT versions to operation-log replay 'so no indexed documents are lost'), open #13044 / #13045 (configurable commitData: adds a fillCommitData hook two lines above the line and leaves the version source unchanged), open #16838, #13142 and #16847 (touch ReplicaNode.java, unrelated), the java-user thread of 2025-02-21 on resetting the NRT version (replicas rejecting a lower version, the intended check), and the feature author's 2017 design post (after a full-cluster crash, start from the latest commit and replay the later operations)
- Lucene commit tested
- main
645d4fe14a20(2026-10-28): the one-test repro through gradle in a scratch worktree (assertions on and off; removed afterwards) and through the hunt repro harness (assertions on and off); the one-line fix through gradle (the repro with assertions on and off; the whole:lucene:replicator:testtask; with nightly tests enabledTestNRTReplication, 13 tests, andTestStressNRTReplication, one run). The 10.6.0 release jars with the one-test repro (assertions on and off). The hunt ran on mainea57360912edwith its two-thread test on main, 10.5.2 and 9.12.3 and history runs on 6.0.0, 6.2.0, 6.6.6, 7.7.3, 8.11.4, 9.0.0 and 10.0.0; re-checked on maina9779ef3c193. Upstream main2f424fd428da(2026-10-30) has no commit underlucene/replicatorsince645d4fe14a20. OpenJDK 25.0.3, Linux aarch64 (the only platform run).
- Finding record
lucene/replicator/src/java/org/apache/lucene/replicator/nrt/ReplicaNode.java:61 (@lucene.experimental), :340 (commitLock), :344-397 (commit: infos captured under synchronized (this) at :350-354, dir.sync(indexFiles) at :365 holding only commitLock, VERSION_KEY = getCurrentSearchingVersion() at :369, infos.commit(dir) at :373, comment at :375 'which may have changed while we were doing dir.sync above'), :399-440 (finishNRTCopy installs a new point with setCurrentInfos under synchronized (this) at :412/:440, never takes commitLock)lucene/replicator/src/java/org/apache/lucene/replicator/nrt/Node.java:42 (@lucene.experimental), :56-60 (VERSION_KEY javadoc 'Key to store the version in the commit data, which increments every time we open a new NRT reader'), :167-174 (getCurrentSearchingVersion reads the then-current searcher's reader version)lucene/replicator/src/java/org/apache/lucene/replicator/nrt/PrimaryNode.java:50 (@lucene.experimental), :104-113 (primary start keeps the commit's VERSION_KEY), :187-198 (getLastCommitVersion returns VERSION_KEY), :219-229 (primary commit; TODO at :222-224 'it's a bit wasteful that we put "last refresh" version here ... on xlog replay we are replaying more ops than necessary')lucene/replicator/src/test/org/apache/lucene/replicator/nrt/TestStressNRTReplication.java:322-336 (random async commit of a primary or replica while replication runs), :425-433 (re-commit of the replica before a live promotion), :462-489 (replay start = the operation-log location of the new primary's commit version; 'the userData in its latest commit point tells us which version it had indexed up to, so we know where to replay from in the xlog'), :970-979 (cold start as primary from a node's last commit, no re-commit)
- The finding in 30 seconds
a Lucene main checkout lucene/replicator/src/java/org/apache/lucene/replicator/nrt/ReplicaNode.java:61, 341-397, 399-440, 516-517, 530 (read)a Lucene main checkout lucene/replicator/src/java/org/apache/lucene/replicator/nrt/Node.java:42, 56-60, 167-174; PrimaryNode.java:50, 99-114, 187-198; SegmentInfosSearcherManager.java:84-101 (read)
- Who is affected
a Lucene main checkout lucene/replicator/src/test/org/apache/lucene/replicator/nrt/TestStressNRTReplication.java:276-294, 322-338, 425-434, 462-526, 702, 970-979; TestSimpleServer.java:234 (read)
- Why it looks unintended
a Lucene main checkout lucene/replicator/src/java/org/apache/lucene/replicator/nrt/PrimaryNode.java:219-229; ReplicaNode.java:341, 375-377; Node.java:56-60 (read)a Lucene main checkout lucene/replicator/src/test/org/apache/lucene/replicator/nrt/TestStressNRTReplication.java:322-338, 425-434, 456, 462-489; NodeProcess.java:77-78; TestNRTReplication.java:419-425, 463-469, 641-647, 896-903 (read)
- Suggested fix
a Lucene main checkout lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java:1031, 1073-1082, 1106-1108; lucene/replicator/src/test/org/apache/lucene/replicator/nrt/TestStressNRTReplication.java:475-484 (read)
- History
a Lucene clone: releases/lucene/10.6.0 and origin/branch_10_6 ReplicaNode.java:375; git log 645d4fe14a20..origin/main (2f424fd428da) -- lucene/replicator (empty)
- TestReplicaCommitVersionKey.java (one test):
repro/LFV-260/src/org/apache/lucene/replicator/nrt/TestReplicaCommitVersionKey.java(168 lines)
assets/LFV-260/bug.png(474 KB)