A doc-values update can be lost during a merge

The finding in 30 seconds

Sequence diagram of the race: three lifelines (indexing thread, thread A, merge thread) and a column of stored values. The merge carries doc 0's on-disk value 100 into the merged segment at delGen 5 while packet P (doc 0 = 200, delGen 4) is still running; P's update arrives later with the lower delGen, so 100 wins and 200 is lost. — The race over time: the indexing thread, thread A (which runs `flush()`) and the merge thread, steps t1 to t7, and the stored values on the right (solid = on disk, dashed = in memory). Select the still to open it at full size. Generated with AI, checked against the code.

Start forceMerge(1) of segments _0 = {doc 0, doc 1} and _1 = {doc 2}. While it runs, set doc 0's ndv to 100 and open an NRT reader: the 100 is written to _0 on disk at delGen 3.
Set doc 0 to 200 and add a delete-by-query; a flush() on another thread freezes both into packet P (delGen 4), which the test holds inside that query. Meanwhile tryUpdateDocValue sets doc 2 = 302 (delGen 5) and doc 1 = 301 (delGen 6); both finish first.
You expect the newest update to win: doc 0 = 200 on the merged segment.
Instead the merge commit gives doc 0's on-disk 100 the delGen 6 - 1 = 5. P then re-applies 200 at delGen 4, below 5, so 100 wins, with no error under -ea or -da, and the loss survives commit() and a reopen.
Without the doc-2 update the 100 gets delGen 4, a tie with P: under -ea a duplicate delGen=4 assertion closes the writer and every uncommitted change is lost; under -da the 100 won in every run observed.

Who is affected

Lucene main, whatever maxDocValuesOverlays is set to (0 included); branch_10_6 only with the experimental setMaxDocValuesOverlays(n>0), new in 10.6.0; not 10.5.2. It needs a concurrent merge, an older update to the document written during it, a still-running packet for that document, and a newer update on the same segment and field that finished first. No production rate was measured.

Mechanism, terms and numbers594 words

While a background merge runs, a newer doc-values update that is still being applied can lose to an older value carried over from disk. The merged segment keeps the old value, with no error, and the loss survives commit() and a reopen. It affects unreleased Lucene main whatever maxDocValuesOverlays is set to, and 10.6.0 (released 2026-10-29) only with the experimental doc-values overlays turned on.

Lucene applies doc-values updates in packets. Each packet gets a number, its delGen; a higher delGen means a newer update. When two updates hit the same document and field, the one with the higher delGen wins.

When a merge finishes, IndexWriter has to carry over the updates that landed on the source segments while the merge was running. Since #16570 it reads back what was already written to disk and collapses it into one packet per source segment and field. A value on disk no longer carries the number of the packet that wrote it, so the merge has to choose one:

diskDelGen = minResidual - 1, where minResidual is the smallest delGen among that segment's and field's updates that are finished but not yet written;
completedDelGen if there are none. The finished-but-unwritten updates keep their own delGens.

Main's javadoc says this number is "below all residual and still-running ones". That holds only if packets finish in number order, and they do not have to. tryUpdateDocValue, or an ordinary update packet on another indexing thread, can finish while an older packet P is still being applied. So P can end up with delGen(P) <= diskDelGen. When P re-applies on the merged segment:

delGen(P) < diskDelGen: the older carried-over value has the higher number and wins. The newer update is silently lost, with assertions on (-ea) and off (-da).
delGen(P) == diskDelGen (a tie): with -ea, ReadersAndUpdates.assertNoDupGen throws duplicate delGen, a tragic exception closes the writer, and all uncommitted changes are lost. With -da the old value won in every run observed; the order is unspecified, because DocValuesFieldUpdates.mergedIterator orders updates to one document only by delGen, so two equal delGens have no defined winner.

The example, step by step

Segments _0 = {doc 0, doc 1} and _1 = {doc 2}, numeric field ndv:

forceMerge(1) starts on a helper thread; the test pauses the merge at its first output file.
updateNumericDocValue(id:0, 100), then DirectoryReader.open(writer). Opening the reader writes the update (delGen 3) to _0 on disk while _0 is being merged.
updateNumericDocValue(id:0, 200) plus a delete-by-query. Another thread calls flush(), which freezes both into one packet P (delGen 4); P blocks inside its delete-by-query.
tryUpdateDocValue(reader, doc 2, 302) (delGen 5, on _1) and tryUpdateDocValue(reader, doc 1, 301) (delGen 6, on _0). Both finish at once, out of order, and stay pending on the merging segments.
The merge commits. On _0 the only finished-but-unwritten update is doc 1's (delGen 6), so doc 0's on-disk 100 is collapsed at delGen 5. P is released and re-applies doc 0 = 200 at delGen 4 on the merged segment. 4 is below 5, so 100 wins.
Expected doc 0 = 200; actual 100, with -ea and with -da. Without the doc-2 update, doc 1 gets delGen 5, the 100 is collapsed at delGen 4, the same number as P, and -ea closes the writer on duplicate delGen=4.

The line

IndexWriter.buildMappedDVUpdatesFromDisk (main IndexWriter.java:4620-4623, branch_10_6 :4734-4737):

final long diskDelGen =
    minResidual != Long.MAX_VALUE
        ? minResidual - 1
        : bufferedUpdatesStream.getCompletedDelGen();

Exposure and likelihood397 words

Who is exposed. Lucene main (the next major version, unreleased) runs the disk carry-over on every merge, whatever maxDocValuesOverlays is set to, 0 included (IndexWriter.java:4531 calls it unconditionally). On branch_10_6 it runs only with the opt-in, experimental IndexWriterConfig#setMaxDocValuesOverlays(n > 0), which is new in 10.6.0; with the default 0 the old buffered path runs, and this bug does not affect it. 10.5.2 is not affected: the code does not exist there.
Symptom. An update the application was told succeeded is replaced by an older value. There is no exception, the loss survives commit() and a reopen from disk, also in a separate JVM. In the tie case with -ea the writer closes instead, and every uncommitted change is lost with it.
What it takes. All four together: a concurrent merge; an older update to the document written to the merging segment during the merge (for example by an NRT reopen or a commit); a packet that updates the same document and is still being applied when the merge commits; and, on the same merging segment and field, a newer update that finished before that packet. Without the last condition the same schedule is correct.
Ordinary scale. Three or four documents are enough. An independent program reproduces it with nothing but updateNumericDocValue on two indexing threads, the default merge policy and scheduler, and FSDirectory; a pause is still needed to pin the interleaving. Without any pause, a study verifier's stress run on main (three updater threads, a slow no-match delete-by-query, a policy that merges whenever two segments exist, a 0.001 MB RAM buffer, an NRT reader opened every 2 ms) hit it in 3 of 3 30-second runs: a tragic duplicate delGen close with -ea, and 54 to 89 already-returned updates missing from NRT readers per run with -da. Pre-#16570 code and the clamp gave none; a milder configuration found nothing in 60-second runs, and no production rate was measured.
Tests do not catch it today. On branch_10_6, LuceneTestCase.newIndexWriterConfig turns overlays on in about 4 of 5 random configs, but no CI failure report was found on GitHub or builds@.
Release timing. 10.6.0 was released on 2026-10-29. On dev@ the release manager wrote that only issues with milestone 10.6 delay a release candidate and planned a 10.6.1 milestone for bug fixes; another participant wrote that 10.6 will be the final 10.x release.

Minimal repro

The test shows that during a merge, a doc-values update whose packet is still being applied loses to an older value carried over from disk, when a newer packet on the same segment and field finishes first (#16570 carry-over path, default config).

TestMergeDocValuesCarryOverRace.java (JUnit test)148 lines

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.index;

import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.NumericDocValuesField;
import org.apache.lucene.document.StringField;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.tests.util.TestUtil;
import org.apache.lucene.util.InfoStream;

public class TestMergeDocValuesCarryOverRace extends LuceneTestCase {

  public void testLostUpdate() throws Exception {
    CountDownLatch mergeWarming = new CountDownLatch(1);
    CountDownLatch releaseMerge = new CountDownLatch(1);
    CountDownLatch packetApplied = new CountDownLatch(1);
    CountDownLatch releasePacket = new CountDownLatch(1);
    AtomicReference<Thread> pauseThread = new AtomicReference<>();
    AtomicReference<MergeScheduler.MergeSource> source = new AtomicReference<>();
    IndexWriterConfig iwc =
        new IndexWriterConfig()
            .setCodec(TestUtil.getDefaultCodec())
            .setMergeScheduler(
                new MergeScheduler() {
                  @Override
                  public void merge(MergeSource mergeSource, MergeTrigger trigger) {
                    source.set(mergeSource);
                  }

                  @Override
                  public void close() {}
                })
            .setMergedSegmentWarmer(
                reader -> {
                  mergeWarming.countDown();
                  await(releaseMerge);
                })
            .setInfoStream(
                new InfoStream() {
                  @Override
                  public void message(String component, String message) {
                    if (message.startsWith("applyDocValuesUpdates")
                        && pauseThread.compareAndSet(Thread.currentThread(), null)) {
                      packetApplied.countDown();
                      await(releasePacket);
                    }
                  }

                  @Override
                  public boolean isEnabled(String component) {
                    return "BD".equals(component);
                  }

                  @Override
                  public void close() {}
                });
    try (Directory dir = new ByteBuffersDirectory();
        IndexWriter w = new IndexWriter(dir, iwc)) {
      w.addDocument(doc("k0"));
      w.addDocument(doc("k1"));
      w.flush();
      w.addDocument(doc("k1"));
      w.flush();
      w.forceMerge(1, false);
      MergePolicy.OneMerge merge = source.get().getNextMerge();
      Thread merger = new Thread(() -> run(() -> source.get().merge(merge)));
      merger.start();
      await(mergeWarming);

      w.getConfig().setRAMBufferSizeMB(0.00001);
      w.updateNumericDocValue(new Term("key", "k0"), "n", 1);
      DirectoryReader.open(w).close();
      Thread updater =
          new Thread(() -> run(() -> w.updateNumericDocValue(new Term("key", "k0"), "n", 2)));
      pauseThread.set(updater);
      updater.start();
      await(packetApplied);
      w.updateNumericDocValue(new Term("key", "none"), "n", 7);
      w.updateNumericDocValue(new Term("key", "k1"), "n", 9);
      releaseMerge.countDown();
      merger.join();
      releasePacket.countDown();
      updater.join();
      w.commit();

      try (DirectoryReader r = DirectoryReader.open(dir)) {
        int doc =
            new IndexSearcher(r).search(new TermQuery(new Term("key", "k0")), 1).scoreDocs[0].doc;
        NumericDocValues n = MultiDocValues.getNumericValues(r, "n");
        assertTrue(n.advanceExact(doc));
        // expected: the last update of the k0 doc (n=2) survives the merge; actual: the older n=1
        assertEquals(2, n.longValue());
      }
    }
  }

  private static Document doc(String key) {
    Document doc = new Document();
    doc.add(new StringField("key", key, Field.Store.NO));
    doc.add(new NumericDocValuesField("n", 0));
    return doc;
  }

  private interface IOAction {
    void run() throws IOException;
  }

  private static void run(IOAction action) {
    try {
      action.run();
    } catch (IOException e) {
      throw new UncheckedIOException(e);
    }
  }

  private static void await(CountDownLatch latch) {
    try {
      assertTrue("timed out", latch.await(30, TimeUnit.SECONDS));
    } catch (InterruptedException e) {
      throw new AssertionError(e);
    }
  }
}

Command

# in a copy of this finding's repro folder (repro/LFV-002/, which holds src/)
# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.index.TestMergeDocValuesCarryOverRace

Expected vs actual: Expected the k0 document's last update n=2 after the merge, commit and reopen; actual: the older n=1 (expected:<2> but was:<1>), with -ea and -da.

All files of this repro: repro/LFV-002/

More files, environment, full output34 words

Environment

lucene	`Lucene main e1faeba08bde (2026-11-02)`
jdk	`25.0.3+9-LTS`

Observed output

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<2> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.index.TestMergeDocValuesCarryOverRace.testLostUpdate(TestMergeDocValuesCarryOverRace.java:117)

Why it looks unintended

Main's javadoc names the rule it breaks: the number is "a gen below all residual and still-running ones so newest-wins ordering holds on the merged segment".
With -ea, the code's own ReadersAndUpdates.assertNoDupGen fires in the tie case.
Without the disk carry-over (branch_10_6 with overlays at 0, or 10.5.2 with an adapted copy) the schedule ends with the newer 200.

The full argument216 words

The code states the rule it breaks. Main's javadoc of the carry-over (IndexWriter.java:4564-4569): "the collapsed packet therefore uses minResidualDelGen - 1 (or completedDelGen when there is no residual), a gen below all residual and still-running ones so newest-wins ordering holds on the merged segment." The bug is a still-running packet that is not below that gen.
The code's own assertion trips. With -ea, ReadersAndUpdates.assertNoDupGen fires in the tie case, on main and on branch_10_6. On branch_10_6, DocValuesFieldUpdates.mergedIterator also asserts that delGens are unique (assert cmp != 0, under the comment "delGens are unique across our subs"); main's version (DocValuesFieldUpdates.java:163-168) has neither that assertion nor a tie-break.
The older path gets it right. On branch_10_6 with overlays at 0, the buffered path keeps the old 100 at its real delGen 3, below P's 4, so the newer 200 wins and the control test passes. The released 10.5.2 jars give doc 0 = 200 for the same schedule (adapted, because 10.5.2 has no overlays setting).
Newest wins is the rule the code itself names. The same javadoc chooses the number "so newest-wins ordering holds on the merged segment", and without the disk carry-over (overlays at 0 on branch_10_6, or 10.5.2) the same schedule ends with the newer value.

The devil's-advocate check102 words

No separate intended-behaviour check was run for this bug; its lenses were code-history-fix, exec-branch106, exec-main-1052 and provenance-dups-policy, plus a claim re-check. The strongest case for "intended" in those reports: on branch_10_6 the overlays feature is experimental and off by default. It does not cover main, where the path is unconditional, and it does not touch the javadoc rule above.

The claim re-check's adjudications: the buffered path is "not affected by this bug" rather than "correct" in general, and a possible separate race that the code-history checker described in the finished == false path was not reproduced and stays out of the issue.

Suggested fix

Recommended Clamp the invented delGen to completedDelGen

A candidate fix on a local branch: one expression in IndexWriter.buildMappedDVUpdatesFromDisk caps the invented delGen at completedDelGen, so it stays below every residual and every still-running packet, and at or above every number already written. Passes on branch_10_6 (1,398 index tests, 0 failures) and main (TestMergeCarryOverFromDisk 6/6); the independent program is correct in all 32 runs.

final long completedDelGen = bufferedUpdatesStream.getCompletedDelGen();
final long diskDelGen =
    minResidual != Long.MAX_VALUE
        ? Math.min(minResidual - 1, completedDelGen)
        : completedDelGen;

All options and trade-offs3 options · 379 words

Three options were on the table; the checks validated the first.

Clamp the invented delGen to completedDelGen (validated by the checks) (recommended)

Local branch r106-merge-dv-carryover-race-fix (7f12368ee2, one commit on the test commit) changes the expression in IndexWriter.java:

final long completedDelGen = bufferedUpdatesStream.getCompletedDelGen();
final long diskDelGen =
    minResidual != Long.MAX_VALUE
        ? Math.min(minResidual - 1, completedDelGen)
        : completedDelGen;

Trade-offs

Why it is safe: the new number is below every residual, below every still-running packet (a still-running packet's delGen is above completedDelGen), and still at or above every number already written, because writes use maxDelGen = completedDelGen. It changes behaviour only when minResidual - 1 > completedDelGen, which is exactly the risky case.
A main PR must also fix main's javadoc at IndexWriter.java:4564-4569, which still describes minResidualDelGen - 1. The commit's CHANGES entry has GITHUB#TBD placeholders and says "With doc-values overlays enabled", which is wrong for main.
The hunk applies to main only with git apply -3 (a _ -> versus k -> lambda in the context).
Open PR #16571 (jimczi, "Let a merge write more than one segment") edits the same method, so expect a rebase or coordination.

What the checks validated

branch_10_6 fix: the test class with -Ptests.iters=25, 200/200 pass; org.apache.lucene.index.*, 1,398 tests and 0 failures on two seeds; nine doc-values update classes, 128 tests on two more seeds; :lucene:core:check -x test passes.
main with the fix: TestMergeCarryOverFromDisk 6/6 with seeds DEADBEEF and 1, -ea and -da.
The independent program: all 32 runs correct on the fix jar (overlays default, 1, 4, 8 by v1/v2 by tie/lt by -ea/-da).

10.6.0 shipped as is; fix in 10.6.1 and main

On branch_10_6 the feature is opt-in and experimental, and 10.6.0 shipped unchanged on 2026-10-29, so the fix can be taken in 10.6.1. Main needs the fix either way, because there the path runs by default.

Trade-offs

Anyone who turns overlays on in 10.6.0 is exposed to silent lost updates until the fix ships.

Revert the backport #16591 on branch_10_6

Not clean: git revert --no-commit 4b1ed860c1 on d04ee73177 conflicts in IndexWriter.java and gives a modify/delete conflict on TestMergeCarryOverFromDisk.java, because #16581 (eaa3a7a9ed) builds on it.

Trade-offs

Larger and riskier than the clamp, and it does nothing for main.

How it was found

AI tools model-checked the merge carry-over (Alloy 6, TLA+, P), starting from a suspicion formed while reading buildMappedDVUpdatesFromDisk: not a blind find. Alloy found it first; TLA+ and P confirmed it with directed runs; TLC later re-found it without scripted actions (14-state trace, 2-document bound chosen after the fact). Randomized single-threaded testing missed it.

The method in full284 words

Method: model checking of the merge carry-over protocol, with the hypothesis formed while modelling (the study's "semi-formal" tier). On 2026-10-22, against Lucene main 7cecf79523, AI tools wrote three models of IndexWriter's NRT and merge carry-over logic from the code, one each in Alloy 6, TLA+ and P.

Alloy came first. While reading buildMappedDVUpdatesFromDisk and its comment, the AI tool suspected the diskDelGen choice and built a merge sub-model to probe it. The solver found a counterexample to the no-duplicate-delGen property in 153 s (8 steps), and its trace was simpler than the AI tool's: a published, not yet applied packet is enough.
TLA+. A blind breadth-first run produced a spurious counterexample (a modelling bug). Diagnosing it against the code led to the real hypothesis, confirmed in Java, then shown in the model with a directed witness. A later TLC run with no scripted actions, but with a 2-document bound chosen after the bug was known, found it again: a 14-state (13-step) trace after 21.1M distinct states and 33 minutes.
P. Unguided hunt of about 40,000 schedules missed it. A directed scenario with long merges and a late second indexing thread, explored with PCT, reached the bad state in 5 to 33 schedules, and the model's copy of assertNoDupGen fired ("duplicate delGen=4"). The same scenario with the clamp in the model found nothing.
Testing missed it. The randomized model-based tester runs one thread with SerialMergeScheduler, so it cannot reach the race.

Not a blind find: each model started from an AI tool's suspicion formed while reading the code. Independent AI verifiers then reproduced it (Alloy, TLA+ and P verifications), and AI tools wrote the Java tests, the standalone repro and the candidate fix.

Prior reports, history and checks

Prior reports

No earlier report found in GitHub issues and PRs (all 410 open PRs checked), review threads or the dev@, java-user@ and builds@ lists, searched 2026-10-23; no separate JIRA search is recorded. Open PR #16571 edits the same method for another purpose.

Searches5 searches

No earlier report of this race was found. The one open item that touches the same code is PR #16571, which edits the same method for another purpose.

What was searched, when, and what turned up
Where	When	How	Found
GitHub issues and PRs (apache/lucene)	2026-10-23	`gh search issues --include-prs` (titles, bodies, comments) for `buildMappedDVUpdatesFromDisk`, `diskDelGen`, `duplicate delGen`, `assertNoDupGen`, `minResidualDelGen`, `stillRunning`, `maxDocValuesOverlays`, `TestMergeCarryOverFromDisk` and more; all 95 items created since 2026-10-10 listed	no report; the method name appears only in #16570 itself, #16581 and an unrelated PR #16643; #16900 is the hunt's own, different issue
Review threads	2026-10-23	inline and review comments of #16570, #16591, #16581, #16418, #16645, #16648 scanned for still-running, out-of-order, delGen, race, tryUpdateDocValue, duplicate	one jimczi comment on #16581 about SORTED_NUMERIC in the carry-over window, a different issue
Open PRs	2026-10-23	all 410 open PRs, file lists checked for IndexWriter, ReadersAndUpdates, BufferedUpdatesStream, FrozenBufferedUpdates, DocValuesFieldUpdates	none changes the computation; #16571 (jimczi) edits the same method
Mailing lists (lists.apache.org)	2026-10-23	dev@ (120 days), java-user@, builds@ (90 days), issues@, commits@	0 hits; issues@ and commits@ show only the #16570, #16591, #16581 traffic
JIRA archive	-	no separate search is recorded for this bug; the code dates from 2026-09-01	-

History

Introduced by #16570 (9092bda09d, merged to main 2026-09-01); the backport #16591 (4b1ed860c1, inherited by branch_10_6) put it behind the overlays setting. Main fails with any overlays setting, branch_10_6 only with overlays on; 10.5.2 and the fix are correct in every run.

Versions tested5 rows

Introduced by #16570 "Stop buffering doc-values updates in heap during merges" (@jimczi), merged to main on 2026-09-01T08:22:50Z as 9092bda09d, milestone 10.6.0. The backport #16591 (@jimczi, base branch_10x, merged 2026-09-07T05:52:49Z, commit 4b1ed860c1) put the same code behind the overlays setting; branch_10_6 inherits it. The overlays feature itself is #16418 (merged 2026-08-13), backported as #16554. git blame of the root-cause lines gives 9092bda09d on main and 4b1ed860c1 on branch_10_6.

Versions and branches executed, with results
Version or commit	Verdict	Result
main `b83101cdd2`, any overlays setting (default 8, 4 and 0 run)	affected	the test branch fails 3 of 6 tests on 5 seeds with `-ea` and `-da`, the overlays=0 control included; the standalone program fails all 16 runs
`branch_10_6` `d04ee73177`, overlays 1, 4 or 8	affected	both overlay tests fail in every run (5 seeds by 10 iterations by `-ea`/`-da`); the standalone program gives 111 instead of 222 in all 24 runs: silent in 18, a tragic close in the 6 tie runs with `-ea`
`branch_10_6` `d04ee73177`, overlays 0 (default)	not affected	the control test passes; the standalone program gives 222
10.5.2 (Maven Central jars, sha1 checked)	not affected	the code is absent (`javap`, byte grep); an adapted copy of the schedule gives doc 0 = 200 in 4 of 4 runs, and the standalone program keeps the newer 222 in 8 of 8
fix `7f12368ee2` (branch_10_6) and main with the fix	not affected	all tests and all 32 standalone runs correct

Independent checks

Independent AI checkers, a lens each, and a claim re-check checked the brief: the bug reproduced on main and branch_10_6, 10.5.2 and the fix were clean, 2 claims were refuted and 1 judged not supportable, all corrected in brief v2. Study verifiers (Alloy, TLA+, P) each reproduced it.

Checks and verdicts366 words

Independent AI checkers each took a lens on the brief, and a claim re-check re-ran every disputed claim.

Each independent check and its verdict
Check	Verdict	What it found
code-history-fix	confirmed, with corrections	The step-5 arithmetic was wrong for the schedule as written (`minResidual` is per source segment and field, so the collapse is at delGen 5, not 4), and the loss is silent with `-ea` too. Fix audit: the clamp is correct. 200/200 iterations, 1,398 index-package tests.
exec-branch106	confirmed	Deterministic on 5 seeds by 10 iterations by `-ea`/`-da`. Wrote `CarryOverRace.java`: the race reproduces with ordinary packets on two threads, survives commit and a fresh JVM; fix jar 32/32 correct; 10.5.2 8/8 correct; main 16/16 wrong.
exec-main-1052	confirmed	Main fails with any overlays setting. An independent repro that dumps pending packets shows the 100 carried at delGen 5 (or 4 in the tie). 10.5.2: doc 0 = 200 in all runs. Main plus the fix: 6/6.
provenance-dups-policy	no duplicates; provenance corrected	No earlier report anywhere searched. The how-it-was-found story needed fixes: Alloy was first, the Java confirmation came before the unscripted TLC run, the trace has 14 states, and "not hinted" cannot be supported.
visual check v4	2 wrong, 3 misleading	Every thread, delGen and segment in the diagram checked out against the code and logs except two solid cells and three labels; those were changed before the final render.
Claim re-check	status ready	Re-ran the instrumented test, commit and reopen, the revert, and the logs. Two claims refuted (the step-5 numbers, the order of the Java confirmation) and one judged not supportable ("not hinted"); seven must-fix items, all applied in brief v2.
release-10_6 fact sheet	main and opt-in overlays affected	A separate fact sheet for the 10.6 release question (2026-10-23). On `branch_10_6` both overlay tests fail in 25 of 25 random iterations and the overlays=0 control passes 25 of 25; main fails 3 of 6 tests; an adapted copy of the test passes on the 10.5.2 jars (`OK (2 tests)`, five seeds).
study verifiers	new bug	Alloy verifier, TLA+ verifier and P-model verifier each reproduced it with their own test.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

PNG 344 KB

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes681 words

Finding record

LFV-002, snapshot of 2026-11-06

Registry title: Merge doc-values carry-over (#16570) can lose an acknowledged update or close the writer on a duplicate delGen
Area: index / IndexWriter merge
Priority: Medium, rank 21: A silently lost acknowledged doc-values update in the merge path, but only in a tight race that CI never hits and with the default path only in unreleased main.
Impact class: data corruption
Impact note: Main's default merge path whatever maxDocValuesOverlays is set to, and branch_10_6 only with the opt-in experimental setMaxDocValuesOverlays(n>0); needs a concurrent merge plus out-of-order update packets; ordinary scale; 10.6.0 (released 2026-10-29) has only the opt-in path, main is unreleased.
Summary: While a merge runs, a document in a merging segment gets an older doc-values update that is still being applied and a newer update to the same segment and field that finishes first. You expect the newest value to win on the merged segment. Instead the merge gives the carried-over on-disk value a delGen at or above the in-flight packet's, so the older value silently wins and survives commit and reopen, or, with assertions on and equal gens, a duplicate-delGen assertion closes the writer and discards uncommitted changes.
Introduced: #16570 (9092bda09d on main); backport #16591 (4b1ed860c1, base branch_10x, inherited by branch_10_6) (2026-09-01 (#16570 merged); 2026-09-07 (#16591 merged))
First affected version: 10.6.0 (released 2026-10-29) only with opt-in overlays; unreleased main with any overlays setting
Affected versions: main b83101cdd2: yes with any maxDocValuesOverlays, 0 included [exec]; branch_10_6 d04ee73177: only with setMaxDocValuesOverlays(n>0), new and experimental in 10.6.0 [exec]; 10.5.2: no, the code is absent [exec]
Status: documented (brief, checks and a per-finding doc exist)
Verification: Confirmed in the earlier formal verification study by independent AI checks and a rerun on Lucene main; a fix PR is open upstream.
Found by: semi-formal: Alloy, code reading; also reached by TLA+/TLC, P
Upstream and prior reports: none found (2026-10-23: gh search of issues/PRs incl. all 95 created since 2026-10-10 and all 410 open PRs; dev@ 120 d, java-user@, builds@ 90 d); open PR #16571 (jimczi) edits the same method
Lucene commit tested: main b83101cdd2 and branch_10_6 d04ee73177 (independent checks, 2026-10-23), the released 10.5.2 jars (not affected), and fix branch 7f12368ee2. The public-list rerun reproduced the repro on main b83101cdd2; the hunt's main ea57360912ed adds only #16883, which narrows LFV-004 alone and changes nothing under org/apache/lucene/index. branch_10_6 has since moved to 2763db11b0a, the #16883 backport; the root-cause lines are unchanged there.

Lucene and upstream sources

The finding in 30 seconds: lucene/core/src/java/org/apache/lucene/index/DocValuesFieldUpdates.java:163-168 (main ea57360912ed, unchanged since b83101cdd2)
Why it looks unintended: lucene/core/src/java/org/apache/lucene/index/ReadersAndUpdates.java:137-159 (main assertNoDupGen)
a Lucene main checkout: git show d04ee73177:lucene/core/src/java/org/apache/lucene/index/DocValuesFieldUpdates.java (lines 172-173); main DocValuesFieldUpdates.java:163-168

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

TestMergeDocValuesCarryOverRace.java (JUnit test): repro/LFV-002/src/org/apache/lucene/index/TestMergeDocValuesCarryOverRace.java (148 lines)

About the diagrams

The v4 sequence diagram: the indexing thread, thread A and the merge thread as lifelines, steps t1 to t7, and a column of stored values (solid = on disk, dashed = in memory).

An independent, read-only visual check (2026-10-23) of the diagram's two source scripts and the sequence still compared every label with the code and the execution logs. In the sequence diagram it found two wrong cells (the two values at t6 and t7 drawn solid, though both are pending in memory) and one misleading label (forceMerge(1) drawn on the merge thread's lifeline); its other two misleading labels were in the timeline script (forceMerge(1) in the merge-thread row, and "one delGen for all of _0's changes", though only the on-disk changes are collapsed). The sequence script was then changed (both cells solid=False, forceMerge(1) marked as a helper-thread call, and the t6 caption reworded to end "one delGen for all ndv changes written to _0 during the merge") and the files were rendered again, after the check. No later visual check is on file.

assets/LFV-002/bug-sequence.png (344 KB)

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.