All findings · Lucene finding · LFV-629 · serial-scheduler-addindexes-fails-with-other-callers-merge-exception

With SerialMergeScheduler, a valid addIndexes(CodecReader...) call can fail with a concurrent call's merge exception

With SerialMergeScheduler, two concurrent addIndexes(CodecReader...) calls share one merge queue, so a call with valid input can run the other call's merge and, when that merge fails, throw the other call's exception and delete its own finished segment. Affects 9.4.0 through 10.6.0 and main; the default ConcurrentMergeScheduler is not affected.

Summary

Priority
Low, rank 844With the non-default SerialMergeScheduler, two concurrent calls and one failing merge, the valid call fails loudly but a retry works, so it follows LFV-436.
Impact
crash
Since
9.4.0 (LUCENE-10216, #633, 2022)introduced 2022-07-06 · about 4 years
Affected
every release 9.4.0 to 10.6.0, and main
Status
documented
Found by
formal: TLA+ model checking (TLC) of the addIndexes merge queue, then a JUnit replay on the real code

AI-assisted page: this finding was first signalled by a formal tool's own output and then checked with AI tools; the checks it went through, and what each found, are under Prior reports, history and checks. Not yet triaged by a human. Snapshot of 2026-11-06.

The finding in 30 seconds

Both calls share one merge queue, and SerialMergeScheduler runs everything in it on whichever thread holds the scheduler. So B runs A's merge, and the failure is rethrown to B instead of only to A.

How addIndexes(CodecReader...) runs its merges

  1. addIndexes(CodecReader...) imports other indexes by merging them into new segments of this writer. Since 9.4.0 it does not merge inline: it puts its merges into a queue that all addIndexes calls on the writer share, and asks the writer's merge scheduler to run them.
  2. The default ConcurrentMergeScheduler runs each merge on a background merge thread. SerialMergeScheduler runs merges on the calling thread, one at a time; the thread that holds it keeps taking merges from the queue until the queue is empty.
  3. If an addIndexes merge fails, for example because a source file fails its checksum, the failure is recorded on that merge and the call that queued it throws it; the writer stays open. A failure of a regular merge (other than an abort), by contrast, closes the writer.
Diagram of the bug: two threads call addIndexes(CodecReader...) on one IndexWriter with SerialMergeScheduler. Both merges go into one shared queue. Thread B holds the scheduler and runs its own merge, then A's queued merge. A's source has one corrupt byte, so the merge-time checksum check throws CorruptIndexException on B's thread; B deletes its finished segment and throws A's exception, A throws the same exception, and 0 of the 2 documents are added.
Calls A and B share one addIndexes merge queue; SerialMergeScheduler, held by B, runs B's merge and then A's on B's thread, so A's checksum failure is thrown in B, which deletes its finished segment. Select the still to open it at full size. Generated with AI, checked against the code.

The finding, step by step

  1. Two threads share one IndexWriter configured with SerialMergeScheduler. Thread B calls addIndexes with a valid one-document index. While B's merge runs, thread A calls addIndexes with a one-document index that has one flipped byte in its stored-fields file.
  2. You expect A to throw CorruptIndexException and B to add its document. That is what happens when the two calls run one after the other, in either order, or under the default ConcurrentMergeScheduler.
  3. Instead, both calls put their merges into one queue shared by the whole writer, and SerialMergeScheduler, held by B, runs everything queued on B's thread: first B's merge, then A's. A's checksum failure is thrown in B.
  4. B throws A's CorruptIndexException: checksum failed (hardware problem?) (naming A's _0.fdt), deletes its own finished segment and adds nothing; A then throws the same exception. 0 of the 2 documents are added.
  5. Nothing is corrupted: the writer stays open, CheckIndex is clean, and retrying B adds all of B's documents (200 of 200 in a larger probe).

Who is affected

Applications that choose the non-default SerialMergeScheduler and run two addIndexes(CodecReader...) calls on one writer at once, when one call's merge fails (a corrupt source, a read error, disk full). The valid call fails with the wrong error and loses its finished merge; a retry works and nothing is corrupted. Rare.

Mechanism, terms and numbers563 words

Every addIndexes(CodecReader...) call registers its merges in one writer-wide queue and then calls mergeScheduler.merge on that shared source. SerialMergeScheduler.merge is synchronized and runs every queued merge on the thread that holds it, so a call B that is merging also runs a merge that a concurrent call A queued while waiting for the scheduler. When A's merge fails, AddIndexesMergeSource.merge records the failure on A's merge and also rethrows it to the running thread: B skips its success path, deletes its own completed merged files and throws A's exception, and A throws the same exception again from its own merge. With a natural trigger, a source with one corrupt byte that the merge-time checksum check rejects, B throws CorruptIndexException naming A's file. The failure is atomic and recoverable: the writer stays open and is not tragic, CheckIndex is clean, and a retry of B succeeds. Under the default ConcurrentMergeScheduler, and in either serial order, B succeeds. Every release from 9.4.0 (LUCENE-10216, 2022) through 10.6.0, and main, is affected.

One queue for all calls

IndexWriter has a single addIndexesMergeSource (IndexWriter.java:291) with one ArrayDeque of pending merges (:3401). Each call does (:3311-3313):

spec.merges.forEach(addIndexesMergeSource::registerMerge);
mergeScheduler.merge(addIndexesMergeSource, MergeTrigger.ADD_INDEXES);
spec.await();

The scheduler drains the queue on the entering thread

SerialMergeScheduler.java:31-40:

public synchronized void merge(MergeSource mergeSource, MergeTrigger trigger) throws IOException {
  while (true) {
    MergePolicy.OneMerge merge = mergeSource.getNextMerge();
    if (merge == null) {
      break;
    }
    mergeSource.merge(merge);
  }
}

A registers its merge before it calls mergeScheduler.merge, so while A waits for the monitor its merge is already in the queue, and B's loop takes it after B's own merge.

The failure goes to whoever runs the merge

AddIndexesMergeSource.merge (IndexWriter.java:3459-3480) ends in catch (Throwable t) { handleMergeException(t, merge); } (:3472-3473). handleMergeException (:5030-5058) records the failure on the merge (merge.setException(t); addMergeException(merge);, :5040-5041) and then rethrows it to the running thread: throw IOUtils.rethrowAlways(t); (:5056) for anything but a MergeAbortedException, and a MergeAbortedException only when the merge is external (:5043-5053; the abort of a non-external merge is swallowed).

What each call then does

  1. B leaves mergeScheduler.merge with A's exception before spec.await() (:3313), so its mergeSuccess stays false; its finally (:3316-3324) deletes the files of its own completed merge, and B throws A's exception.
  2. A gets the scheduler afterwards, finds the queue empty, returns from spec.await() and rethrows its merge's recorded exception (:3376-3384). In the probe it is the same exception object that B threw.

Why the rethrow exists

handleMergeException was written for regular merges: IndexWriter.merge (:5065) calls it (:5085), and its outer catch calls tragicEvent(t, "merge") (:5112) for whatever it rethrows (every failure except a swallowed abort), which closes the writer, so it does not matter which thread sees the exception. addIndexes(CodecReader...) merge failures are not tragic (only an Error is, :3390-3392) and belong to the one call that queued them, which already rethrows them. LUCENE-10216 (#633, 2022) routed both kinds through the same rethrow.

Under ConcurrentMergeScheduler the rethrow ends in a merge thread (ConcurrentMergeScheduler.java:747-754), and B waits only for its own merges, so B succeeds. The window under SerialMergeScheduler is the time B holds the scheduler: mainly while B's own merge runs, but also while its loop runs any other queued merges and checks the queue.

Exposure and likelihood294 words
  • Who. Applications that set SerialMergeScheduler (an Expert, non-default choice) and call addIndexes(CodecReader...) from two threads at once on one writer without serializing those calls themselves, when one of the two merges fails: a corrupt or bit-rotted source segment caught by the merge-time checksum check, a read error, or disk full.
  • Not affected. The default ConcurrentMergeScheduler; addIndexes(Directory...), which copies segments and does not use this queue (IndexWriter.java:3103); releases before 9.4.0.
  • Effect. The call whose input is valid throws an exception that describes the other call's input (its message names the other source's file) and deletes its own completed merge. An application that reacts to CorruptIndexException from addIndexes by quarantining the source it passed, or by flagging its own index, acts on the wrong input.
  • What it does not do. The failure is atomic: 0 documents of either call become visible, the writer stays open and is not tragic, CheckIndex is clean, and a retry adds exactly the valid call's documents (200 of 200 in the probe). Both calls returned within milliseconds and the probe saw no hang. As after any failed addIndexes(CodecReader...) merge, even without this bug, the partial files of the failed merge stay in the directory, unreferenced, until a writer is reopened on it.
  • Scale. Any index size; the minimal repro uses two one-document sources. Assertions on or off. The window is mainly as long as the valid call's own merge runs (plus any other queued merges its loop runs).
  • Likelihood: rare. It needs the non-default scheduler, overlapping calls and a failing merge; how often the overlap happens without the repro's forcing hooks was not measured. A maintainer once proposed removing SerialMergeScheduler as having no good use for applications, with test reproducibility as the objection (#6727, LUCENE-5665, still open).

Minimal repro

One JUnit class, one method, public API only: two one-document sources, A's with one flipped byte in its .fdt. A SerialMergeScheduler subclass whose hooks only signal starts call A during B's merge and waits until A has queued its merge, so the order is forced without sleeps. It asserts that B succeeds; it fails with assertions on and off.

TestSerialMergeSchedulerAddIndexesForeignFailure.java (one test class)116 lines
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.index;

import java.io.IOException;
import java.util.Arrays;
import java.util.concurrent.FutureTask;
import java.util.concurrent.Semaphore;
import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.StoredField;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.IOContext;
import org.apache.lucene.store.IndexInput;
import org.apache.lucene.store.IndexOutput;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.tests.util.TestUtil;

public class TestSerialMergeSchedulerAddIndexesForeignFailure extends LuceneTestCase {

  private static Directory oneDocIndex() throws IOException {
    Directory dir = new ByteBuffersDirectory();
    IndexWriterConfig iwc =
        new IndexWriterConfig().setCodec(TestUtil.getDefaultCodec()).setUseCompoundFile(false);
    try (IndexWriter w = new IndexWriter(dir, iwc)) {
      Document doc = new Document();
      doc.add(new StoredField("f", "v"));
      w.addDocument(doc);
    }
    return dir;
  }

  private static void flipLastStoredFieldsDataByte(Directory dir) throws IOException {
    String fdt = Arrays.stream(dir.listAll()).filter(n -> n.endsWith(".fdt")).findFirst().get();
    byte[] bytes;
    try (IndexInput in = dir.openInput(fdt, IOContext.READONCE)) {
      bytes = new byte[(int) in.length()];
      in.readBytes(bytes, 0, bytes.length);
    }
    bytes[bytes.length - CodecUtil.footerLength() - 1] ^= 1;
    dir.deleteFile(fdt);
    try (IndexOutput out = dir.createOutput(fdt, IOContext.DEFAULT)) {
      out.writeBytes(bytes, bytes.length);
    }
  }

  public void testValidCallDoesNotFailWithConcurrentCallsMergeException() throws Exception {
    try (Directory srcA = oneDocIndex();
        Directory srcB = oneDocIndex();
        Directory dir = new ByteBuffersDirectory()) {
      flipLastStoredFieldsDataByte(srcA);
      try (DirectoryReader readerA = DirectoryReader.open(srcA);
          DirectoryReader readerB = DirectoryReader.open(srcB)) {
        CodecReader corruptA = (CodecReader) readerA.leaves().get(0).reader();
        CodecReader validB = (CodecReader) readerB.leaves().get(0).reader();

        IndexWriter[] writer = new IndexWriter[1];
        Thread callA = new Thread(new FutureTask<>(() -> writer[0].addIndexes(corruptA)));
        Semaphore aQueuedItsMerge = new Semaphore(0);
        SerialMergeScheduler scheduler =
            new SerialMergeScheduler() {
              @Override
              public void merge(MergeSource source, MergeTrigger trigger) throws IOException {
                if (Thread.currentThread() == callA) {
                  aQueuedItsMerge.release();
                }
                super.merge(source, trigger);
              }

              @Override
              public Directory wrapForMerge(MergePolicy.OneMerge merge, Directory in) {
                if (callA.getState() == Thread.State.NEW) {
                  callA.start();
                  aQueuedItsMerge.acquireUninterruptibly();
                }
                return super.wrapForMerge(merge, in);
              }
            };

        IndexWriterConfig iwc =
            new IndexWriterConfig()
                .setCodec(TestUtil.getDefaultCodec())
                .setMergeScheduler(scheduler);
        try (IndexWriter w = new IndexWriter(dir, iwc)) {
          writer[0] = w;
          IOException failureOfB = null;
          try {
            w.addIndexes(validB);
          } catch (IOException e) {
            failureOfB = e;
          }
          callA.join();
          if (failureOfB != null) {
            // expected: B adds its document; actual: B runs A's queued merge, throws A's exception
            throw new AssertionError("expected B to succeed, it threw " + failureOfB, failureOfB);
          }
        }
      }
    }
  }
}
Command
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestSerialMergeSchedulerAddIndexesForeignFailure -Ptests.seed=DEADBEEF
Expected
B's addIndexes(validB) returns normally and adds its document; only A throws CorruptIndexException.
Observed
B throws A's CorruptIndexException: checksum failed (hardware problem?) on _0.fdt from inside SerialMergeScheduler.merge, so the test fails with AssertionError: expected B to succeed.

All files of this repro: repro/LFV-629/

More files, environment, full output408 words

Two threads, because concurrency is the bug. Natural trigger: A's source has one flipped byte, the last data byte before the footer of its stored-fields file; it opens fine and the writer's merge-time checksum check rejects it. B's source is valid.

The interleaving is forced without sleeps, polling or timeouts. wrapForMerge, which the merging thread calls inside the synchronized SerialMergeScheduler.merge during B's own merge (IndexWriter.java:3499), starts call A and waits on a semaphore until A enters merge. A does that only after it has queued its merge in the shared queue (IndexWriter.java:3311-3312). Both overrides then call the stock methods.

The codec is pinned to the default codec and compound files are off on the sources, so the .fdt is a plain file. Only B's outcome is asserted, because B's input is valid. A's own failure, the document count, CheckIndex and the retry were checked by earlier probes (see "Who is affected").

This same test is the suggested regression test; it passes the core module's format, ECJ and forbidden-apis checks as written.

Environment

lucenemain 645d4fe14a20 (11.0.0-SNAPSHOT): fails; releases 10.6.0 and 9.4.0: fail; with the fix: passes
jdkOpenJDK 25.0.3 (build 25.0.3+9-LTS)
osLinux, aarch64 (the only platform run)
Command (environment)
cp $REPRO/TestSerialMergeSchedulerAddIndexesForeignFailure.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestSerialMergeSchedulerAddIndexesForeignFailure -Ptests.seed=DEADBEEF
# assertions off fails the same way: -Ptests.asserts=false

Observed output

# gradle, scratch worktree of 645d4fe14a20, -Ptests.seed=DEADBEEF, assertions on (frames below the test left out)
TestSerialMergeSchedulerAddIndexesForeignFailure > testValidCallDoesNotFailWithConcurrentCallsMergeException FAILED
    java.lang.AssertionError: expected B to succeed, it threw org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=d8476681 actual=cf3c72c2 (resource=BufferedChecksumIndexInput((clone of) (clone of) (clone of) ByteBuffersIndexInput (file=_0.fdt, buffers=87 bytes, block size: 1, blocks: 1, position: 0)))
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:A6E3B613D9CB9317]:0)
        at org.apache.lucene.index.TestSerialMergeSchedulerAddIndexesForeignFailure.testValidCallDoesNotFailWithConcurrentCallsMergeException(TestSerialMergeSchedulerAddIndexesForeignFailure.java:110)
        Caused by:
        org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=d8476681 actual=cf3c72c2 (resource=BufferedChecksumIndexInput((clone of) (clone of) (clone of) ByteBuffersIndexInput (file=_0.fdt, buffers=87 bytes, block size: 1, blocks: 1, position: 0)))
            at app//org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:442)
            at app//org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:657)
            at app//org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:630)
            at app//org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.checkIntegrity(Lucene90CompressingStoredFieldsReader.java:824)
            at app//org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:608)
            at app//org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:252)
            at app//org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:293)
            at app//org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:123)
            at app//org.apache.lucene.index.IndexWriter.addIndexesReaderMerge(IndexWriter.java:3589)
            at app//org.apache.lucene.index.IndexWriter$AddIndexesMergeSource.merge(IndexWriter.java:3463)
            at app//org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:38)
            at app//org.apache.lucene.index.TestSerialMergeSchedulerAddIndexesForeignFailure$1.merge(TestSerialMergeSchedulerAddIndexesForeignFailure.java:82)
            at app//org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3312)
            at app//org.apache.lucene.index.TestSerialMergeSchedulerAddIndexesForeignFailure.testValidCallDoesNotFailWithConcurrentCallsMergeException(TestSerialMergeSchedulerAddIndexesForeignFailure.java:103)
1 test completed, 1 failed

# line 103 is w.addIndexes(validB) on the test thread (call B); B's source is not modified, so the corrupt _0.fdt is A's,
# merged on B's thread inside SerialMergeScheduler.merge. Same failure with -Ptests.asserts=false, with the standalone
# runner on 645d4fe14a20 (on and off), and on releases 10.6.0 and 9.4.0 (each on and off).

# with the fix: 1 test, 0 failures (assertions on and off)

Why it looks unintended

  • IndexWriter promises concurrent calls to any method, and an addIndexes merge failure already has an owner: the call that queued it throws it.
  • In either serial order, and under the default ConcurrentMergeScheduler, the valid call succeeds.
  • Rethrowing to the running thread suits regular merges, whose failures close the writer; addIndexes failures leave it open.
The full argument294 words

No javadoc explicitly promises which call receives a merge failure, so this is an argument rather than a quoted contract: the behaviour is inconsistent with how addIndexes(CodecReader...) handles failures per call, and is very likely unintended.

  • Concurrent use is promised. "IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently" (IndexWriter.java:167-168).
  • The failure already has an owner. The call that queued a merge reads its recorded exception and rethrows it (:3376-3384). The introducing PR described this design: "If the merge threads triggered for addIndexes readers fail, they throw a merge exception, which is silently recorded in that OneMerge object. After all threads join, we check if merge passed on all threads, and if not, we rethrow this MergeException, to surface the actual cause of API failure" (vigyasharma, #633 review comment r862140381, 2022-04-29; written about merge threads under ConcurrentMergeScheduler).
  • The result depends on timing and on the scheduler. Serial orders and ConcurrentMergeScheduler give the expected result; only the overlap under SerialMergeScheduler fails the valid call, which also throws away work it had finished.
  • An old statement of the contract. In LUCENE-1335 (#2412, 2008), concurrent commit, close, rollback, optimize and addIndexes calls were said to carry no promises "except that the methods all work correctly, IndexWriter won't throw an errant exception, and your index won't become corrupt" (a discussion, not javadoc; the code path fixed then no longer exists).
  • Nothing we found documents or tests the cross-call delivery. No javadoc, CHANGES entry or core test, and none of the upstream discussions found in the prior-report searches, says a call may fail with another call's merge exception. TestAddIndexes.testAddIndexesWithThreads (:1129) asserts that all concurrent calls succeed, under the default scheduler and with valid inputs only.
The devil's-advocate check247 words

The strongest case for "intended" (rated moderate by the cross-model review, OpenAI GPT-5.6; the devil's-advocate verifier called it moderate as context but weak as a defence): this is how SerialMergeScheduler works. Its javadoc says it "simply does each merge sequentially, using the current thread" and is synchronized "so that even if the application is using multiple threads, only one merge may run at a time" (SerialMergeScheduler.java:21, :28-29); for regular merges too, any thread that enters it runs, and receives the exceptions of, merges other threads queued. MergeScheduler is Expert and @lucene.experimental; upstream called the scheduler's blocking "by design ... if you want concurrency you should use CMS?" (#6374), and a maintainer proposed removing it, with test reproducibility as the objection (#6727, still open). The addIndexes all-or-nothing sentence (IndexWriter.java:3084-3086) still holds for the valid call, which adds nothing, and no javadoc says which call receives a merge failure.

Why it fails: for regular merges the receiving thread does not matter, because their failures (aborts aside) are tragic and close the writer for every caller (IndexWriter.java:5112; executed as a control: a corrupt destination segment merged by the trailing maybeMerge() closes the writer). addIndexes(CodecReader...) merge failures are deliberately per-call and non-tragic, and their owner already rethrows them, so the extra rethrow tells no caller anything new: it only fails an unrelated, valid call and deletes its finished work. "Using the current thread" says where merges run, not whose exceptions a caller gets.

Suggested fix

Recommended Do not rethrow addIndexes merge failures to the running thread (validated)

In AddIndexesMergeSource.merge, keep recording the failure on its merge but stop rethrowing non-Error exceptions to the thread that happens to run it; the owning call already throws it (+9/-1 lines). The repro then passes, as do 8 related suites and the core lint checks. Trade-off: ConcurrentMergeScheduler.handleMergeException no longer sees addIndexes merge failures.

} catch (Throwable t) {
  try {
    handleMergeException(t, merge);
  } catch (Throwable recorded) {
    // The exception is recorded on the merge (setException) and the addIndexes call that
    // owns this merge rethrows it; do not fail whichever thread happened to run the merge.
    if (recorded instanceof Error) {
      throw recorded;
    }
  }
} finally {
All options and trade-offs4 options ยท 609 words

One fix was built and validated through gradle in a scratch worktree of 645d4fe14a20. The other options are sketches, not built.

Do not rethrow addIndexes merge failures to the running thread (validated) (recommended)

In AddIndexesMergeSource.merge (IndexWriter.java:3472-3473), wrap the handleMergeException call: the exception stays recorded on its merge (setException, addMergeException), an Error is still thrown, and any other exception is not rethrown to the thread that runs the merge. The call that queued the merge throws it from merge.getException() (:3376-3384), as it already does under ConcurrentMergeScheduler.

} catch (Throwable t) {
  try {
    handleMergeException(t, merge);
  } catch (Throwable recorded) {
    // The exception is recorded on the merge (setException) and the addIndexes call that
    // owns this merge rethrows it; do not fail whichever thread happened to run the merge.
    if (recorded instanceof Error) {
      throw recorded;
    }
  }
} finally {

Trade-offs

  • Behaviour. The owning call still throws its own failure, and still reports an aborted merge through its isAborted() check (:3377-3378); an Error stays tragic (:3390-3392). Under SerialMergeScheduler the drain loop keeps going, so the valid call finishes; a single caller whose own merge fails gets the same exception as today, from the owner path.
  • ConcurrentMergeScheduler observability. Today an addIndexes merge failure also reaches ConcurrentMergeScheduler.handleMergeException in the merge thread (ConcurrentMergeScheduler.java:747-754), whose default throws MergePolicy.MergeException there (:760-762). With this fix that stops for non-Error failures, so an application that overrides the hook to watch merge failures no longer sees addIndexes ones (the calling thread still gets the exception). Option C keeps that path.
  • Side effect. By reading, it also removes the SerialMergeScheduler trigger of a sibling defect (a failed call's unstarted merges left in the shared queue), because the loop no longer abandons the remaining merges after a failure; that sibling's other triggers remain.

What the checks validated

  • Scratch worktree of 645d4fe14a20, gradle, OpenJDK 25.0.3, seed DEADBEEF. Stock: the repro fails (1 test, 1 failure).
  • With the fix, assertions on and off: the repro passes; TestAddIndexes 39 tests, TestConcurrentMergeScheduler 21, TestIndexWriter 119 (1 skipped), TestIndexWriterExceptions 40 (3 skipped), TestIndexWriterExceptions2 1, TestIndexWriterOnDiskFull 4 (including testAddIndexOnDiskFull), TestIndexWriterWithThreads 14 and TestIndexRearranger 3 (concurrent addIndexes under ConcurrentMergeScheduler): 0 failures, the same counts and skips as stock.
  • checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest of :lucene:core pass. The full test suite was not run.

Give each addIndexes call its own merge source

Each addIndexes(CodecReader...) call gets its own queue, so SerialMergeScheduler drains only that call's merges.

Trade-offs

Removes the cross-call delivery at the root and, by reading, the sibling leftover-merges defect too. A larger change: abortMerges() (IndexWriter.java:2821, :2837) aborts the one shared source on rollback, close and deleteAll, and would have to reach every live call's source. Not built.

Skip the rethrow only when another addIndexes caller runs the merge

Keep rethrowing in merge threads, but not when the running thread is a different addIndexes caller, for example by remembering on each merge which call registered it.

Trade-offs

By reading it should keep today's ConcurrentMergeScheduler behaviour, but that is untested; it needs that bookkeeping, and its abort and Error paths would have to be checked. Not built.

Workaround for users today

Use the default ConcurrentMergeScheduler, or serialize addIndexes(CodecReader...) calls on a writer that uses SerialMergeScheduler. A valid call that failed this way can simply be retried.

Trade-offs

Avoids the symptom only: it stops one call from running another call's queued merge, but each call can still fail on its own source, and SerialMergeScheduler still blocks callers while it merges. The code stays as it is.

Downstream note

Downstream, Solr's core-admin MERGEINDEXES action calls addIndexes(CodecReader...), so two concurrent merge requests on a core configured with SerialMergeScheduler (not Solr's default) could see one request fail with the other's error; Elasticsearch uses only the unaffected addIndexes(Directory...).

How it was found

TLA+ model checking (TLC) of how addIndexes(CodecReader...) queues and runs its merges. With 2 callers, SerialMergeScheduler and one failing merge, TLC found an 11-state trace in which a caller fails with the other caller's merge exception. That trace was then replayed as a JUnit test on the real code.

The method in full197 words

Method: formal (TLA+ model checking with TLC, then a replay on the stock jar).

  • Model. A TLA+ model of the addIndexes(CodecReader...) merge-source protocol: the shared pending queue, registerMerge / getNextMerge / onMergeFinished, merge states, spec.await and handleMergeException (record, then rethrow to the running thread), with a serial and a concurrent scheduler, rollback, deleteAll, a running natural merge and addIndexes(Directory...). Each action is mapped to IndexWriter.java lines (and to SerialMergeScheduler.merge()).
  • Property. The target property was "addIndexes(CodecReader...) returns normally iff every merge of its specification finished successfully"; the model split that and added the invariant NoForeignFailure: a caller never fails with another caller's merge exception.
  • Result. In the configuration with 2 callers, SerialMergeScheduler and at most 1 merge failure, TLC reported Invariant NoForeignFailure is violated with an 11-state trace after 132 generated states (83 distinct).
  • Replay and checks. The trace was replayed as a deterministic JUnit test on the stock jar. An independent reproducer, a cross-model review (OpenAI GPT-5.6) with lenses, a devil's-advocate verifier with a natural corrupt-source trigger, an adjudication, a follow-up (history, prior reports, security) and a credibility audit followed; the minimal repro was then reduced to one test on 645d4fe14a20.

Prior reports, history and checks

Prior reports

No prior report found (2026-10-27: 120 queries over GitHub issues, PRs, review comments and commits, the JIRA archive, the mailing lists, Solr, Elasticsearch and OpenSearch, plus the files of all 409 open PRs; rechecked 2026-10-31). Closest: #633, which introduced the shared queue, and #9738 on non-tragic addIndexes failures.

Searches3 searches

Verdict new: no issue, PR, review comment, commit, JIRA entry, list thread or downstream report describes one addIndexes(CodecReader...) call failing with another concurrent call's merge exception, or discusses concurrent addIndexes calls together with SerialMergeScheduler. Background, not this bug: #633 / #11253 / #1051 (LUCENE-10216, the introducing change; its review mentions SerialMergeScheduler only for test reproducibility); #2412 (LUCENE-1335, 2008, the "errant exception" statement); #6374 and #6727 (scheduler blocking by design; removal proposed); #9738 (LUCENE-8692, open: non-tragic addIndexes() failures and rollback() as the remedy; it does not say which concurrent call receives a failure). Same queue, other defects: #15863, #11684 / #1012, #14993 / #15120, #14994 / #15025, #11512. A sibling defect on the same queue (a failed call leaves its unstarted merges queued) is tracked as known because of #9738; this one keeps "new" because here the writer stays consistent and the call that fails is not the one whose input failed.

What was searched, when, and what turned up
WhereWhenHowFound
GitHub issues, PRs, review comments and commits (apache/lucene incl. the migrated JIRA archive; Elasticsearch, OpenSearch, Solr), mailing lists (dev@, java-user@, issues@, commits@, builds@), web2026-10-27scripted prior-report search runs (53 + 15 queries), 42 manual GitHub searches (class names, error and symptom phrases, LUCENE-10216, in:comments), 3 code searches, the changed files of all 409 open PRs; 23 hits readno report; closest #633, #2412, #9738
Finder and devil's-advocate verifier2026-10-26/2743 + 40 queries, and a listing of all 69 issues that mention SerialMergeSchedulerno report
GitHub issues and PRs (apache/lucene), refresh2026-10-31read-only searches of issues and PRs updated since 2026-10-26 (SerialMergeScheduler, AddIndexesMergeSource, addIndexes exception terms)nothing new

History

Introduced in 9.4.0 by LUCENE-10216 (#633, 698f40ad51af, 2022; backport 5dd8e9bdc5ae), which sent addIndexes(CodecReader...) merges through the merge scheduler via one shared queue. 9.3.0 passes; 9.4.0, 9.12.3, 10.0.0, 10.5.2, 10.6.0 and main fail. Never fixed.

Versions tested5 rows

Introduced by 698f40ad51af0c42b0a4a8321ab89968e8d0860b "LUCENE-10216: Use MergeScheduler and MergePolicy to run addIndexes(CodecReader[]) merges. (#633)" (vigyasharma, merged by mikemccand, main, 2022-07-06; issue #11253), backported to branch_9x as 5dd8e9bdc5ae72fc726a98a64bfce5119c77b558 (#1051, 2022-08-01). It added the writer-wide queue, the register-then-schedule call, the finally that deletes a call's merged files, and the routing of failures to handleMergeException, whose record-and-rethrow dates from 2008 (2befbd45b79f, LUCENE-1397; its current throw IOUtils.rethrowAlways(t) form is from e52d86099493, 2017, LUCENE-7796). SerialMergeScheduler dates from 2007 (299d6357dd2), and its loop is unchanged since 2020.

Before 9.4.0, addIndexes(CodecReader...) merged inline on the calling thread and never gave its merge to the scheduler. 5dd8e9bdc5ae is the only IndexWriter.java commit between 9.3.0 and 9.4.0, and the result flips there. No commit-level bisect on main was run (a 2022 checkout needs that period's toolchain). The releases between the executed ones were checked by git tag --contains and a static scan of every 9.x/10.x tag.

Versions and branches executed, with results
Version or commitVerdictResult
main 645d4fe14a20; 10.6.0 affectedminimal repro fails with assertions on and off
9.4.0 first releaseminimal repro (assertions on and off) and the version-neutral history test fail (history test 20 of 20)
9.12.3, 10.0.0, 10.5.2, main a9779ef3c193 affectedhistory test fails with assertions on and off
9.3.0 not affectedhistory test passes 20 of 20: the merge runs inline on the calling thread
upstream main 2f424fd428da, branch_11x readblob-identical IndexWriter.java and SerialMergeScheduler.java

Independent checks

The finder, an independent reproducer, a cross-model reviewer (OpenAI GPT-5.6, lenses), a devil's-advocate verifier, an adjudication and a credibility audit each reproduced it with their own test and called it real. This doc's gradle run reproduced it again on 645d4fe14a20 and validated the fix.

Checks and verdicts281 words

Every executed check ran on stock builds; the fix checks shadowed or applied the same +9/-1 change.

Each independent check and its verdict
CheckVerdictWhat it found
Finder (TLA+ model, JUnit replay) candidateTLC violated NoForeignFailure (11-state trace); the JUnit replay fails on main with assertions on and off, on 10.5.2 and 9.4.0, and passes with the fix.
Independent reproducer realOwn public-API test (a FilterDirectory read error on A's source): 20 of 20 fail with assertions on and with them off; also 10.5.2, 9.12.3, 9.4.0; 9.3.0 not affected; records that A's source was read on B's thread; both controls pass; passes with the fix.
Cross-model review, refute and intended lenses (OpenAI GPT-5.6) real, highEach lens wrote and ran its own test, which fails; intended-behaviour argument rated moderate; agreed with the root cause and the fix.
Devil's advocate realNatural trigger (one corrupt byte): fails on main (20 of 20 with assertions on; also with them off), 10.5.2 and 9.4.0 (one run each); both serial orders, ConcurrentMergeScheduler and a single caller behave as expected; a retry of B adds 200 documents.
Adjudication confirmed, highRe-ran the independent reproduction (on and off) and the intended-behaviour check (off) with a new seed, and the independent reproduction with the fix (passes).
Follow-up and credibility audit keepHistory bracket 9.3.0 / 9.4.0; prior-art verdict new; security probe: atomic and recoverable; the audit re-ran three tests on main with fresh seeds.
This doc (2026-10-31) reproduced, fix validatedGradle on 645d4fe14a20: the repro fails on stock and passes with the fix, 8 related suites and the lint checks pass; the repro also fails on 10.6.0 and 9.4.0.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,453 words

Finding record

LFV-629, snapshot of 2026-11-06

Registry title
With SerialMergeScheduler, a valid addIndexes(CodecReader...) call can run a concurrent call's queued merge on its own thread and fail with that merge's exception, discarding its own completed merge
Area
index / IndexWriter addIndexes(CodecReader...) (writer-wide AddIndexesMergeSource queue, SerialMergeScheduler)
Priority
Low, rank 844: With the non-default SerialMergeScheduler, two concurrent calls and one failing merge, the valid call fails loudly but a retry works, so it follows LFV-436.
Impact class
crash
Impact note
Hits only applications that configure the non-default SerialMergeScheduler and run two addIndexes(CodecReader...) calls on one IndexWriter at the same time, and only when one call's merge fails (a corrupt or bit-rotted source segment that the merge-time checksum check rejects, a read error, disk full): the other call, whose input is valid and whose own merge already completed, then throws the failing call's exception, deletes its merged files and adds nothing; 'crash' means a legal API call throws, not a JVM crash; the failure is atomic and recoverable (the writer stays open and not tragic, CheckIndex is clean, a retry adds exactly the valid call's documents), the default ConcurrentMergeScheduler is not affected, any index size, every release 9.4.0-10.6.0 and main, -ea and -da alike [exec].
Summary
Two threads call addIndexes(CodecReader...) on one IndexWriter configured with SerialMergeScheduler; call A's source is corrupt (or hits a read error) and call B's source is valid. You expect A to fail with its own exception and B to add its documents, as happens in either serial order and with the default ConcurrentMergeScheduler, because IndexWriter is documented as safe for concurrent calls to any method and each addIndexes call already rethrows the failures of its own merges. Instead B, which holds the scheduler, runs A's queued merge on its own thread after finishing its own, receives A's exception (for example CorruptIndexException 'checksum failed (hardware problem?)' naming A's source file), deletes its own completed merge and adds nothing, while A throws the same exception.
Introduced
698f40ad51af0c42b0a4a8321ab89968e8d0860b on main ('LUCENE-10216: Use MergeScheduler and MergePolicy to run addIndexes(CodecReader[]) merges. (#633)', by vigyasharma, merged by mikemccand; issue #11253) and its branch_9x backport 5dd8e9bdc5ae72fc726a98a64bfce5119c77b558 (#1051). It added the writer-wide AddIndexesMergeSource queue, the per-call registerMerge plus mergeScheduler.merge (IndexWriter.java:3311-3312 on a9779ef3c193) with the finally block that deletes the call's own merged files on failure (:3316-3324), and AddIndexesMergeSource.merge routing failures to handleMergeException (:3472-3473), whose older rethrow (:5056, from e52d86099493, 2017) delivers them to whichever thread runs the merge; SerialMergeScheduler's synchronized drain loop (SerialMergeScheduler.java:31-40) is older and unchanged in every 9.x/10.x release [read: git log -G and blame in the history clone; exec: release bracket, 9.3.0 passes and 9.4.0 fails, and 5dd8e9bdc5ae is the only IndexWriter.java commit between the two tags; no commit-level bracket on main was executed] (2022-07-06 on main (698f40ad51af); 2022-08-01 on branch_9x (5dd8e9bdc5ae))
First affected version
9.4.0 [exec: the version-neutral history test fails under -ea and -da, 20 of 20 with --iters 20]; 9.3.0 and older are not affected [exec 9.3.0: passes, 20 of 20; read: addIndexes(CodecReader...) merged inline on the caller thread and never used the merge scheduler]
Affected versions
every release from 9.4.0 through 10.6.0 and main a9779ef3c193 and 645d4fe14a20; not fixed upstream. [exec] the minimal test fails on the 10.6.0 release jars and on main 645d4fe14a20, -ea and -da (2026-10-31); the follow-up's version-neutral public-API history test fails on 9.4.0, 9.12.3, 10.0.0, 10.5.2 and main under -ea and -da (its both-valid control passes everywhere); the independent reproduction's test fails on 10.5.2 (-ea and -da), 9.12.3 and 9.4.0; the intended-behaviour check's natural-trigger probe fails on 10.5.2 and 9.4.0; the finder's repro fails on 9.4.0, 9.12.3, 10.0.0 and 10.5.2; the security probe reproduces the defect on 10.5.2. [read] the releases in between (9.4.1-9.12.2, 10.1.0-10.5.1) by git tag --contains and a static scan of every tag (shared queue present, merge() calls handleMergeException, handleMergeException rethrows, SerialMergeScheduler.java identical); branch_10_6 (10.6.0, released 2026-10-29) and the unreleased branch_10x carry the same code; no commit on upstream main (head a95c754ee003, 2026-10-26) or branch_10x after a9779ef3c193 touches IndexWriter.java or SerialMergeScheduler.java. Not affected: 9.3.0 and older [exec 9.3.0].
Status
documented (brief, checks and a per-finding doc exist)
Verification
Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by
formal: TLA+/TLC
Upstream and prior reports
none found (2026-10-27, read-only; the prior-report record verdict new, 120 queries, 23 hits read, all same_bug false): scripted prior-report search follow-up run (16 apache/lucene issue/PR queries, 3 commit queries, 4 downstream queries over Elasticsearch, OpenSearch and Solr, 20 queries over the dev@, java-user@, issues@ and commits@ lists, 4 JIRA-archive queries, 6 web queries; 1458 distinct hits, 7 high-overlap, none this bug), a builds@/dev@/issues@ run (15 queries; the only matching CI mails predate 2022), 42 manual GitHub searches (class names, error and symptom phrases, LUCENE-10216, in:comments), the touched files of all 409 open apache/lucene PRs (20 touch IndexWriter, MergePolicy, ConcurrentMergeScheduler or TestAddIndexes; none changes AddIndexesMergeSource.merge, the addIndexes rethrow or finally path, or SerialMergeScheduler), upstream main and branch_10x commits; on top of the finder's 43-query run and the intended-behaviour check's 40-query run plus its listing of all 69 issues that mention SerialMergeScheduler. Related, not this bug: #633 / #11253 / #1051 (LUCENE-10216, the introducing change; its review states the owning-call rethrow design and mentions SerialMergeScheduler only for test reproducibility; nobody discusses concurrent calls sharing the queue); #2412 (LUCENE-1335, 2008: concurrent addIndexes calls must not 'throw an errant exception', fixed then by serializing addIndexes, a code path that no longer exists); #6374 (LUCENE-5310, open: SerialMergeScheduler blocking is 'by design') and #6727 (LUCENE-5665, open: proposal to remove SerialMergeScheduler, kept for reproducible tests); #9738 (LUCENE-8692, open: non-tragic addIndexes() failures, rollback() as the remedy; no word on which concurrent call receives a merge failure); same queue, different defects: #15863 (LFV-184), #11684 / #1012, #14993 / #15120, #14994 / #15025, #11512 (one merge per reader, would widen the window), #13116 (relevant only to the fix caveat); downstream OpenSearch #21422 configures SerialMergeScheduler on a writer that receives addIndexes during refresh, under a refresh lock per its description (a use of the same pieces, not a report; exposure not checked).
Lucene commit tested
main 645d4fe14a20 (2026-10-28): the minimal test fails through gradle in a scratch worktree and through a standalone runner with assertions on and off; with the recommended fix it passes, as do TestAddIndexes, TestConcurrentMergeScheduler, TestIndexWriter, TestIndexWriterExceptions, TestIndexWriterExceptions2, TestIndexWriterOnDiskFull, TestIndexWriterWithThreads and TestIndexRearranger (assertions on and off; the same counts and skips as stock) and the core module's format, ECJ and forbidden-apis checks. Releases 10.6.0 and 9.4.0 fail with the same test (assertions on and off); the version-neutral history test fails on 9.4.0, 9.12.3, 10.0.0, 10.5.2 and main a9779ef3c193, and passes on 9.3.0. Upstream main 2f424fd428da (2026-10-30) has blob-identical IndexWriter.java and SerialMergeScheduler.java [read].

Lucene and upstream sources

Finding record
  • lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:167-168 (completely thread safe, any method concurrently), :3084-3086 (addIndexes transactional sentence, referenced by the CodecReader overload at :3259-3260), :3311-3312 (registerMerge, then mergeScheduler.merge with the writer-wide addIndexesMergeSource), :3313-3315 (spec.await, mergeSuccess), :3316-3324 (finally: delete this call's merged files when mergeSuccess is false), :3376-3384 (owning call rethrows merge.getException()), :3390-3392 (catch Error -> tragicEvent), :3459-3480 (AddIndexesMergeSource.merge: catch Throwable -> handleMergeException), :5030-5058 (handleMergeException: setException, addMergeException, rethrow non-abort exceptions at :5056)
  • lucene/core/src/java/org/apache/lucene/index/SerialMergeScheduler.java:21 ('does each merge sequentially, using the current thread'), :28-29 (synchronized so only one merge runs at a time), :31-40 (synchronized loop over getNextMerge()/mergeSource.merge(merge), no catch)
The finding in 30 seconds
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:291, :3311-3313, :3316-3324, :3376-3384, :3390-3392, :3401, :3459-3480 (:3472-3473), :5030-5058 (:5040-5041, :5043-5053, :5056), :5065, :5085, :5112
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/SerialMergeScheduler.java:21, :28-29, :31-40
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java:747-754, :760-762
  • a Lucene clone: git rev-parse of IndexWriter.java and SerialMergeScheduler.java (blob-identical at a9779ef3c193 and 645d4fe14a20)
Who is affected
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:3103 (addIndexes(Directory...))
Minimal repro
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:3311-3312, :3499 (wrapForMerge called by the merging thread)
Why it looks unintended
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:167-168, :3084-3086, :3376-3384, :5112
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/SerialMergeScheduler.java:21, :28-29
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/MergeScheduler.java:30-34 (Expert, @lucene.experimental)
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/test/org/apache/lucene/index/TestAddIndexes.java:975-999, :1129-1150 (testAddIndexesWithThreads, default scheduler)
Suggested fix
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java:747-754, :760-762
  • a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2821, :2837, :3376-3384, :3390-3392, :3472-3473
History
  • a Lucene clone: git rev-parse origin/main (2f424fd428da, 2026-10-30), origin/branch_11x, releases/lucene/10.6.0 for IndexWriter.java and SerialMergeScheduler.java blobs

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

About the diagrams

Made in parallel with this doc from the code at 645d4fe14a20 and the minimal repro's output.

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.