addIndexes returns a sequence number that contradicts concurrent deletes, updates and commits

Call addIndexes while another thread deletes or updates a document whose term also matches an incoming document. The delete returns sequence number 5 and addIndexes returns 6, so by the IndexWriter contract the delete came first and must not touch the added document. It does: in the repro the added document (val=100) is gone at the next refresh, live values [] instead of [100]. With updateDocument the added document is lost, with a doc-values update it is overwritten, and a commit() that returns 7 can contain an addIndexes that returns 8. No error is raised. Default settings suffice (executed with a plain new IndexWriterConfig(), the default codec and FSDirectory); it needs concurrent calls, and every release since sequence numbers were added (6.2.0) has it.

IndexWriter#addIndexes copies segments from other indexes (addIndexes(Directory...)) or merges readers (addIndexes(CodecReader...)) into a live writer. IndexWriter is documented as "completely thread safe", so other threads may keep deleting, updating and committing meanwhile.

Every call that changes the index returns a sequence number (seqNo). The class javadoc says it "expresses the effective order in which each change was applied", and commit() promises that "all sequence numbers <= this value will be reflected in the commit, and all others will not". So two overlapping calls still get a definite order: the smaller number came first.

The writer w holds one committed document id:k, val=1; a second index src holds one document id:k, val=100.

What you get: []. The delete also removed the document that, by the sequence numbers, was added after it. Run the two calls one after the other and you get [100].

The same window with other calls (all with seqNo 5 against addIndexes 6): updateDocument(id:k, val=2) gives [2] instead of [2, 100] (the added document is lost); updateNumericDocValue(id:k, "val", 7) gives [7, 7] instead of [7, 100] (the added document is rewritten); addIndexes(CodecReader...) gives [] too. And a commit() that runs while addIndexes is copying returns 7 while addIndexes returns 8, yet the commit point holds [1, 2, 100]: it contains a change whose sequence number says it is not in it.

Inside the writer, deletes do not follow sequence numbers. A delete by term waits in a buffer and is later frozen into a packet with a delete generation; a packet applies to every segment whose own generation is not newer. addIndexes takes its place in the first order but never in the second:

1. It flushes only at the start. addIndexes begins with flush(false, true) (IndexWriter.java:3119), which keeps deletes buffered before the call off the incoming documents (the 2011 fix LUCENE-2996). Then it copies the files without holding the IndexWriter monitor (:3121-3192; copySegmentAsIs, :3648-3705). The copies go into a local list, not yet into the writer's segment list segmentInfos (:3183).
2. The delete during the copy waits in the buffer. deleteDocuments appends it to the delete queue and returns seqNo 5 (:1864-1867 -> DocumentsWriterDeleteQueue.java:129-133). Nothing is applied yet.
3. addIndexes publishes with a later seqNo but no generation. Under the IndexWriter monitor only, it takes docWriter.getNextSequenceNumber() = 6 and appends the segments (:3194-3211; CodecReader :3350-3369). The new segments keep bufferedDeletesGen = -1, the default (SegmentCommitInfo.java:88); only flushed segments (IndexWriter.java:2989) and merged segments (:4554) get one, so the added segments keep -1 until a merge replaces them.
4. At the next refresh the delete reaches the added segment. The buffered delete is frozen into a packet with the next generation (BufferedUpdatesStream.java:75), and a packet applies to every segment with bufferedDeletesGen <= delGen (IndexWriter.java:6784; the skip test for term deletes is FrozenBufferedUpdates.java:460). In a trace of the repro the old segment has generation 1, the added segment -1, and the delete packet 2: it applies to both.
5. Commit cuts at a full flush that addIndexes ignores. prepareCommit holds fullFlushLock from the full flush that fixes its seqNo (:3806-3809) until it clones the segment list (:3850). addIndexes publishes without taking that lock (only its up-front flush holds it, :4400), so it can publish in between: its seqNo (8) comes after the cut (7), its segments are in the clone.

Sequence numbers say "delete, then addIndexes"; delete generations say "addIndexes, then delete"; the generations decide. The index stays consistent with that second order, so nothing looks broken, except that it contradicts the numbers IndexWriter returned.

• Who is hit. Applications that call addIndexes on a live writer while other threads delete or update documents whose terms can match incoming documents, or while another thread commits, and that rely on sequence numbers. LUCENE-7302 added them for two stated uses: telling "which update 'won'" without per-id locking, and crash recovery that replays "only the truly missing operations" after a commit's sequence number. Here the seqNos say addIndexes (6) came after updateDocument (5) while the index kept the update; and a replay of everything after commit 7 re-runs addIndexes (8), whose documents are already in that commit, so they are added twice (reasoning from the code and the stated use, not executed).
• Default settings. Default settings suffice: a plain main() program without the test framework, with new IndexWriterConfig() (only an InfoStream added for the hooks; default codec, TieredMergePolicy, ConcurrentMergeScheduler, commitOnClose=true) and FSDirectory, shows all three violations with assertions on and off, and passes under fix B. Both public overloads, public API only. The JUnit repro sets NoMergePolicy and SerialMergeScheduler only to make its thread hooks deterministic; the study's stress harness hit it under randomized configurations.
• It needs concurrency. An addIndexes overlapping a matching delete or update, or a commit. The window is the whole copy or merge, so it grows with the size of the incoming index. Under plain stress the study's harness hit it in 9 of 100 seeds in 10 s.
• Silent and durable. No exception, no assertion under -ea; the next commit persists the deletion or overwrite. The index is internally consistent, it just follows the opposite order.
• Pattern in use. Merging per-group writers into a live writer with addIndexes is discussed upstream (#13387) and in a downstream design (OpenSearch#18576 / #19530) that applies deletes around that step. Whether that code is exposed was not checked.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: live docs after delete seqNo 2, addIndexes seqNo 3 expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.index.TestAddIndexesConcurrentDeleteSeqNo.testDeleteWithSmallerSeqNoDoesNotDeleteAddedDocument(TestAddIndexesConcurrentDeleteSeqNo.java:75)

• The ordering contract. IndexWriter class javadoc (IndexWriter.java:117-120): "Each method that changes the index returns a long sequence number, which expresses the effective order in which each change was applied. commit also returns a sequence number, describing which changes are in the commit point and which are not." Both addIndexes overloads: "@return The sequence number for this operation", linking there (:3097, :3269). CHANGES.txt:8925-8927 (LUCENE-7302): the numbers indicate "the effective equivalent single-threaded execution order".
• The commit contract. commit() (:4203-4210): "all indexing operations prior to this sequence will be included in the commit point, and all other operations will not"; "All sequence numbers <= this value will be reflected in the commit, and all others will not" (also prepareCommit(), :3730-3732).
• Concurrency is allowed. "IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently" (:167-168). No javadoc exempts addIndexes.
• The sequential form was fixed as a bug. LUCENE-2996 (#4070, 2011): "addIndexes applies the previous deletes on the incoming indexes as well, which is incorrect"; fixed by the up-front flush (CHANGES.txt:17452-17454). The concurrent window remained, and since LUCENE-7302 (6.2.0) the returned numbers define which delete is "previous".
• No upstream test covers it. TestIndexingSequenceNumbers never calls addIndexes; the concurrent TestAddIndexes cases (CommitAndAddIndexes*, TestAddIndexes.java:1065 ff.) check exceptions and document counts.

Strongest case for "intended": in LUCENE-1335 (#2412, 2008) @mikemccand wrote "I think we should not make any API promises about what it means when these methods (commit, close, rollback, optimize, addIndexes) are called concurrently from different threads, except that the methods all work correctly, IndexWriter won't throw an errant exception, and your index won't become corrupt", and @yonik agreed: "higher level synchronization by the user is the right way to ensure/enforce an ordering." On dev@ in February 2022 the LUCENE-10216 author (@vigyasharma) noted that "we may still get such a delete before the API completes, and those deletes would get applied, this is likely a best effort scenario, right?", and @msokolov replied "I think your analysis is correct, but I wasn't around for that so I am just reading the historical record". The NRT javadoc also lets a reader see only the foreign segments "successfully copied over, so far" while an addIndexes runs (IndexWriter.java:504-505), which fits a best-effort reading. The index is never corrupt, and addIndexes is usually a bulk step without concurrent writes.

Why it fails: the sequence-number javadoc was written in 2016, eight years after LUCENE-1335, and promises the effective order for every index-changing method, addIndexes included, with no exception; commit() promises that larger numbers are not in the commit. The 2008 caveat is not in today's docs, and it was about concurrent admin calls, not ordinary deletes and updates. If callers must order addIndexes themselves, the number it returns says nothing about its order relative to overlapping calls. Partial NRT visibility is about what a reader sees mid-call; it does not let a commit contain an operation with a larger seqNo. The 2022 exchange was a list question during a refactor, without the sequence-number framing, and was never documented or filed; it does show maintainers may accept a documentation fix. An independent reviewer from another model family (OpenAI GPT-5.6) weighed the LUCENE-1335 counterpoint it was given (it did not know the 2022 thread) and rated it moderate; a second cross-model fact check rated the intended case moderate for the delete/update part and weak-to-moderate for the commit part. Verdict: a contract violation as the docs stand; code fix or javadoc fix is the maintainers' call.

The root cause is that addIndexes takes a place in the sequence-number order without taking one in the delete-generation order or in the commit cut. A code fix publishes the added segments at a full-flush boundary, the way prepareCommit cuts a commit point: freeze the buffered deletes first (those in the current delete queue, all with smaller seqNos), give the added segments the next generation, return the boundary as the seqNo, and hold fullFlushLock so commit and NRT reopen see the segments on the right side of their cut. Options B and A were built and validated in a scratch worktree at ea57360912ed (removed afterwards), C was tested with the harness, D and E are untested ideas. A javadoc fix (E) is a legitimate alternative given LUCENE-1335 and the 2022 thread.

Both overloads drop their up-front flush(false, true) and publish through one new method, publishAddedSegments, which runs the full flush at the end of the call, under fullFlushLock. The added segments get a generation through a new SegmentCommitInfo#setBufferedDeletesGenKeepId, because the existing setter also assigns a new commit id and addIndexes(Directory...) must keep the source segment's id (TestIndexWriter#testSegmentCommitInfoId). Dropping the up-front flush is safe because its documented job (LUCENE-2996, deletes buffered before the call) is now done by the publication flush, which freezes those deletes before the added segments get their later generation. +84/-49 lines in IndexWriter.java and SegmentCommitInfo.java. AI tool-designed; no maintainer has reviewed it.

private long publishAddedSegments(List<SegmentCommitInfo> infos, long totalDocs) throws IOException {
  doBeforeFlush();
  synchronized (fullFlushLock) {
    long seqNo = docWriter.flushAllThreads();   // freezes the current delete queue (all smaller seqNos)
    if (seqNo < 0) seqNo = -seqNo;
    publishFlushedSegments(true);               // ... and gives them their delete generation
    synchronized (this) {
      ensureOpen();
      reserveDocs(totalDocs);
      long gen = bufferedUpdatesStream.getNextGen();
      bufferedUpdatesStream.finishedSegment(gen);
      for (SegmentCommitInfo info : infos) info.setBufferedDeletesGenKeepId(gen);
      segmentInfos.addAll(infos);
      checkpoint();
    }
    return seqNo;                               // the boundary, as prepareCommit returns
  } // finally: finishFullFlush, processEvents(false), doAfterFlush; on failure delete the copied files
}

Trade-offs

• Same number of full flushes as today: one per call, moved from the start to the end (counted from the writer's InfoStream: stock 1, B 1, A 2). Documents buffered before the call or indexed during the copy are flushed just before the added segments are appended, so they get smaller doc IDs, in line with their smaller seqNos.
• The flush no longer applies all deletes (the old flush(false, true) did); they are applied lazily, as after any full flush.
• addIndexes now waits for fullFlushLock at the end, so it waits for a concurrent commit's or NRT reopen's flush window, and they wait for its publication: the point of the fix, but new blocking.
• The doBeforeFlush / doAfterFlush hooks now run around the publication instead of at the start, and doAfterFlush runs while fullFlushLock is still held (stock doFlush calls it after releasing the lock); a subclass hook that blocks or throws there was not tested. Two of the cross-model reviewer's test classes pause addIndexes in doAfterFlush; under B that pause comes after the publication, so their delete gets the larger seqNo and their precondition fails (under A they pass). They are not regression tests for B.
• The reviewer's third class has a commit test that waits for addIndexes while the commit is paused inside its flush. Any fix that orders addIndexes after the in-progress commit turns that wait into a 30 s timeout that closes the writer: it fails that way under A (the reviewer's own run) and under B; its delete test and controls pass under B.
• A small cost probe, 3 reps per option: 200 rounds of add + delete + addIndexes on one thread give the same 400 segments for stock, A and B (stock 1197/608/567 ms, A 1634/918/559 ms, B 1397/705/457 ms: too few reps to compare times); with a concurrent indexer the flushed-segment counts varied from rep to rep (stock 186/75/96, A 137/190/188, B 180/84/90), too much to rank the options. Not a benchmark.

What the checks validated

• The repro passes 4/4 with assertions on and off; the study verifier's 9-test class passes 9/9 (its seqNos shift from stock, e.g. delete 4 < addIndexes 5 and commit 6 < 7, and the states now match them); the default-settings program shows 0 violations; -Ptests.iters=5 of the repro, TestAddIndexes and TestIndexingSequenceNumbers: 255 tests, 0 failures.
• All of :lucene:core:test: 748 classes, 8,869 tests, 0 failures, 322 skipped (one seed).
• With -Ptests.asserts=false: the repro and 12 IndexWriter / addIndexes / doc-values-update suites, 412 tests, 0 failures.
• :lucene:misc TestBPReorderingMergePolicy, :lucene:benchmark TestAddIndexesTask and all of :lucene:backward-codecs (2,339 tests): 0 failures.
• :lucene:core:check -x test passes (google-java-format, ecjLint, forbiddenApis) after tidy re-wrapped two comment lines of the new test.
• Not done: neither fix was run through the study's randomized stress harness or Fray.

The same publishAddedSegments idea, but the up-front flush(false, true) stays (the study verifier's verification-only patch, comments marked "FV-VERIFY"). +63/-39 lines.

Trade-offs

Two full flushes per call (measured). #14994 / PR #15025 (2025) removed per-merge full flushes from addIndexes(CodecReader...) because they cost too much, so expect the same objection. The smallest conceptual change.

What the checks validated

• The repro 4/4; all of :lucene:core:test: 748 classes, 8,869 tests, 0 failures, 322 skipped.
• Earlier: the study verifier's 9 tests on 5 seeds and 384 tests in 10 existing classes; the cross-model reviewer's TestAddIndexes 39/39 and TestIndexingSequenceNumbers 4/4.

Wrap both publication blocks in synchronized (fullFlushLock). No extra flush; +19/-13 lines.

Trade-offs

Fixes the commit part only: with the harness the commit test passes and both delete tests still fail ([] instead of [100]).

Freeze the global delete buffer instead of running a full flush, then give the added segments the next generation.

Trade-offs

Cheaper (no segment flush), but the freeze and the seqNo must be atomic with respect to concurrent deletes (DocumentsWriterDeleteQueue.add synchronizes on the queue, DocumentsWriterDeleteQueue.java:174), it must cooperate with a full flush already in progress, and it still needs fullFlushLock for commit.

Say in both addIndexes javadocs, and in the class-level sequence-number paragraph, that deletes and updates running concurrently with addIndexes may also apply to the added documents and that a concurrent commit may contain them, whatever their sequence numbers.

Trade-offs

Matches LUCENE-1335 and the 2022 "best effort" reading; leaves the returned seqNo without ordering meaning for calls that overlap addIndexes. The cheapest change.

Method: multi-threaded stress testing with an exact oracle (the study's non-formal baseline), re-found by controlled-concurrency testing.

• A 2026-10-22 study of Lucene by AI tools covered the IndexWriter/NRT layer because an earlier bug there (now #16900) was the study's starting point. So the area was hinted; this bug was not (the study tags it a blind find).
• An AI tool wrote an NRT harness: 2-3 writer threads and 1-2 reader threads over randomized writer configurations, checked by an oracle built only from API contracts: linearizability of what readers saw, and a sequence-number check (replaying the operations in seqNo order must give the final state; each commit point must equal the operations with seqNo <= the commit's). With addIndexes added to the operation mix, plain stress flagged seqNo-order and commit-point violations in 9 of 100 seeds at 20 runs each (10 s), 16 of 100 at 200 runs each.
• The controlled-concurrency tester Fray re-found it (PCT 14, POS 13, Random 5 of 100 seeds at 20 iterations each). None of this is formal verification.
• An independent AI verifier wrote its own deterministic test (9 tests), swept six releases and confirmed the root cause with a patch. This hunt minimised the test, re-verified it on ea57360912ed, had it reviewed adversarially by OpenAI GPT-5.6, pinned the history across 89 releases and the introducing commits, searched for prior reports and validated three fixes.