addIndexes(CodecReader...) counts soft deletes before OneMerge#wrapForMerge removes documents, and later merges build on the wrong count

IndexWriter.addIndexes(CodecReader...) merges the incoming readers as the merge policy's findMerges(CodecReader...) says, and since 9.9.0 it calls OneMerge#wrapForMerge on each reader, which may "add/remove fields or documents". But it takes two counts from the readers before that call: the documents it reserves (getPendingNumDocs) and the soft-delete count it stores in the new segment. When the wrapper drops an incoming soft-deleted document, the committed segment claims a soft delete it does not hold. In the 3-document example the new segment holds c and a2, records 1 soft delete, reports numDocs 1 instead of 2, and the writer reserves 3 documents instead of 2. With assertions off (production) later operations build on the wrong count: a later ordinary forceMerge(1) after adding z commits a 2-document segment for 3 documents in which id:z returns a2's stored document and CheckIndex finds nothing (LogDocMergePolicy), or the merge throws CorruptIndexException (TieredMergePolicy); in other shapes a live document disappears at a later commit, once an NRT reader is opened or a delete is applied, or that commit throws invalid softDelCount. The regular merge path takes both counts after wrapping. Only a custom findMerges(CodecReader...) with a document-removing wrapForMerge triggers it; no in-tree policy removes documents on this path. Introduced by #12622 (9.9.0); affects 9.9.0 to 10.6.0 (10.6.0 by reading) and main.

• addIndexes(CodecReader...): imports readers into the index by merging them into new segments. Which readers go into which merge is decided by MergePolicy#findMerges(CodecReader...), which "can be overridden in custom merge policies"; by default it is one merge for all readers.
• OneMerge#wrapForMerge: a hook that wraps each reader "prior to merging in order to add/remove fields or documents". IndexWriter calls it on the regular merge path and, since 9.9.0, on the addIndexes path.
• Soft deletes: with IndexWriterConfig.setSoftDeletesField("soft"), a document is hidden by giving it a doc value in that field (for example by softUpdateDocument). Each segment stores how many of its documents are soft-deleted, and the writer derives the live count from it: numDocs = maxDoc - deletes - soft deletes. A reader from DirectoryReader.open(dir) does not apply soft deletes, so soft-deleted documents arrive at addIndexes as live.
• getPendingNumDocs: "the number of documents in the index including documents are being added (i.e., reserved)", used to enforce IndexWriter.MAX_DOCS.

1. OneMerge(CodecReader...) sizes the merge from the incoming readers: totalMaxDoc is the sum of their numDocs() (MergePolicy.java:245-256).
2. addIndexesReaderMerge counts soft deletes on those same unwrapped readers (IndexWriter.java:3500-3515): 1 in the example (a).
3. Only then does it wrap them, merge.wrapForMerge(mr.codecReader) (IndexWriter.java:3542-3546). The wrapper hides a, the merge writes c and a2, and the new segment is created with the old count: new SegmentCommitInfo(segInfo, 0, numSoftDeleted, ...) (:3600-3601), that is maxDoc 2 and 1 soft delete.
4. addIndexes then reserves merge.totalMaxDoc, 3, although the merged segment already knows it holds 2 (IndexWriter.java:3341-3356).
5. Later code trusts the stored count. The live count becomes 2 - 1 = 1 (PendingDeletes.java:243-250). A segment whose stored deletes equal its maxDoc counts as fully deleted and is dropped at commit (PendingDeletes.java:219-220, IndexWriter.java:6429-6435). A merge sizes its output from the readers' live counts and gives each reader a doc base advanced by its live count (MergeState.java:170-173, :197-208). The segment has no document with the soft-deletes field, so no document is marked deleted, but its live count is 1: c and a2 map to doc 0 and 1, and z, whose doc base is 1, also maps to 1, in a 2-document segment. This explanation is read from the code and fits the LogDocMergePolicy observation; it was not stepped through in a debugger. TieredMergePolicy merges z's segment first (executed: an infoStream run logs "merging _1(11.0.0):c1 _0(11.0.0):c2"); by the same reading that gives doc IDs 0, 1 and 2 in a 2-document segment, which fits its "invalid docCount: 3 maxDoc: 2".

The regular merge path takes both counts after wrapping: commitMerge returns the documents a merge did not write to the reservation (IndexWriter.java:4990-5000), and mergeMiddle derives the soft-delete count from the wrapped reader (:5494-5540). A regular forceMerge with a dropping wrapper keeps getPendingNumDocs equal to maxDoc. (That mergeMiddle code has its own, separate flaw, LFV-730: for a segment without hard deletes it counts every live document a wrapper hides as a dropped soft delete.)

Source: one segment with a (marked expired:true, then soft-deleted by softUpdateDocument(id:a -> a2)), c and a2. After addIndexes with a wrapper that hides expired documents, and commit():

Then addDocument(z), commit(), forceMerge(1), assertions off, default codec Lucene104: with LogDocMergePolicy the merged segment has maxDoc 2, id:c finds doc 0 (c), id:a2 finds doc 1 (a2), id:z finds doc 1 (a2), and CheckIndex is clean at levels 1, 2 and 3; with TieredMergePolicy the forceMerge throws CorruptIndexException: invalid docCount: 3 maxDoc: 2 and the earlier commit, with its wrong count, stays. With assertions on the forceMerge fails with AssertionError: pendingNumDocs 4 != 3 totalMaxDoc; underneath, the merge's tragic exception is softDeleteCount doesn't match 1 != 0 (PendingSoftDeletes.java:89-90), after which the writer closes itself and its close-time assertion fires.

Other shapes, assertions off: source a (soft-deleted), b, c with a wrapper that drops a and b leaves a 1-document segment that records 1 soft delete; after an NRT reader is opened or a delete is applied, the next commit drops it and the live c is gone (hits 0, maxDoc 0). With a and b both soft-deleted, numDocs is -1 and the next such commit throws invalid softDelCount=2 (maxDoc=1). Without soft deletes, 5 documents of which the wrapper drops 2 leave getPendingNumDocs at 10 after two calls while the index holds 6; a wrapper that revives a hard-deleted document makes the writer under-reserve (maxDoc 3, getPendingNumDocs 2; shown at this small size only).

• Who. Applications that import with addIndexes(CodecReader...) through their own MergePolicy whose findMerges(CodecReader...) returns a OneMerge with a document-removing wrapForMerge, for example to drop expired or tombstoned documents while importing. MergePolicy is an Expert API. OneMergeWrappingMergePolicy, the base of SoftDeletesRetentionMergePolicy, does not override findMerges(CodecReader...), so it does not wrap these merges. The misc BPReorderingMergePolicy does wrap them (it overrides findMerges(CodecReader...) and delegates wrapForMerge), but it only reorders documents; no in-tree policy removes documents on this path.
• The soft-delete half also needs a soft-deletes field on the destination writer and a wrapper that drops incoming soft-deleted documents. Readers from DirectoryReader.open(dir) do not apply soft deletes, so such documents arrive as live, and a filter that drops them is a natural thing to write.
• Effect. A wrong soft-delete count is committed whenever the wrapper changes how many soft-deleted documents survive and the merge writes a non-empty segment; no large size is needed (the repro has 3 documents). A wrapper that removes only documents that are not soft-deleted over-reserves without a wrong soft-delete count. DocStats.numDocs under-counts. With assertions off (production), later operations can drop a live document at a later commit (once an NRT reader is opened or a delete is applied), fail such a commit with invalid softDelCount, or make an ordinary forceMerge commit a segment in which one document's terms point at another document's stored fields; or the forceMerge throws CorruptIndexException.
• Detection. CheckIndex checks the soft-delete count only at level 2 and up; its command-line default is level 1. After the silent forceMerge, no level reports anything.
• The over-reservation alone is minor. It lasts for the writer session (a reopened writer recomputes it). Its demonstrated production effect is a stricter MAX_DOCS admission check, shown with a lowered test limit, not near 2^31; it also makes the experimental getPendingNumDocs() wrong and, with assertions on, trips the writer's close-time assertion. A wrapper that revives documents under-reserves instead (shown at small size only).
• How rare. A judgement: no downstream project with such a policy was found (keyword search only), and nobody has reported it in the almost three years since 9.9.0.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: DocStats.numDocs after addIndexes expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.index.TestAddIndexesWrapForMergeSoftDeletes.testSoftDeleteCountAfterAddIndexes(TestAddIndexesWrapForMergeSoftDeletes.java:104)
       Suppressed: java.lang.AssertionError: pendingNumDocs 2 != 1 totalMaxDoc
         org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2596)
         org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1337)
         org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1365)
         org.apache.lucene.index.TestAddIndexesWrapForMergeSoftDeletes.testSoftDeleteCountAfterAddIndexes(TestAddIndexesWrapForMergeSoftDeletes.java:99)

• Contract. OneMerge#wrapForMerge: "Wrap a reader prior to merging in order to add/remove fields or documents" (MergePolicy.java:301-309). #12622 replaced the older wording ("add/remove information to the merged segment") in the same commit that added the wrapForMerge call to addIndexesReaderMerge. findMerges(CodecReader...) "can be overridden in custom merge policies" (MergePolicy.java:619-623), and addIndexes(CodecReader...) says readers "are merged as specified by the MergePolicy#findMerges(CodecReader...) API" and points to overriding it (IndexWriter.java:3264-3267). OneMerge(CodecReader...) is public.
• Invariants the code enforces. IndexWriter asserts pendingNumDocs == segmentInfos.totalMaxDoc() on close and rollback (IndexWriter.java:2592-2597). CheckIndex.checkSoftDeletes rejects a stored soft-delete count that differs from the segment (CheckIndex.java:4938-4947). TestAddIndexes.testAddIndicesWithSoftDeletes asserts that the added segments' soft-delete counts equal maxDoc minus the live documents (TestAddIndexes.java:1767-1806).
• History. The count sites come from #633 (LUCENE-10216, 2022); while that PR was in development the same close-time assertion fired ("pendingNumDocs 7 != 5 totalMaxDoc"), and its author suspected their code was "reserving more docs than it should" and fixed it before merge. The counts were right until #12622 (9.9.0) made the merged readers differ from the counted ones; #12622's review never discusses them.
• Consistency. The regular merge path takes both counts after wrapForMerge (IndexWriter.java:4990-5000, :5494-5540); only addIndexes takes them before. (The regular path has a separate flaw of its own for wrappers that hide live documents in segments without hard deletes, LFV-730.)

The intended-behaviour verifier and reviews by a model from another family built the strongest case for "not a bug": the javadoc of findMerges(CodecReader...) names only concurrency control as the reason to override it; no in-tree policy and no test removes documents on this path; MergePolicy is Expert and getPendingNumDocs is @lucene.experimental; the usual way to filter is to wrap the readers before calling addIndexes, which is counted correctly; 9.4.0 to 9.8.0 ignored wrapForMerge on this path entirely, and #12622 has no CHANGES entry of its own (the BP algorithm it builds on is listed under GITHUB#12489 in 9.8.0; no entry mentions wrapForMerge on the addIndexes path); and over-reserving is conservative.

Why it fails: #12622 made removal the explicit contract at the moment the hook started running here; the API is public and documented as overridable; the writer's own assertions, CheckIndex and TestAddIndexes treat both counts as invariants; nothing documents a limitation; and the outcome is lost or merged documents, not a clean error. At most, the argument limits how common the trigger is. Verdict: a defect (every reviewer rated the intended-behaviour argument weak). Expect a maintainer to suggest filtering before addIndexes.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF) with the three repro classes (stock: 11 tests, 9 failures, assertions on and off) and 12 closest upstream test classes. The other options are sketches, not implemented. Which one, or another design, is the maintainers' call.

In addIndexes(CodecReader...), reserve merge.getMergeInfo().info.maxDoc() for each merge that produced a segment, instead of merge.totalMaxDoc. In addIndexesReaderMerge, move the soft-delete count from the loop before wrapping into the wrapping loop, on the wrapped reader and its live docs. One file, +9/-8.

// addIndexes(CodecReader...): reserve each merged segment's real size
for (MergePolicy.OneMerge merge : spec.merges) {
  if (merge.getMergeInfo() != null) {
    totalDocs += merge.getMergeInfo().info.maxDoc(); // was: totalDocs += merge.totalMaxDoc
    infos.add(merge.getMergeInfo());
  }
}
// addIndexesReaderMerge: count soft deletes on the wrapped readers
for (MergeReader mr : merge.getMergeReader()) {
  CodecReader reader = merge.wrapForMerge(mr.codecReader);
  readers.add(reader);
  if (softDeletesEnabled) {
    numSoftDeleted += PendingSoftDeletes.countSoftDeletes(
        FieldExistsQuery.getDocValuesDocIdSetIterator(config.getSoftDeletesField(), reader),
        reader.getLiveDocs());
  }
}

Trade-offs

• For an identity wrapper the wrapped reader's live docs represent the same liveness state as the old hardLiveDocs, so nothing changes for existing users; the counting work is the same.
• The best-effort pre-flight checks still use the unwrapped counts (IndexWriter.java:3296, :3518): a removing wrapper can still be rejected near MAX_DOCS for documents it would drop, and a reviving wrapper is caught only by the final reserveDocs. Both reviews by the other model family called this a defensible, conservative trade-off.
• merge.totalMaxDoc keeps its current values (the inputs' maxDoc sum for segment merges, the unwrapped readers' numDocs() sum for addIndexes merges); only the addIndexes reservation stops using it.
• Not done: the full core suite, check, a CHANGES entry. The sibling LFV-730 needs its own fix in mergeMiddle.

What the checks validated

• Stock: the 3 repro classes run 11 tests with 9 failures, assertions on and off.
• With A: 11 of 11 pass with assertions on and with assertions off.
• With A, the 3 repro classes and the 12 closest upstream classes (core TestAddIndexes, TestIndexWriterMaxDocs, TestSoftDeletesRetentionMergePolicy, TestSoftDeletesDirectoryReaderWrapper, TestPendingSoftDeletes, TestOneMergeWrappingMergePolicy, TestFilterMergePolicy, TestMergePolicy, TestIndexWriterMergePolicy, TestIndexWriterForceMerge, TestCheckIndex; misc TestBPReorderingMergePolicy, the in-tree policy that wraps addIndexes merges to reorder them): 15 classes, 143 tests, 8 skipped (4 nightly, 1 monster, 3 @AwaitsFix), 0 failures, with assertions on and with assertions off.
• With A, TestBPReorderingMergePolicy with the nightly group enabled: 3 of 3 pass, including testReorderOnAddIndexes.
• :lucene:core:checkGoogleJavaFormat passes with A and the compact test.
• Earlier checks with the same change: upstream TestAddIndexes 39 of 39; the independent reproducer's 6 tests, the devil's-advocate probes (the forceMerge and the reviving-wrapper cases) and both plain main() variants pass (3 distinct documents, CheckIndex clean at every level).

Throw in addIndexesReaderMerge when wrapForMerge returns a reader with different live documents (comparing which documents are live, not only how many, since a wrapper can swap live documents and keep the count), and document the limitation in the wrapForMerge and findMerges(CodecReader...) javadoc.

Trade-offs

Simple and fails loudly, but it takes away a capability the javadoc grants and the regular merge path honours. Not implemented.

After merger.merge(), count the documents of the new segment that carry the soft-deletes field (a new segment has no hard deletes).

Trade-offs

Robust to any wrapper and to reorder, but it reads the new segment's soft-deletes doc values once more per addIndexes merge, and the reservation still needs option A's change. Not implemented.

State that removing documents in wrapForMerge is unsupported for addIndexes merges.

Trade-offs

Leaves existing users exposed to silent corruption; not recommended.

Method: code reading (non-formal tier), in a hunt unit that tests how merge-policy wrappers compose with IndexWriter's merge paths (a randomized composition matrix with deliberately broken wrappers to check that its oracles can fail).

• One of those deliberately broken wrappers drops documents; on the regular merge path it exposed the sibling bug LFV-730 (mergeMiddle). While checking whether addIndexes has the same problem, the finder read addIndexesReaderMerge and noticed that the reserved count and the soft-delete count are both taken before wrapForMerge.
• The matrix's own addIndexes triggers use only in-tree wrappers, none of which removes documents, so the matrix did not reach this path.
• The finder wrote a public-API JUnit (2 tests) and a patched IndexWriter: the tests fail on main, 10.5.2, 9.12.3 and 9.9.0 and pass with the patch.
• Found 2026-10-27 on main a9779ef3c193; then checked by an independent reproducer, a devil's-advocate verifier, reviews by a model from another family that ran their own tests, an adjudication, a follow-up (history, prior reports, security), the registration check and an audit.