MultiPassIndexSplitter cannot split an index that has a soft-deletes field or a parent field

MultiPassIndexSplitter (lucene/misc) splits an index into N parts by copying the input's segments, with the other parts' documents hidden, into one new IndexWriter per part. Every part's writer is opened with new IndexWriterConfig(null).setOpenMode(OpenMode.CREATE), and split(IndexReader, Directory[], boolean), the only API, takes no config. Since LUCENE-8335 (7.4.0) IndexWriter.addIndexes rejects an incoming field marked as the soft-deletes field unless the writer's config names the same field, and since #12829 (9.10.0) the same holds for the parent field. So the splitter throws IllegalArgumentException on every index whose segments still mark a soft-deletes field (in practice any index with a soft update or soft delete in a surviving segment; the mark survives merges), and on every index written with setParentField, in both split modes and from its command line. It fails before any part gets a document; the source is only read. No release has been able to split an index whose segments carry the soft-deletes mark: 7.4.0 shipped soft deletes and the check together. Configuring each part with the input's two field names fixes it.

1. The split. split() (MultiPassIndexSplitter.java:70) wraps the input in a private FakeDeleteIndexReader (:81), and for each part marks every document outside the part as deleted (:84-107), opens the part's writer with new IndexWriterConfig(null).setOpenMode(OpenMode.CREATE) (:108-109) and copies the leaves with w.addIndexes(...) (:113). The command line main() opens a plain DirectoryReader (:157) and calls split() (:182); run on such an index it throws the same exception and exits with code 1 (executed).
2. The writer's field roles. The writer builds its field-number map from its config: new FieldNumbers(config.getSoftDeletesField(), config.getParentField()) (IndexWriter.java:1267-1269), so here both are null.
3. The check. addIndexes(CodecReader...) runs "Best effort up front validations" over every incoming field (IndexWriter.java:3288-3295, verifyFieldInfo at :3292). verifySoftDeletedFieldName throws "this index has [soft] as soft-deletes already but soft-deletes field is not configured in IWC" (FieldInfos.java:459-465); verifyParentFieldName throws "can't add field [_parent] as parent document field; this IndexWriter has no parent document field configured" (FieldInfos.java:484-490).
4. What carries the mark. A segment's field infos mark the soft-deletes field once a soft update or soft delete wrote it. In the minimal test the writer has already merged the soft-deleted old b away when it closes: the index is one segment from a merge with 2 documents, both live, and none has a value in soft, yet the mark alone is rejected. The mark is per segment: once every segment that carried it has been dropped as fully deleted, the index has no soft field info and splits (executed). A configured but never-written soft-deletes field splits fine. A parent field is added to every index written with setParentField.
5. No supported way around it. The config is built inline and the reader wrapper is private; the command line has no option for these settings.
6. Side effect. The exception leaves the first part's writer open with its write lock (a separate registry row, LFV-554).

• Who. Users of MultiPassIndexSplitter (its command line or split()) on an index whose segments carry a soft-deletes field (7.4.0 and later; at least one soft update or soft delete in a surviving segment, whose mark survives merges) or has a parent field (9.10.0 and later). Both settings are opt-in, but MIGRATE.md requires the parent field since 10.0 for index sorting combined with document blocks. A default index is not affected.
• Effect. Every split of such an index fails at once, in both split modes, with assertions on and off, at any size and with no timing condition. No release has been able to split an index whose segments carry the soft-deletes mark, because 7.4.0 shipped soft deletes and the check together.
• Workarounds. None supported through this API. PKIndexSplitter with its config constructors, or addIndexes into writers configured with the same field, do work.
• Not affected. The source index (only read). The parts get no document and no commit; on a file-system directory the first part keeps only its write.lock (LFV-554). It is not data loss or corruption.
• How likely: plausible. Deterministic for anyone who runs the tool on such an index, but the tool is little used, and no report was found in the searched sources (see Prior reports).

Two documents and one softUpdateDocument under setSoftDeletesField("soft"), read through SoftDeletesDirectoryReaderWrapper: 2 live documents, the minimum split() accepts, split sequentially into 2 ByteBuffersDirectory parts. The test sits in org.apache.lucene.misc.index for locality and uses no package-private API. ByteBuffersDirectory keeps the first part's leaked writer (LFV-554) from tripping the test framework's open-file check. The test body makes no random choices; the test framework still randomizes its usual environment (codec, similarity, locale and the like), and seeds DEADBEEF and 5EED1234 fail the same way.

The parent-field half shares the root cause and was executed in the hunt's longer tests (listed above); it is not part of this repro.

Regression test: this same test; it passes with the recommended fix. Run ./gradlew tidy before committing it: google-java-format wraps its line 36, which is over 100 characters.

cp $REPRO/TestMultiPassIndexSplitterSoftDeletes.java $LUCENE/lucene/misc/src/test/org/apache/lucene/misc/index/
cd $LUCENE && ./gradlew :lucene:misc:test --tests TestMultiPassIndexSplitterSoftDeletes -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true
TestMultiPassIndexSplitterSoftDeletes > testSplitIndexWithSoftDeletes FAILED
    java.lang.AssertionError: expected the 2 live docs split into 2 parts, got: java.lang.IllegalArgumentException: this index has [soft] as soft-deletes already but soft-deletes field is not configured in IWC
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:6CC305A8ED4B11FF]:0)
        at org.apache.lucene.misc.index.TestMultiPassIndexSplitterSoftDeletes.testSplitIndexWithSoftDeletes(TestMultiPassIndexSplitterSoftDeletes.java:49)

        Caused by:
        java.lang.IllegalArgumentException: this index has [soft] as soft-deletes already but soft-deletes field is not configured in IWC
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.index.FieldInfos$FieldNumbers.verifySoftDeletedFieldName(FieldInfos.java:462)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.index.FieldInfos$FieldNumbers.verifyFieldInfo(FieldInfos.java:403)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3292)
            at org.apache.lucene.misc.index.MultiPassIndexSplitter.split(MultiPassIndexSplitter.java:113)
            at org.apache.lucene.misc.index.TestMultiPassIndexSplitterSoftDeletes.testSplitIndexWithSoftDeletes(TestMultiPassIndexSplitterSoftDeletes.java:47)

1 test completed, 1 failed

# -Ptests.asserts=false: the same failure
# with the fix applied: BUILD SUCCESSFUL (assertions on and off)

• Contract. "This tool splits input index into multiple equal parts" (MultiPassIndexSplitter.java:43); @param in "source index, can have deletions, can have multiple segments (or multiple readers)" (:63); the only stated limitation is that it "can easily break up" document blocks (:53-55). setSoftDeletesField: a soft-deleted document "is treated as if it has been hard-deleted through the IndexWriter API" (IndexWriterConfig.java:541-545).
• Tests. TestMultiPassIndexSplitter builds plain indexes only. No test splits a soft-deletes or parent-field index or asserts the rejection for this tool. The core tests that assert the two messages are about a writer whose config disagrees with its index (TestIndexWriter.testPreventChangingSoftDeletesField, testIndexWithParentFieldIsCongruent), and TestAddIndexes.testAddIndicesWithSoftDeletes and testIllegalParentDocChange copy such an index into a destination configured with the same field.
• History. The splitter's fixed config is older than both checks (LUCENE-2294, 2010; LUCENE-5871, 2014). The checks came from core changes that did not touch lucene/misc and mention no tool: LUCENE-8335 (#9382, 2018) and #12829 (2024). At the branch_7x parent of LUCENE-8335, which already had soft deletes, the split worked; at that commit it throws (executed).
• Normal indexes. Lucene's own RandomIndexWriter turns soft deletes on at random in ordinary tests, and PKIndexSplitter and IndexUpgrader accept a config (IndexRearranger's constructors take one too, but since 10.0 it keeps only the analyzer, a separate row, LFV-695).

The strongest case: an old command-line tool that never took a config; in LUCENE-3296 (#4369, 2011, an enhancement) committers gave PKIndexSplitter config constructors and left this class without one; soft deletes are an expert feature; and for the parent field the class already warns that it breaks blocks, so rejecting such an index may even be the safer outcome.

Why it fails for soft deletes: the 2011 decision was about defaults, seven years before soft deletes existed; no javadoc, test, CHANGES or MIGRATE entry excludes such indexes; "can have deletions" covers them; and core shows the supported copy path. For the parent field the argument is partly right (see the fix's trade-off), which is why that half is secondary. Three reviews rated the works-as-designed case moderate and found it does not hold for the soft-deletes half (verifier and the refute-lens review, both OpenAI GPT-5.6, and the devil's-advocate verifier): a classification risk (bug or improvement), not evidence of intent.

Option A was built and run; B changes the API; C is not recommended.

In split() (MultiPassIndexSplitter.java:83-109), read the input's merged field infos once before the loop and set both names on each part's config (null, the default, when the input has none). The hunt's fix sketch did the same inside the loop with fully qualified names.

// before the loop:
// each part must name the input's soft-deletes and parent fields (null if none), or
// IndexWriter#addIndexes rejects the input's field infos
FieldInfos inputFieldInfos = FieldInfos.getMergedFieldInfos(in);
// in the loop, instead of new IndexWriterConfig(null).setOpenMode(OpenMode.CREATE):
IndexWriter w =
    new IndexWriter(
        outputs[i],
        new IndexWriterConfig(null)
            .setOpenMode(OpenMode.CREATE)
            .setSoftDeletesField(inputFieldInfos.getSoftDeletesField())
            .setParentField(inputFieldInfos.getParentField()));

Trade-offs

• Parent field. With the fix a split that cuts a block succeeds but leaves a part whose segment ends in child documents with no parent after them, under a declared parent field. A round-robin split always cuts blocks (executed by the intended-behaviour verifier); a sequential split does when a part boundary falls inside a block (executed by the fact check on an index-sorted block index). That is the documented "can easily break up such document groups" behaviour, now in an index that declares blocks, and the precondition of a separate registry row (LFV-417: BP reordering merges then throw). So for the parent field this option is a policy choice for the maintainers, not a finished fix: keep rejecting parent-field input with a clearer message, or make the split block-aware.
• Soft deletes. The markers are copied with the documents into parts configured with the same field, so soft-deleted documents stay deleted (an independent check read the input with a plain reader, as main() does). Never strip the mark instead: that would bring soft-deleted documents back.
• Mixed inputs. One config cannot name two fields: getMergedFieldInfos throws if the input's leaves (for example a MultiReader over two indexes) disagree on the parent field or use different soft-deletes field names (read from the code). Such an input is rejected today anyway.
• Other settings. Only the two field names are carried over; other settings of the input, such as an index sort, are still not set on the parts (the config-supplier option would let the caller choose them).

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-30):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off; the hunt's and an independent verifier's longer tests (soft deletes and parent field) pass 5 of 5; TestMultiPassIndexSplitter (2 tests) and the whole :lucene:misc:test pass (24 classes, 317 tests, 0 failures, 9 skipped); ecjLint and forbiddenApis for misc pass, and google-java-format accepts the fix but wraps line 36 of the minimal test (run ./gradlew tidy before committing it); the diff was reverted and the worktree removed afterwards.
• With the same change shadowed into the stock jars, a probe splits the minimal input into a and b (1 live document per part, soft configured on both), for the wrapped and the plain-reader input.

A split(..., Supplier<IndexWriterConfig>) overload or constructor, as PKIndexSplitter has (LUCENE-3296).

Trade-offs

Fully explicit, and it also lets a caller choose codec or index sort and covers the previous-major-version rejection noted in LFV-554. But callers must know to set the fields, the command line still needs option A or new options, and it adds public API. Not run.

Javadoc change only.

Trade-offs

Leaves the tool unusable for soft-deletes and parent-field indexes, while the core already supports copying them.

Method: randomized metamorphic round-trip differential testing of the index tools (the hunt classes it as non-formal).

• A harness built random source indexes, some with a soft-deletes field, a parent field or both, split or merged them with each tool, and compared the parts' live documents with the source's.
• In one batch of 10 JVMs (5 with assertions on, 5 off), MultiPassIndexSplitter threw IllegalArgumentException on 525 of 597 such sources (412 parent-field and 113 soft-deletes messages; the other 72 split), while PKIndexSplitter with matching configs reproduced all 597 exactly.
• The finder minimised it to a two-test public-API JUnit (soft deletes, parent field) on the stock jars; the minimal repro above is its soft-deletes test cut down to two documents and one soft update.
• The same harness saw IndexMergeTool, which has the same bare config, reject such sources too; no verifier traced or ran that, so it is only a sibling here.