IndexSplitter drops a segment's doc-values update metadata: the copy breaks once an IndexWriter opens it, and reads wrong values on main

IndexSplitter.split(destDir, segments) (lucene/misc, @lucene.experimental) copies every file of each chosen segment, including the files that hold its doc-values updates, but writes a destination segments_N whose entry for the segment lists none of them: it rebuilds the per-commit SegmentCommitInfo from generation numbers only and never carries the field-infos files, the doc-values update files or (on main) the doc-values overlays. In every executed release from 4.6.0 to 10.5.2 and on main, the first IndexWriter opened on the copy deletes those files as unreferenced, after which the copy cannot be opened (NoSuchFileException: _0_1.fnm). On main, where overlays (#16418) are on by default, the copy is wrong before any writer touches it: an updated field reads only its newest sparse update, so price 10, 999, 12, 13 reads as none, 999, none, none, and CheckIndex reports the copy clean. Any numeric or binary doc-values update that the copied segment still carries (not yet folded in by a merge) triggers it, soft deletes included; default configuration, deterministic, no scale. The source index is never modified, and the tool is rarely used. It is the IndexSplitter half of LUCENE-8243 (2018), which fixed the same omission in IndexWriter.addIndexes(Directory...).

• Segment and commit: an index is a set of segments; the commit file segments_N lists, per segment, which files belong to it. When an IndexWriter opens an index it deletes every file that no commit lists.
• Doc-values update: IndexWriter.updateNumericDocValue / updateBinaryDocValue (and soft deletes, which use the same mechanism) change a doc value without reindexing. The new values go into extra files of a new generation (_0_1.fnm, _0_1_Lucene90_0.*), and the commit lists them for that segment.
• Doc-values overlays (#16418, main; opt-in on branch_10_6): a set-only update writes a sparse delta with just the updated documents, and the commit records how to layer it over the original column. Without that record, the delta is read alone.
• IndexSplitter: a lucene/misc command-line tool and class that copies chosen whole segments of an index into a new index (IndexSplitter <src> <dest> <segments>).

1. For each segment, split() builds a new SegmentInfo on the destination (IndexSplitter.java:118-131) and then new SegmentCommitInfo(newInfo, delCount, softDelCount, delGen, fieldInfosGen, docValuesGen, id) (:132-140). That object starts with no field-infos files, no doc-values update files and no overlays, and split() never sets them.
2. It still copies infoPerCommit.files() (:142-147), which includes the update files (SegmentCommitInfo.java:283-304).
3. It commits (IndexSplitter.java:149-150); SegmentInfos.write persists exactly those three per-segment items (SegmentInfos.java:727-747), here empty. The generations are written, so a reader still finds _0_1.fnm by name and the copy opens.
4. Every release since 4.6.0: an IndexWriter opened on the copy deletes the unlisted files (IndexFileDeleter.java:219-233, "init: removing unreferenced file"); the segment still needs _0_1.fnm, so the next open fails.
5. Main: SegmentDocValuesProducer opens an overlay only when the commit has an entry for the field, and otherwise reads the field's newest generation file alone (SegmentDocValuesProducer.java:77-94). With overlays (IndexWriterConfig.java:115, default 8) that file holds only the updated document, so the others read as missing. Before overlays the newest file was a full rewrite of the column, which is why releases read correctly.
6. Hard deletes survive because delGen is passed and files() adds the live-docs file (SegmentCommitInfo.java:290-293).
7. Core's own as-is copy, IndexWriter.copySegmentAsIs (IndexWriter.java:3647-3705), carries all three items (:3677-3686) and asserts copiedFiles.equals(newInfoPerCommit.files()) (:3701-3702), the invariant IndexSplitter breaks.

Source: price 10,999,12,13; its commit lists 8 files on main (7 on 10.5.2, 6 on 9.12.3). Copy: all files copied, commit lists _0.cfe, _0.cfs, _0.si. Main: the copy reads none,999,none,none. After an IndexWriter open/close: only _0.cfe, _0.cfs, _0.si, segments_1 and write.lock are left, and DirectoryReader.open throws NoSuchFileException: _0_1.fnm. Binary updates: v0, v1, v2, X, v4 reads none, none, none, X, none on main.

• Who is hit. Users of IndexSplitter (command line or new IndexSplitter(src).split(dest, segments)) who copy a segment that still carries a numeric or binary doc-values update (a segment written by a merge starts without any, IndexWriter.java:5343). Soft deletes count: in the follow-up probe a soft-deletes index split and then opened by a writer fails with NoSuchFileException: _0_2.fnm on main and on 10.5.2.
• Default, not opt-in. The release symptom needs no setting. The wrong values are the default on main; on branch_10_6 (10.6.0, released 2026-10-29) overlays default to 0, so they need setMaxDocValuesOverlays(n > 0) (executed with 8 on branch_10_6 2763db11b0a).
• No scale, no timing. Deterministic; 4 documents and one update are enough.
• What is lost. Releases: the copy works until any writer touches it (adding documents, merging), then its update files and field infos are gone and it cannot be opened. Main: silent wrong values with a clean CheckIndex. In the follow-up probe (main, assertions off, re-run on a9779ef3c193): a MUST_NOT rule on an updated numeric doc-values field returns ids [0, 1, 3] on the copy instead of [1, 3]; after a soft delete and then a soft update on the same segment, the soft-deleted document is live again in the copy (with assertions on, an internal assert in SoftDeletesDirectoryReaderWrapper stops that open).
• Recoverable in practice. split() only reads the source, so the copy can be made again, unless the segment was since removed from the source with IndexSplitter <src> -d and a writer deleted its files (not executed).
• Reach. A rarely used experimental tool; a read-only code search found no common search server that calls it (Solr's shard split uses its own SolrIndexSplitter). No prior report was found (see Prior reports); the bug dates from 4.6.0 (2013).

• The contract. The class javadoc: "copying specific segments to another index" (IndexSplitter.java:36-37). Its listed limits: it cannot split a single segment, it does file-level copying so it needs an FSDirectory, and "You can easily accidentally remove segments from your index so be careful!" (:39-44). Nothing about doc-values updates or soft deletes.
• The tool copies the update files (files(), :142-147), then commits an entry that does not own them.
• Core enforces the invariant on the same operation: copySegmentAsIs sets the field-infos files, the update files and every overlay and asserts copiedFiles.equals(newInfoPerCommit.files()) (IndexWriter.java:3677-3686, :3701-3702).
• LUCENE-8243 (#9290, commit 99364584fff, 2018, Bug Fixes in 7.4.0): "IndexWriter.addIndexes(Directory[]) did not properly preserve index file names for updated doc values fields". The same omission, fixed in core only; its description and comments never mention IndexSplitter.
• LUCENE-3188 (#4261, 3bc70d2368d, 2011): "contrib/misc IndexSplitter creates indexes with incorrect SegmentInfos.counter", fixed as a bug; its test opened an IndexWriter on the split destination.
• History is consistent with intended support (a reading, not a statement by the maintainers). When doc-values updates arrived, the maintainers passed the new generations through IndexSplitter (LUCENE-5189 49684b82701, LUCENE-5618/5636 c863a7251d5); the per-commit file sets were missed. No javadoc, CHANGES entry or test says updated segments are unsupported; TestIndexSplitter never applies a doc-values update.

The strongest case for "intended": the tool is @lucene.experimental and does "basic file-level copying" (IndexSplitter.java:41), and TestIndexSplitter switches off the unreferenced-files check because "IndexSplitter.split makes its own commit directly with SIPC/SegmentInfos, so the unreferenced files are expected" (TestIndexSplitter.java:39-43). So the tool never promised a destination that survives a writer. The full case adds: the javadoc states no postcondition that the destination stays writable; the tool predates doc-values updates; on releases the copy reads correctly as long as no writer opens it; IndexWriter.addIndexes(Directory...) is the supported way to copy segments with updates; and the wrong-values symptom is in no release by default (10.6.0 only with overlays enabled).

Why it fails: @lucene.experimental says the API may change, not that the tool may write a commit that disowns files it needs. The test waiver is set on the source directory (:38), which keeps the files of segments removed with -d; those really are garbage. It came with LUCENE-4456 (cb402315131, 2012-10-03), before doc-values updates existed. LUCENE-3188 already treated an unusable destination as a bug. Every reviewer, including the cross-model reviews, rated the intended reading weak. What partly survives is reach: a maintainer may prefer to refuse segments with field updates rather than support them, which is a choice of fix, not a defence of today's silent success.

Two fixes were built and validated on a scratch worktree of main a9779ef3c193; two more are listed with their trade-offs.

Workaround until fixed: do not split segments that have doc-values updates or soft deletes. IndexWriter.addIndexes(Directory...) carries this metadata, but copies every segment of its source.

Do not bundle: the same constructor call also drops the in-memory SegmentInfo attributes and index sort (IndexSplitter.java:130-131); judged harmless by reading, because readers use the copied .si file, and not executed.

In IndexSplitter.split, replace the new SegmentInfo and the SegmentCommitInfo built from generation numbers (IndexSplitter.java:116-140) with destInfos.add(infoPerCommit.clone()). clone() is public and deep-copies the field-infos files, the doc-values update files and the overlays (SegmentCommitInfo.java:427-451). lucene/misc only; +4/-27 lines (two imports go).

SegmentCommitInfo infoPerCommit = getInfo(n);
// Copy the whole per-commit state (deletes, field-infos and doc-values update files and
// doc-values overlays), not only the generations: the destination commit must reference
// the update files it copies and describe how they combine with the segment's own files.
destInfos.add(infoPerCommit.clone());
// now copy files over
Collection<String> files = infoPerCommit.files();

Trade-offs

• The clone shares the source's SegmentInfo, whose directory is the source FSDirectory. split() uses the new SegmentInfos only to write segments_N into the destination, and SegmentInfos never reads a segment's directory (no .dir use in SegmentInfos.java), so it did not matter in any test. But the in-memory clone points at the source: SegmentCommitInfo.sizeInBytes() reads file lengths through info.dir (SegmentCommitInfo.java:270-274), so it would measure the source's files; no split() path calls it (listSegments() calls it on the source's own infos, IndexSplitter.java:88). A maintainer may prefer a destination-bound SegmentInfo for this reason (next option).
• It also keeps the in-memory attributes and index sort that the current rebuild drops (Collections.emptyMap(), null).
• New per-commit state added to SegmentCommitInfo later is carried for free, as long as clone() copies it.

What the checks validated

• Gradle, -Ptests.seed=DEADBEEF, assertions on and off: the new test and TestIndexSplitter 5/5; the whole :lucene:misc:test suite with the new test, 319 tests, 0 failures (9 skipped); :lucene:misc:check -x test (google-java-format, ecjLint, forbiddenApis) passes.
• Repro harness on a9779ef3c193 with the fix shadowed: the independent reproduction's independent test 5/0 and the follow-up probe (deny rule, soft deletes) 5/0, against 5/4 and 5/2 without it.
• Earlier, on ea57360912ed: the finder's tests, the independent reproduction's 9 tests and the adjudication's re-run passed with it.

Keep the rebuilt SegmentInfo on the destination directory, then call setFieldInfosFiles, setDocValuesUpdatesFiles and, for every overlay, setDocValuesOverlay, as IndexWriter.java:3677-3686 does. setDocValuesOverlay is package-private (SegmentCommitInfo.java:165), so it becomes public and @lucene.internal.

Trade-offs

Keeps the destination directory in the in-memory object and mirrors core. Costs a new public (internal) method on a core class, and three pieces of state to keep in sync if SegmentCommitInfo gains more. Using only the two setters that are already public is not enough: it removes the NoSuchFileException but still reads wrong values on main (the independent reproduction: 9 run, 6 failed).

What the checks validated

The new test 4/4; :lucene:misc:test 319 tests, 0 failures, assertions on and off; lucene/misc static checks plus core checkGoogleJavaFormat, ecjLintMain, forbiddenApisMain and renderJavadoc pass. Core tests were not run (only a visibility change there).

Throw IllegalArgumentException in split() when infoPerCommit.hasFieldUpdates() (SegmentCommitInfo.java:325).

Trade-offs

Small and safe, but segments with soft deletes or any doc-values update can then no longer be split. Not run.

Document that segments with doc-values updates or soft deletes are unsupported.

Trade-offs

Leaves a tool that reports success while writing a broken or wrong copy. Not run.

Non-formal finding. The hunt's hunt unit on the misc reorder and split tools had the contract "splitters partition without loss". While checking it, an AI tool (the finder) compared the SegmentCommitInfo that IndexSplitter.split rebuilds with the per-segment fields SegmentInfos.write persists, saw that the field-infos files, the doc-values update files and the overlays are never carried, and witnessed both symptoms on main, and the writer-open symptom on 10.5.2 and 9.12.3 (a public-API JUnit on the stock jars) and on 9.0.0, 8.11.4 and 7.0.0 (portable main() programs, where the copy still read 10, 999, 12, 13 before the writer).

The unit's formal checks (Z3 proofs, bounded-exhaustive doc-map checks, a merge-level differential) covered BPIndexReorderer, BpVectorReorderer, BPReorderingMergePolicy and MultiPassIndexSplitter, not IndexSplitter, and produced no signal here.