BpVectorReorderer's command line cannot reorder an index created by Lucene 9.x or 10.x, or one that uses soft deletes

BpVectorReorderer (lucene/misc, added for 11.0 by #14097) has a command line, main(indexPath, field), that renumbers the documents of an existing index in place so that documents with similar vectors get nearby doc ids. It calls reorderIndexDirectory, which opens a reader on the index, then a new IndexWriter on the same directory with new IndexWriterConfig() and OpenMode.CREATE, and adds each segment back with addIndexes(SortingCodecReader.wrap(...)). A CREATE writer takes the index's created major from its config, by default the running major (11 on Lucene 11), and the default config has no soft-deletes field, while each wrapped segment keeps its own created major. So addIndexes refuses the first segment of every index created by 9.x or 10.x, and of every index that used soft deletes, with an IllegalArgumentException. Lucene 11 documents 9.x and 10.x indexes as fully writable and an index's creation version as permanent, so on 11.0 the tool cannot reorder a compatible index carried over from 9.x or 10.x. On main, now 12.0, it also refuses 11.x indexes. The javadoc of IndexWriterConfig.setIndexCreatedVersionMajor names exactly this addIndexes restriction and its remedy; copying the source's created major and soft-deletes field into the rewrite's config fixes it (+8 lines). Unreleased: the tool is only on the 11.0 release branch branch_11x and on main.

1. main (BpVectorReorderer.java:653-682) takes the index path, the field and three tuning options (--max-iters, --min-partition-size, --thread-count); it has no option for the writer configuration. It calls reorderIndexDirectory(FSDirectory.open(path), executor) (:679-680).
2. reorderIndexDirectory (:684-697) opens a DirectoryReader on the directory, then new IndexWriterConfig() with setOpenMode(OpenMode.CREATE) (:686-687) and a writer on the same directory, and for each segment calls writer.addIndexes(SortingCodecReader.wrap(segment, computeDocMap(...), null)) (:689-694).
3. In CREATE mode the writer starts new SegmentInfos(config.getIndexCreatedVersionMajor()) (IndexWriter.java:1009); the default is the major of Version.LATEST, 11 at this commit (IndexWriterConfig.java:177-179). No soft-deletes field is configured.
4. SortingCodecReader.wrap keeps the segment's createdVersionMajor (SortingCodecReader.java:454-457).
5. addIndexes validates each segment first (IndexWriter.java:3290): validateMergeReader throws when the index's created major differs from the segment's (:3228-3236), 11 vs 10 or 9. For segments that carry a soft-deletes field, the field check that follows (verifyFieldInfo, IndexWriter.java:3292 -> FieldInfos.java:401-403) reaches FieldInfos.verifySoftDeletedFieldName, which throws because the writer has none configured (FieldInfos.java:459-465).
6. All segments of an index share its created major, so the very first addIndexes call fails and nothing is reordered. A soft-deletes field that was configured but never used leaves no trace in the segments and reorders fine.
7. On main after #16942 (2026-10-29) Version.LATEST is 12, so there the same code also refuses indexes created by 11.x: the test below fails on main 716d6f216f5 with "created with major version 11 into this index which has been created by major version 12".

After each stock failure the index held 0 documents; that data loss is the sibling bug LFV-1022 and is not counted here.

• Who is hit. Users of the 11.0 release branch branch_11x or of main (unreleased) who run BpVectorReorderer's command line, or call reorderIndexDirectory from the same package. BPReorderingMergePolicy and computeDocMap with the caller's own IndexWriter are not affected.
• Default vs opt-in. The tool is an opt-in expert utility. Once it runs, default index settings and default options are enough, and no option avoids the failure.
• Scale. No size or timing condition: the first segment fails, every time, with assertions on and off. An index's creation version "never changes" (MIGRATE.md:165), also after IndexUpgrader or forceMerge, so an index created by 9.x or 10.x can never be reordered by the tool. On main (12.0) the same holds for indexes created by 11.x.
• Effect. A clear IllegalArgumentException naming the cause; nothing is reordered. Today the failure also empties the index (LFV-1022, a separate bug in the same method); with LFV-1022 fixed, the index would be kept.

One test class, one method, single-threaded, unchanged by the module's tidy task. The expert setter gives the index the same created major as one written by the previous major (only that: its segment is still written by the running version), so this is the minimal regression test for the created-major check. The compatibility evidence is an index written by the real 10.5.2 jar, which fails with the same exception through the default-options command line (see the steps above). The seed only randomizes the test environment; the failure does not depend on it.

It fails the same way on the 11.0 release branch branch_11x (603af870375), and on main 716d6f216f5, already 12.0, where it writes an 11.x index and gets "created with major version 11 into this index which has been created by major version 12".

The suggested regression test is this same test.

cp $REPRO/TestBpVectorReorderEarlierMajorIndex.java $LUCENE/lucene/misc/src/test/org/apache/lucene/misc/index/
cd $LUCENE && ./gradlew :lucene:misc:test --tests TestBpVectorReorderEarlierMajorIndex -Ptests.seed=5EED1023
# the same failure with -Ptests.asserts=false

TestBpVectorReorderEarlierMajorIndex > testReorderIndexCreatedByPreviousMajor FAILED
    java.lang.IllegalArgumentException: Cannot merge a segment that has been created with major version 10 into this index which has been created by major version 11
        at __randomizedtesting.SeedInfo.seed([5EED1023:A364E3378B2403E]:0)
        at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.validateMergeReader(IndexWriter.java:3235)
        at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3290)
        at org.apache.lucene.misc.index.BpVectorReorderer.reorderIndexDirectory(BpVectorReorderer.java:691)
        at org.apache.lucene.misc.index.BpVectorReorderer.main(BpVectorReorderer.java:680)
        at org.apache.lucene.misc.index.TestBpVectorReorderEarlierMajorIndex.testReorderIndexCreatedByPreviousMajor(TestBpVectorReorderEarlierMajorIndex.java:43)
1 test completed, 1 failed

# the same with -Ptests.asserts=false, on branch_11x 603af870375, and with the standalone runner (assertions on and off); passes with the fix

• Contract. main: "@param args two args: a path containing an index to reorder" (BpVectorReorderer.java:648-651), with no version restriction; neither the class nor main is marked @lucene.experimental. MIGRATE.md:143: "Set to 9 for Lucene 11.0.0, so indexes created with Lucene 9.x remain fully supported (open and write)"; :165: "The index's creation version is still 9 (this never changes)"; CHANGES 11.0, GITHUB#16741: "Indexes created with 9.x remain fully supported in Lucene 11, no reindexing required".
• The documented remedy. IndexWriterConfig.setIndexCreatedVersionMajor: "It is sometimes useful to set the previous major version for compatibility due to the fact that IndexWriter#addIndexes only accepts indices that have been written with the same major version as the current index" (IndexWriterConfig.java:174-178), added by LUCENE-8508 (#9554) so that a new index can take an older created major and so accept older segments through addIndexes (the issue's stated motivation is resharding with addIndexes).
• Other tools. IndexSplitter.split writes new SegmentInfos(infos.getIndexCreatedVersionMajor()) (IndexSplitter.java:112); IndexUpgrader writes into the existing index with the default open mode CREATE_OR_APPEND (IndexUpgrader.java:183), so its created major is kept.
• Tests and history. TestBpVectorReorderer runs main() as "the index reordering tool" (:353-354, :412-413), only on indexes it just created on main. #13683, the tool's first version, describes renumbering "an existing HNSW graph ... without the need to re-index it" and lists limitations (float vectors only, no index sort), but no index-version or soft-deletes limit; #14097 discusses none either.

Strongest case (built by the devil's-advocate check and the cross-model reviews, rated moderate): the same-major rule of addIndexes is deliberate and tested (LUCENE-7756; TestBasicBackwardsCompatibility.testAddOldIndexesReader asserts the same message); the command line began as a research tool (in #13565 its author wrote "it would be easier to start with a tool that rewrites the index" if "we want to just get an idea of what the impact of such a change might be"; on #14097 a reviewer accepted open questions "especially for the misc module"); CHANGES describes #14097 as a merge policy and never mentions the command line; and IndexMergeTool and MultiPassIndexSplitter use the same default-config addIndexes pattern without a report.

Why it fails: the guard stops mixing majors in one index, but this tool rewrites one index's own segments into the same directory, which mixes nothing, and the documented setter exists for exactly this case. The command line is public, unmarked and tested upstream, and ships in 11.0, where an index carried over from 9.x or 10.x is a documented, supported index state; failing on it is not a research trade-off. IndexMergeTool and MultiPassIndexSplitter write a new index elsewhere, where stamping the current major is arguable; this tool overwrites the user's index in place, whose creation version main says never changes. Verdict: a defect. Residual risk: a maintainer may prefer to document the limit (option C).

Option A was built and validated with gradle in a fresh scratch worktree at 645d4fe14a20, and earlier with the standalone runner and through the stock command line on a real 10.5.2 index; options B and C were not built.

In reorderIndexDirectory, after setOpenMode(CREATE), set the config's created major to the one recorded in the index's latest commit, and its soft-deletes field to the one recorded in the index's field infos, if any (+8 lines and two imports in BpVectorReorderer.java; the hunt's fix sketch). It carries over only these two settings: commit user data, a parent field and an index sort are still not carried over (the parent field deliberately, see below; the other two behave as today and are not claimed).

// addIndexes only accepts segments of the index's own created major, and checks the
// soft-deletes field: keep both from the index being rewritten
iwc.setIndexCreatedVersionMajor(
    SegmentInfos.readLatestCommit(directory).getIndexCreatedVersionMajor());
String softDeletesField = FieldInfos.getMergedFieldInfos(reader).getSoftDeletesField();
if (softDeletesField != null) {
  iwc.setSoftDeletesField(softDeletesField);
}

Trade-offs

• The success path for indexes created by the running major is unchanged.
• The rewritten index keeps its created major, as IndexUpgrader does, so it stops being openable when MIN_SUPPORTED_MAJOR is later raised, exactly like the original (the note on setIndexCreatedVersionMajor).
• readLatestCommit re-reads segments_N; it is the commit the reader just opened unless another writer commits in between (the tool assumes none). Reading the reader's own commit, SegmentInfos.readCommit(directory, ((DirectoryReader) reader).getIndexCommit().getSegmentsFileName()), closes even that window (not built).
• Parent-field indexes are still refused ("this IndexWriter has no parent document field configured"); that is arguably right, because the tool does not keep document blocks (LFV-093), so the fix deliberately does not copy the parent field.
• It does not fix LFV-1022: a failed run still commits an empty index. Land both; each fix alone leaves the other bug's test failing (LFV-1022's fix alone: this test still fails; this fix alone: LFV-1022's test still fails, 10 documents expected, 0 found).
• Open PR #15430 moves the class to core with this method unchanged; if it merges first, the fix moves with it.

What the checks validated

Gradle, fresh scratch worktree at 645d4fe14a20, JDK 25, seed 5EED1023: without the fix the test fails (assertions on and off); with it TestBpVectorReorderEarlierMajorIndex + TestBpVectorReorderer pass 9 of 9 with assertions on and again with -Ptests.asserts=false, and :lucene:misc:test passes 24 suites, 317 tests, 0 failures (9 skipped by the test framework). Standalone runner: the test passes with -ea and -da; indexes written by the real 10.5.2 and 9.12.3 jars are reordered keeping created major 10 / 9 and all 10 documents; the soft-deletes index keeps its 11 documents; the default-options command line on a 10.5.2 index exits 0 with 10 documents, created major 10.

Take the created major from reader.leaves().get(0).reader().getMetaData().createdVersionMajor() instead of re-reading segments_N.

Trade-offs

Same behaviour, but needs a special case for an empty index (no segments).

Check the created major and the soft-deletes field before opening the writer, and refuse with "reindex with the current major first".

Trade-offs

Documents the limit instead of removing it: still no reorder for any index from an earlier major. It never opens the CREATE writer for these inputs, so it also avoids LFV-1022's wipe for them.

Method: code reading confirmed by execution (non-formal tier: the first signal was code reading, not a model).

• The hunt unit was testing the failure atomicity of the tool's in-place rewrite (the sibling bug LFV-1022). One leg of its harness ran config-only natural inputs through the stock command line: an index with an earlier created major, a soft-deletes index, a parent-field index and a control. The earlier-major and soft-deletes indexes were refused; the control was reordered.
• It then wrote indexes with the real 9.12.3 and 10.5.2 jars and ran them through main's command line: both refused. It wrote the JUnit, later reduced to the 1-document test above. The parent-field refusal was not claimed (refusing it is reasonable, LFV-093).