IndexRearranger ignores the IndexWriterConfig it is given, except the analyzer, since 10.0.0

IndexRearranger (lucene/misc, @lucene.experimental) copies an index into a new directory with a chosen segment layout, and both public constructors take an IndexWriterConfig ("@param config index writer config"). Up to 9.12.3, execute() forced NoMergePolicy on that config and opened its writer with it. Since #11815 ("Support deletions in rearrange", first released in 10.0.0), execute() builds its two writers with new IndexWriterConfig(config.getAnalyzer()) and sets only NoMergePolicy, so every other setting is dropped, and the analyzer it keeps is never used by addIndexes(CodecReader...). The copy is silently written with the default codec and without the configured index sort; a second run with OpenMode.CREATE appends and fails, and an index with a soft-deletes or parent field, or one created with the previous major version, cannot be rearranged. Search results on the copy are correct and the input is only read. Executed on every 10.x release up to 10.6.0 and on main; all 29 releases from 8.9.0 to 9.12.3 (8.9.0 to 8.11.4 and the 21 9.x releases) pass the same assertions.

1. The configs. execute() builds createSegmentsConfig and applyDeletesConfig as new IndexWriterConfig(config.getAnalyzer()) (:119-120) and sets only NoMergePolicy on each (:122-124). Nothing else reads the caller's config, which the class keeps in a protected final field (:70).
2. The defaults they get. Codec.getDefault(), no index sort, OpenMode.CREATE_OR_APPEND, no soft-deletes or parent field (LiveIndexWriterConfig.java:135, :138), creation major Version.LATEST.major (:58).
3. First writer (:126): addIndexes (:209) writes each segment with the writer's codec and index sort (IndexWriter.java:3525 config.getCodec(), :3540 config.getIndexSort()), so with the default codec and indexSort=null. With no index sort the merge only maps around deletions instead of sorting (MergeState.java:214-226), so two sorted input segments are concatenated: [1, 3] then [0, 2].
4. Field checks. addIndexes validates every incoming field against the writer's config up front (IndexWriter.java:3288-3293); an input soft-deletes field or parent field is rejected because the fresh config names none (FieldInfos.java:459-467, :484-490), and so is an input segment created with another major version than the new output (IndexWriter.java:3228-3236). These checks run in the executor task that calls addIndexes, so execute() throws them as the cause of an ExecutionException (:189-198).
5. Open mode. A second run into an existing output appends instead of replacing, so a selector then matches two segments and finalizeRearrange throws (:154-161).
6. Opening the copy later. A writer configured with the original sort checks every segment's sort and rejects indexSort=null (IndexWriter.java:1213-1229).

execute() did config.setMergePolicy(NoMergePolicy.INSTANCE) and new IndexWriter(output, config) with the caller's config (9.12.3 IndexRearranger.java:80-82).

The first two are silent; the other four fail loudly. Search results on the copy are correct, requested deletes are applied, and the input index is only read (:127). Similarity is dropped too, by code reading only. setIndexCreatedVersionMajor is the setting IndexWriterConfig documents for exactly this addIndexes case (IndexWriterConfig.java:175-177).

Where each row was run: the index sort by the minimal test on main, 10.0.0, 10.5.2 and 10.6.0 and by the hunt's tests on main and 10.0.0 to 10.5.2 (the reopen exception in the hunt's probe and in a fact-check probe on main); the codec, OpenMode.CREATE and soft deletes by the hunt's tests on main and 10.0.0 to 10.5.2; the parent field by the independent reproducer on main, 10.5.2 and 10.0.0; the creation major by a fact-check probe on main and on the 10.6.0 jars (assertions on). IndexRearranger.java is identical on 10.6.0 and main.

• Who. Callers of IndexRearranger who put a non-default codec, an index sort, OpenMode.CREATE, a soft-deletes field, a parent field or an older creation major in the config they pass. A configured codec or index sort is ignored on every run, with assertions on and off, at any size and with no timing condition. The other settings show only in their scenario: OpenMode.CREATE when the output already holds an index, the soft-deletes and parent fields when the input has such a field, the creation major when the input was created with an older major. A default config is not affected. The tool is experimental and in lucene/misc, so it is opt-in.
• Silent. The copy's codec and index sort differ from the configured ones, and so does its doc order within segments. The class's own use case, testing search performance after a change to indexing, then measures the default codec instead of the configured one when the change is chosen through the config. A writer configured with the original sort cannot open the copy.
• Loud. OpenMode.CREATE reruns, soft-deletes indexes, parent-field indexes and indexes created with the previous major fail with a clean exception. For the soft-deletes and parent fields there is no workaround through the API short of reimplementing execute() in a subclass (its helper methods are private); for the codec only the global, static Codec.setDefault.
• Not affected. Search results on the copy, requested deletes, the input index (only read).
• How likely: plausible. Upstream's own TestIndexRearranger and TestBinaryDocValueSelector pass a config with an index sort and OpenMode.CREATE, but the tool is little used, the codec and sort loss is invisible unless someone inspects the output, and nobody has reported it since 10.0.0.

One document, one input segment written with the same sort (the realistic case of rearranging a sorted index), and one live selector from BinaryDocValueSelector.createLiveSelectorsFromIndex. The test sits in org.apache.lucene.misc.index for locality and uses no package-private API. It starts no threads of its own; execute() runs the selector on its internal pool. newDirectory() is randomized; the test failed the same way with seeds DEADBEEF and C0FFEE.

The other symptoms (codec, OpenMode.CREATE, soft-deletes field, parent field) share the root cause and were executed in the hunt's longer tests (the creation major in a fact-check probe); they are listed in the table above and are not part of this repro.

Regression test: this same test; it passes with the recommended fix.

cp $REPRO/TestIndexRearrangerKeepsIndexSort.java $LUCENE/lucene/misc/src/test/org/apache/lucene/misc/index/
cd $LUCENE && ./gradlew :lucene:misc:test --tests TestIndexRearrangerKeepsIndexSort -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed)
> Task :lucene:misc:test FAILED

TestIndexRearrangerKeepsIndexSort > testIndexSortOfConfigIsKept FAILED
    java.lang.AssertionError: expected:<<long: "ord">> but was:<null>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:3151E5276FE88533]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.misc.index.TestIndexRearrangerKeepsIndexSort.testIndexSortOfConfigIsKept(TestIndexRearrangerKeepsIndexSort.java:50)

1 test completed, 1 failed

# -Ptests.asserts=false: the same AssertionError
# with the fix applied: BUILD SUCCESSFUL (assertions on and off)

• Contract. "@param config index writer config" (IndexRearranger.java:85), kept in a protected final field (:70). The class javadoc's use case is testing search performance "after a change to indexing" (:57-60).
• The tool's author on the original contract. Before 10.0, when the tool was new, Patrick Zhai, who wrote it (LUCENE-9694), told a user on java-user (2021-05-24, "Multiple merge-runs from same set of segments") that "the IndexRearranger let you pass in an IndexWriterConfig so that you could set it there", about setIndexSort.
• Tests. TestIndexRearranger.java:209-214 and TestBinaryDocValueSelector.java:123-128 still build the config they pass with OpenMode.CREATE, NoMergePolicy and setIndexSort(ord). TestIndexRearranger's helper is unchanged from 9.x, where those settings were applied (9.12.3 TestIndexRearranger.java:123-128); TestBinaryDocValueSelector was added by #11815 itself, with the same helper. They check segment contents and live docs, never the output's sort or codec.
• History. #11815 needed a second writer to apply deletes, and one config cannot serve two writers (IndexWriterConfig.java:125-131). Its issue (#11814), body, comments and review comments never mention the config, codec, index sort or open mode; its only compatibility remark ("This change is technically not backwards compatible") names "changes to the rearrange API" and deleted documents no longer disappearing from the rearranged index ("They become live instead"). CHANGES.txt:2210 says only "GITHUB#11814: Support deletions in IndexRearranger."; MIGRATE.md has no entry. So the loss reads as an unintended side effect; that is an inference, the authors did not say so.
• Precedent. PKIndexSplitter in the same package solves the same two-writer problem by taking one caller config per output (PKIndexSplitter.java:62-68, LUCENE-3296, #4369).

The strongest case: the class is @lucene.experimental; "index writer config" promises nothing about which settings apply; even 9.x overrode the merge policy; the tool's purpose is geometry (segment membership and order), which the upstream tests check; and one config cannot serve two writers, so fresh configs were a pragmatic choice.

Why it fails: the merge-policy override is required by the algorithm and commented, while dropping everything but the one setting addIndexes never uses is a choice nobody stated; #11815 lists its incompatible changes and this is not among them; the upstream tests still pass a sort and an open mode; the javadoc's own use case needs the codec kept; and PKIndexSplitter shows the two-writer constraint has a known solution. Cross-model reviews (OpenAI GPT-5.6) and an intended-behaviour verifier rated the works-as-designed case weak.

Option A was built and run; B avoids its merge-scheduler coupling; C changes the API; D is not recommended.

In execute() (IndexRearranger.java:119-120) replace the two fresh configs: the rearrange writer uses config itself, and the deletes writer gets a fresh config that carries over the output's settings. Of those, only the soft-deletes and parent fields must match when that writer opens the output (IndexWriter.java:1267-1278, FieldInfos.java:417-418), and an index sort, when one is set (IndexWriter.java:1213-1229); the codec and similarity are copied for consistency, as that writer only marks deletes. The rest of execute(), including setMergePolicy(NoMergePolicy.INSTANCE) on both configs, is unchanged.

// The caller's config writes the rearranged segments (as before the deletions support).
IndexWriterConfig createSegmentsConfig = config;
// A writer config can only be used once, so the deletes pass gets a fresh config that
// carries over the output's settings.
IndexWriterConfig applyDeletesConfig =
    new IndexWriterConfig(config.getAnalyzer())
        .setCodec(config.getCodec())
        .setSimilarity(config.getSimilarity())
        .setOpenMode(IndexWriterConfig.OpenMode.APPEND);
if (config.getIndexSort() != null) {
  applyDeletesConfig.setIndexSort(config.getIndexSort());
}
if (config.getSoftDeletesField() != null) {
  applyDeletesConfig.setSoftDeletesField(config.getSoftDeletesField());
}
if (config.getParentField() != null) {
  applyDeletesConfig.setParentField(config.getParentField());
}

Trade-offs

• Merge scheduler. The first writer also takes the caller's merge scheduler. With NoMergeScheduler, addIndexes(CodecReader...) blocks since 9.4.0 (a separate bug in the registry, LFV-586); today's fresh configs avoid that only because they use the default ConcurrentMergeScheduler (LiveIndexWriterConfig.java:137). On 9.4.0 to 9.12.3 the tool had the same exposure (code reading).
• Single use. As in 9.x, the caller's config is attached to the first writer and changed (setMergePolicy), so it, and the IndexRearranger holding it, can run execute() once (IndexWriterConfig.java:125-131; code reading, not executed). Today a second execute() into the same output fails anyway (the OpenMode.CREATE row above).
• Other settings come back too. As in 9.x, the first writer also applies the caller's commitOnClose, deletion policy, setIndexCommit and setIndexCreatedVersionMajor (the last one lets an input created with the previous major be rearranged, as the 9.12.3 probe shows for 9.x; not run with the fix). With commitOnClose(false) the first writer rolls back on close (IndexWriter.java:1363-1368), so the rearranged segments are not committed before finalizeRearrange reads the output (code reading, not executed); the PR should handle it (for example reject it or commit explicitly) or document it.
• Maintenance. The settings copied into the second config must be kept in step with new IndexWriterConfig settings that must match an existing index.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-30):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off; the closest suites TestIndexRearranger (3 tests) and TestBinaryDocValueSelector (2 tests) pass; the whole :lucene:misc:test passes (24 classes, 317 tests, 0 failures, 9 skipped); checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for misc pass.
• Earlier, with the same change shadowed into the stock jars on 645d4fe14a20, the hunt's longer tests for the codec, OpenMode.CREATE, soft-deletes and parent-field symptoms passed (4 of 4 and 6 of 6).
• After these runs the diff's second code comment was reworded (comment only, same +18/-2): it applies cleanly to 645d4fe14a20, and the minimal test passes with it, assertions on and off.
• Scope: validated for these tests and settings only; it is not a general config-preservation patch.

Keep two fresh configs and copy codec, index sort, soft-deletes field and parent field into both (similarity too, for consistency); the first gets the caller's open mode and creation major version, the second APPEND.

Trade-offs

Leaves the caller's config untouched and reusable and keeps the default merge scheduler (no LFV-586 exposure). Copying only these settings is a deliberate choice, not just a list to maintain: the first writer then does not do everything the caller's config would (for example its commitOnClose, deletion policy or merge scheduler), which may be what maintainers want; settings that are not copied stay silently ignored. Not run as a separate test; option A's second config is this copy (without the creation major, which only matters when the output is created).

Like PKIndexSplitter (PKIndexSplitter.java:62-68), let the caller supply a config for each writer.

Trade-offs

Fully explicit, but changes a public (experimental) API, and callers must make the two configs agree. Not run.

Javadoc change only.

Trade-offs

Leaves soft-deletes and parent-field indexes impossible to rearrange and the codec impossible to choose, and contradicts the upstream tests and the 9.x behaviour.

Method: code reading, then executed public-API tests (the hunt classes it as non-formal).

• Probe. A small program printed the input and output segments' codec and index sort: IN _0 ... sort=<long: "ord">, OUT _0 ... sort=null; reopening with the sort, a second run and a soft-deletes input threw.
• Release differential. Two JUnit classes on the public API (index sort and codec; OpenMode.CREATE and soft deletes) fail 4 of 4 on main and every 10.x release, and their 9.x-API variants pass on every release from 8.9.0 to 9.12.3. A commit-level A/B on identical 9.4.2 jars pins the change to #11815.
• The minimal repro above is the index-sort test cut down to one document and one assertion.