MIGRATE.md's 11.0 upgrade example says forceMerge(1) rewrites all segments, but it leaves a single-segment index in its old format

Lucene main's MIGRATE.md, in the 11.0 section "Relaxed Index Upgrade Policy", tells users who open an index created with Lucene 9.x or 10.x how to upgrade its segment formats: Scenario 1 says "Optional: Call forceMerge() or use UpgradeIndexMergePolicy to upgrade segment formats immediately", and the "Upgrade Example" opens new IndexWriter(dir, config) and calls writer.forceMerge(1); // Rewrites all segments to latest format.

forceMerge(1) only promises to leave at most one segment, and the merge policy picks the merges. The default TieredMergePolicy returns no merge for an index that already has one segment and no deletions ("already merged"), so the old segment is never rewritten and the call returns normally. On Lucene's own backward-compatibility index moreterms.9.0.0 (one segment, 50 documents) the segment is still version 9.0.0 with codec Lucene90 afterwards; without lucene-backward-codecs that index then fails to open with IndexFormatTooOldException. Indexes with two or more segments or any deletion are rewritten as promised.

The no-op is old and deliberate (LUCENE-3082 added IndexUpgrader and UpgradeIndexMergePolicy for exactly this case in 2011), so the code is correct and only the guide is wrong. The text came with #15012 (2025-09-18) and is on main and on the 11.0 release branch branch_11x, not in any release. Fix: wrap the configured merge policy in UpgradeIndexMergePolicy in the example and correct Scenario 1; with that one line the same test passes.

• Line 153 (Scenario 1): "Optional: Call forceMerge() or use UpgradeIndexMergePolicy to upgrade segment formats immediately".
• Lines 169-183, "Upgrade Example": "// Opening an index created with Lucene 9.x or 10.x in Lucene 11.x+", then try (IndexWriter writer = new IndexWriter(dir, config)) { writer.forceMerge(1); // Rewrites all segments to latest format } (line 179). config is not defined in the snippet.
• On today's upstream main and branch_11x the same block sits 14 lines lower (lines 167 and 193), unchanged.

• IndexWriter.forceMerge javadoc (IndexWriter.java:2187-2189): "Forces merge policy to merge segments until there are <= maxNumSegments. The actual merges to be executed are determined by the MergePolicy." It ties the call to static indexes: "usually you should only call this if the index is static" (:2191-2193).
• forceMerge marks every current segment as an original segment to merge (IndexWriter.java:2251-2255). new IndexWriterConfig() uses TieredMergePolicy (LiveIndexWriterConfig.java:143).
• TieredMergePolicy.findForcedMerges (TieredMergePolicy.java:834-847): with no deletions, maxSegmentCount == 1, one eligible segment and that segment in segmentsToMerge, it logs "already merged" and returns null. The segment's version and codec play no part.
• UpgradeIndexMergePolicy (UpgradeIndexMergePolicy.java:26-31) "is used for upgrading all existing segments of an index when calling IndexWriter#forceMerge(int)": it selects every segment whose version is not Version.LATEST (:68-70) and merges the ones its delegate declines itself (:117-137). IndexUpgrader.upgrade() uses it (IndexUpgrader.java:180, :196).
• The LogMergePolicy variants also skip a lone segment without deletions when its compound-file flag already matches (MergePolicy.isMerged, MergePolicy.java:750-758).

On the stock main jars, 10 index states (Lucene's own fixtures plus states derived from them) x 6 policy setups = 60 runs. A decision model written from the code predicts all 60. The guide's claim holds in 50 and fails in 10, all on a single segment without deletions: both TieredMergePolicy setups on the 9.0.0, 9.12.3 and 10.5.2 fixtures, and both LogMergePolicy setups on the compound 9.12.3 and 10.5.2 fixtures. The other 8 single-segment, deletion-free runs are rewritten (UpgradeIndexMergePolicy and IndexUpgrader on all three, and the LogMergePolicy setups on the non-compound 9.0.0 fixture). UpgradeIndexMergePolicy and IndexUpgrader rewrite all 20 of their cases; document counts are unchanged in all 60.

• Who. Users upgrading an existing 9.x or 10.x index to 11.0 by the guide's optional "upgrade segment formats" step, whose index already has a single segment without deletions. forceMerge's javadoc ties the call to static indexes ("usually you should only call this if the index is static"), and a static index force-merged to one segment has exactly this shape.
• Default vs opt-in. The default IndexWriterConfig (TieredMergePolicy), the example as written. The upgrade step itself is optional.
• Silent. The call returns normally; only an enabled InfoStream shows "already merged".
• What stays wrong. For 9.x and 10.0-10.3 indexes the segment keeps a codec that lives in lucene-backward-codecs; an application that drops that jar after "upgrading" gets IndexFormatTooOldException ... Could not load codec 'Lucene90' ... Did you forget to add lucene-backward-codecs.jar?. After the corrected example the same index opens with lucene-core alone (numDocs=50). 10.4 and 10.5 segments already use Lucene104 and keep only an older version stamp and older sub-format versions.
• Not affected. No data is lost or changed, the index stays readable, and how long it stays openable is unchanged (that depends on the creation version and MIN_SUPPORTED_MAJOR). Multi-segment indexes and indexes with deletions are rewritten.
• Scale and timing. Deterministic for any such index the writer configuration can open (20 of 20 random seeds). Unreleased: main and the 11.0 release branch only, so it would first reach users in 11.0.0.

The test must sit in package org.apache.lucene.backward_index of the backward-codecs test sources: getDataInputStream("moreterms.9.0.0.zip") loads the fixture next to the test class.

Regression test: optional. If wanted, it is this same test with the corrected example's one extra line, config.setMergePolicy(new UpgradeIndexMergePolicy(config.getMergePolicy()));, which passes with the fix and pins the recipe the guide shows (the test then also imports org.apache.lucene.index.UpgradeIndexMergePolicy). Upstream already tests upgrades through IndexUpgrader.

# ./gradlew :lucene:backward-codecs:test --tests org.apache.lucene.backward_index.TestMigrateUpgradeExample -Ptests.seed=DEADBEEF
# main 645d4fe14a20, assertions on (-Ptests.asserts=false gives the same failure)
TestMigrateUpgradeExample > testForceMergeUpgradesSingleSegmentIndex FAILED
    java.lang.AssertionError: expected:<11.0.0> but was:<9.0.0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:5CDB23FA4FF1066B]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.backward_index.TestMigrateUpgradeExample.testForceMergeUpgradesSingleSegmentIndex(TestMigrateUpgradeExample.java:40)
...
1 test completed, 1 failed

# the index before and after, same jars (finder's matrix)
default(TMP):  before _0 ver=9.0.0 codec=Lucene90 maxDoc=50 del=0  ->  after _0 ver=9.0.0 codec=Lucene90 (unchanged)
Upgrade(TMP):  before _0 ver=9.0.0 codec=Lucene90 maxDoc=50 del=0  ->  after _1 ver=11.0.0 codec=Lucene104 maxDoc=50

• Unconditional text. Line 179 credits the rewrite to forceMerge(1) itself, and line 153 offers "Call forceMerge() or use UpgradeIndexMergePolicy" as two alternatives.
• No recorded decision to accept this limitation. markrmiller's proposal comment on #13797 says "Merging rewrites segments to the current codec" and lists "optional forceMerge" as the upgrade step. Uwe Schindler's comment there named IndexUpgrader / UpgradeIndexMergePolicy as the existing rewrite path, but that did not reach MIGRATE.md. The #15012, #16741 and #16756 threads never mention forceMerge, single segments or IndexUpgrader.
• The code is intended, and documented as such elsewhere. LUCENE-3082 (#4155, 2011) added IndexUpgrader and UpgradeIndexMergePolicy; as Uwe Schindler put it there, "The optimize() approach fails if your index is already optimized" and "optimizing a one-segment index is a no-op without the special merge-policy". LUCENE-5526 (#6589, 2014) says forcing a merge of a lone segment needs "a custom merge policy such as UpgradeIndexMergePolicy". LUCENE-5481 (#6544, 2014) removed unneeded forced merges, and BaseMergePolicyTestCase.testForceMergeNotNeeded fails any policy that merges when nothing is needed. Lucene's own upgrade tests use IndexUpgrader, not a bare forceMerge.
• Every completed review called it real: an independent reproducer, a reviewer from another model family (OpenAI GPT-5.6) under lenses, a devil's-advocate check and the adjudication.

Strongest case for "intended" or "noise": forceMerge is a segment-count operation, never a format converter, as its javadoc, TieredMergePolicy's "already merged", LUCENE-5481 and testForceMergeNotNeeded all say. The snippet leaves config undefined, so the reader may be meant to supply an UpgradeIndexMergePolicy. Line 152 already says segments are upgraded "gradually through normal merging", and the index works either way.

Why it fails: the first point defends the code, which nobody proposes to change; it does not make the sentence true. The undefined-config reading is ruled out by the guide itself, because line 153 presents forceMerge() as an alternative to UpgradeIndexMergePolicy and the comment on line 179 is unconditional. "It still works" only lowers the severity, which is already Low: a reader who runs the guide's recipe is told the index is in the latest format when it is not. Verdict: a real, minor documentation defect.

One fix was validated, with the two-hunk diff applied to 645d4fe14a20; the alternatives were not run.

Two hunks in lucene/MIGRATE.md; it also applies to today's upstream main (14 lines lower).

-- Optional: Call `forceMerge()` or use `UpgradeIndexMergePolicy` to upgrade segment formats immediately
+- Optional: Use `IndexUpgrader` or `UpgradeIndexMergePolicy` to upgrade segment formats immediately (a plain `forceMerge(1)` does not rewrite an index that already has a single segment without deletions)
@@
     // Optional: Upgrade segment formats
+    config.setMergePolicy(new UpgradeIndexMergePolicy(config.getMergePolicy()));
     try (IndexWriter writer = new IndexWriter(dir, config)) {
-        writer.forceMerge(1); // Rewrites all segments to latest format
+        writer.forceMerge(1); // Rewrites all segments written by older versions to the latest format
     }

Trade-offs

• UpgradeIndexMergePolicy rewrites only old segments: an index that also has new-format segments can end with more than one segment, all at the latest version. That is what an upgrade step needs.
• Its javadoc warns it may reorder documents if the index was partially upgraded; not an issue for a single segment.
• Its own class-javadoc example calls a removed constructor (LFV-1021), so a reader the guide sends there meets that error too.
• With a SoftDeletesRetentionMergePolicy inside, its leftover merge bypasses retention (LFV-617); at most a caveat.
• UpgradeIndexMergePolicy is marked @lucene.experimental; it has existed since 2011 and is what IndexUpgrader uses.
• The snippet stays a sketch: it still leaves config to the reader, and real code also needs import org.apache.lucene.index.UpgradeIndexMergePolicy. Whatever config is must match the index (for example its index sort or parent field), or new IndexWriter(dir, config) fails before forceMerge runs; that holds for the current example too.

What the checks validated

• The diff applies cleanly to 645d4fe14a20 and, with an offset of 14 lines, to today's upstream main.
• The minimal test with the fix's one extra line passes 1 of 1 through the same Gradle command with assertions on and off; the stock example fails it, so the test detects the bug. After the corrected example the index opens with lucene-core alone.
• TestIndexUpgradeBackwardsCompatibility, TestUpgradeIndexMergePolicy and TestTieredMergePolicy pass (340 + 5 + 17 tests, 0 failures); a sanity check, since the change is documentation only.
• ./gradlew :lucene:documentation:markdownToHtml succeeds and the rendered page shows the corrected text.

new IndexUpgrader(dir).upgrade() or the command line java -cp lucene-core.jar:lucene-backward-codecs.jar org.apache.lucene.index.IndexUpgrader indexDir. Simple and already tested upstream. A 9.x or 10.0-10.3 index needs lucene-backward-codecs on the class path to be read at all: with lucene-core alone the command fails with IndexFormatTooOldException ("Could not load codec 'Lucene90'"); with both jars it prints "All segments upgraded to version 11.0.0". (IndexUpgrader's own javadoc shows only lucene-core.jar.)

Trade-offs

It refuses an index with more than one commit unless told to delete prior commits, keeps only the last commit, and its convenience constructors use a bare IndexWriterConfig, which drops an index sort and cannot upgrade an index with a parent field (LFV-615, known upstream). The recommended option keeps the application's own config.

Say that forceMerge(1) rewrites only the segments it merges, and that an index already at one segment without deletions needs UpgradeIndexMergePolicy or IndexUpgrader.

Trade-offs

Correct, but leaves the guide without a working recipe for the single-segment, deletion-free shape of a force-merged static index.

Change TieredMergePolicy (and the other policies) to merge a lone segment whose version is not current.

Trade-offs

Contradicts LUCENE-5481 and testForceMergeNotNeeded, changes the cost of every forceMerge, and duplicates UpgradeIndexMergePolicy. Every completed review agreed the code should not change.

• First signal. A pass that looked for upgrade problems ran the MIGRATE.md example on the stock main 645d4fe14a20 jars against indexes written by the 9.12.3 jar (2026-10-29): the single-segment index stayed Lucene912 / 9.12.3, a three-segment index was rewritten.
• Hunt. A decision model of "forceMerge(1) leaves every segment at the latest version", written from the code, was checked against 60 executed (index state x merge policy) runs: the model matched 60 of 60, the guide's claim 50 of 60. The one-test repro uses Lucene's own moreterms.9.0.0 fixture.
• Tier. Non-formal: an executed check of a documentation claim, with a model-based cross-check.