After a writer reopen, forceMerge(1) rewrites an already merged segment: MergePolicy.isMerged reads the compound-file setting from the shared codec instance, not the writer's (main and branch_11x, unreleased)

Since #15295 the compound-file settings (setShouldUseCompoundFile, the doc and byte thresholds, the maximum compound segment size) are fields of each CompoundFormat instance, and Lucene104Codec holds one per codec instance. MIGRATE.md tells users to set them on iwc.getCodec().compoundFormat().

A merge decides the compound flag of its new segment with the writer's codec. MergePolicy.isMerged, which decides whether a lone segment needs a forced merge at all, asks the codec the segment carries. For a segment the writer merged in this session that is the writer's codec; for a segment read from a commit it is Codec.forName(name), the shared SPI instance, whose settings are the defaults unless someone changed that instance. So a writer with its own codec instance and non-default compound settings sees its merged segment as "not merged" after every reopen, and the next forceMerge rewrites it through LogMergePolicy.findForcedMerges. The rewrite uses the writer's codec again, so this repeats once in every writer session that force-merges. No result changes; it costs I/O. It is on main and branch_11x only, unreleased.

isMerged ends with info.info.getCodec().compoundFormat().useCompoundFile(size(info, mergeContext), this) == info.info.getUseCompoundFile() (MergePolicy.java:756-757). The settings are per-instance fields (CompoundFormat.java:48-57), and Lucene104Codec creates one CompoundFormat per instance (Lucene104Codec.java:75, 157-159). A merged segment gets config.getCodec() (IndexWriter.java:5334) and the merge asks it (IndexWriter.java:5683-5691). A segment read from a commit gets Codec.forName (SegmentInfos.java:407, 423, 561-564), which returns the cached SPI instance (Codec.java:118-119), by default the same object as Codec.getDefault() (Codec.java:59; Codec.setDefault at :141-153 can install another). Segments loaded at an older commit through IndexWriterConfig.setIndexCommit and segments copied by addIndexes(Directory...) also carry the Codec.forName instance (IndexWriter.java:1083-1103, 3130; read, not executed); a writer opened from an NRT reader instead clones that reader's in-memory segment infos (IndexWriter.java:1049-1053) and may keep an earlier writer's codec objects (read, not executed). The isMerged javadoc is partly stale: the body no longer checks its "same dir as the writer" part.

• The rewrite is written with the writer's codec, so the new segment has the writer's compound flag again. In the same session it carries the writer's codec and is kept, so one forceMerge call cannot loop; after the next reopen it carries the shared instance again (a reopen alone rewrites nothing; the first forceMerge(1) after it does). Executed: exactly 1 rewrite per writer session over 5 forceMerge(1) calls, in 3 sessions and 3 configurations.
• A writer that uses the default codec (with no Codec.setDefault call) shares one object with the segments it reads, so nothing goes wrong. Neither does an own instance left at the default settings.
• LogMergePolicy.findForcedMerges asks isMerged for a lone original segment (LogMergePolicy.java:181-203, 352, 379; also :241, 257, 289 for size-limited and multi-segment forced merges, read only). The default TieredMergePolicy first drops every segment that is not in segmentsToMerge (TieredMergePolicy.java:775-777) and then reports a lone remaining segment as already merged before it asks (:836-843), so its isMerged call is never reached. Background merges and forceMergeDeletes never ask isMerged.

With such a policy, a lone segment without deletions is rewritten when the writer codec's compound decision for its size differs from that of the shared Codec.forName instance as it currently is (the defaults unless changed):

• setShouldUseCompoundFile(false) (the MIGRATE.md recipe): segments the defaults would make compound, up to 64 MB with byte-size policies or 65,536 docs with LogDocMergePolicy. Measured over 3 sessions: a 60,000-doc LogDocMergePolicy segment rewritten every time; a 70,000-doc one and an 80.4 MB LogByteSizeMergePolicy segment kept.
• Raised thresholds: setCfsThresholdDocSize(100000) with a 70,000-doc compound segment, rewritten after a reopen; setCfsThresholdByteSize(1 GB) with an 80.4 MB compound segment, rewritten in all 3 sessions.

• Who is hit. Applications on Lucene main (11.0.0-SNAPSHOT) that use LogByteSizeMergePolicy, LogDocMergePolicy or a subclass, give the writer a codec instance other than the one Codec.forName returns (new Lucene104Codec(Mode), any per-field subclass, an instance installed with Codec.setDefault(...), or a separately constructed instance of a custom codec registered through SPI; executed only with the first and with TestUtil.getDefaultCodec(false)), set compound-file settings on it whose answer differs from the shared instance's for the segment's size, and call forceMerge after reopening the writer.
• Default configuration is not hit: the default TieredMergePolicy never reaches its isMerged call (it drops segments outside segmentsToMerge first), and the default codec, with no Codec.setDefault call, is the same object the segments are read with.
• Cost: one needless rewrite of an already merged segment per writer session that calls forceMerge, with the transient disk space of any forced merge. Small under the MIGRATE.md recipe (segments up to 64 MB, or 65,536 docs with LogDocMergePolicy); large segments only with raised thresholds (an 80.4 MB segment was rewritten in every session under a 1 GB threshold). The runs counted rewrites and checked content; they did not measure bytes written, I/O time, commit or fsync churn, or effects on open readers and caches.
• Silent: nothing logs it; live documents, stored values, doc values, deletions and CheckIndex are unchanged by the rewrite.
• Not released yet: it is on main and branch_11x (created from main on 2026-10-29); 10.6.0, earlier releases and branch_10x are not affected, so it can be fixed before 11.0.0 ships.

The test sits in org.apache.lucene.index by upstream convention but uses nothing package-private. It builds IndexWriterConfig directly (no randomized writer settings) and is single-threaded. Session 1 flushes _0 and _1 and merges them into the non-compound _2; session 2 opens a new writer with the same codec instance and calls forceMerge(1) on the single merged segment. The same test is the suggested regression test; it passes with the fix below and passes the core format and lint checks as written.

Not asserted here (shown by the verification runs): one rewrite per writer session, the opposite direction (a raised doc threshold rewrites a 70,000-doc compound segment), and that TieredMergePolicy keeps the segment.

# stock main 645d4fe14a20, seed DEADBEEF (same with -Ptests.asserts=false)
TestForceMergeKeepsMergedSegmentAfterReopen > testForceMergeOfMergedIndexAfterReopen FAILED
    org.junit.ComparisonFailure: segment after forceMerge(1) expected:<_[2]> but was:<_[3]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:6AE676CFFF132E72]:0)
        at app//org.junit.Assert.assertEquals(Assert.java:117)
        at app//org.apache.lucene.index.TestForceMergeKeepsMergedSegmentAfterReopen.testForceMergeOfMergedIndexAfterReopen(TestForceMergeKeepsMergedSegmentAfterReopen.java:51)
1 test completed, 1 failed

# with the recommended fix: BUILD SUCCESSFUL, 1 test, 0 failures (assertions on and off)

• Documented intent. The isMerged javadoc (since LUCENE-5038, 2013; partly stale, as the body no longer checks the "same dir" part): "Returns true if this single info is already fully merged (has no pending deletes, is in the same dir as the writer, and matches the current compound file setting" (MergePolicy.java:746-749). On main the only place to configure that setting is the codec, and MIGRATE.md's recipe is CompoundFormat cfs = iwc.getCodec().compoundFormat(); cfs.setShouldUseCompoundFile(false); (MIGRATE.md:322-324). LiveIndexWriterConfig.setUseCompoundFile points to CompoundFormat for merges.
• One decision, two answers. The merge uses the writer's codec; isMerged uses the segment's. Executed: a second forceMerge(1) in the same session keeps the segment, and the first one after a reopen rewrites it.
• Regression. Before #15295 all three compound-file decisions asked the writer's merge policy, so they always agreed; 10.5.2 and 9.12.3 keep the segment with the equivalent merge-policy settings (executed).
• Upstream uses private instances itself. TestUtil.getDefaultCodec(boolean) sets compound settings on a fresh Lucene104Codec, and newIndexWriterConfig randomizes them on the config's codec; with that helper and LogDocMergePolicy the bug reproduces (segments _4, _5, _6 over 3 sessions).
• Precedents fixed as bugs: #15616 (compound settings lost through a different CompoundFormat instance), LUCENE-5481 / #6544 (forceMerge ran an unneeded merge on a merged segment), LUCENE-9905 / #176 (a merge used the SPI default instead of configured format parameters).
• No upstream test asserts whether forceMerge keeps an already merged single segment after a writer reopen. Tests do reopen writers with per-codec compound settings, for example TestDeletionPolicy (TestDeletionPolicy.java:333, 353, 357-359), but only to merge an index of more than one segment (:342-348).

The strongest case for "intended": forceMerge promises only "at most n segments" and leaves the choice to the merge policy; MergePolicy and CompoundFormat are @lucene.experimental; a segment's codec is the codec that wrote it, so asking it is self-consistent; Lucene104Codec(Mode) documents its options for "newly flushed/merged segments"; the MIGRATE.md recipe works with the default codec; since LUCENE-8330 the segment's codec is the only codec isMerged can reach; and in an index whose segments use several codecs, a maintainer may prefer to judge each segment by its own codec.

Why it fails: nothing documents that the settings work only on the named instance, and setting them there changes every writer in the JVM; compression-mode and per-field codecs are necessarily separate instances; the merge itself uses the writer's codec, so asking the segment's codec does not answer "would a merge change this segment?"; the outcome flips across a reopen with an identical config; and 10.x behaves as expected. The checkers who built this case rated it moderate and weak to moderate, and both rejected it.

One option was built and validated in a scratch checkout of main 645d4fe14a20. A clean fix needs a design choice, because MergeContext exposes no codec and MergePolicy has had no writer access since LUCENE-8330; the other options come from the verifiers and the adjudication.

+35/-1 in three files of lucene/core. A default method MergeContext#getCompoundFormat() returns null (unknown); IndexWriter overrides it with config.getCodec().compoundFormat(); CachingMergeContext, which wraps the writer for findForcedMerges, delegates; isMerged asks it and falls back to the segment's codec when it is null. The check then asks exactly what the merge asks, so "merged" means "a merge would not change the compound flag". Writers on the default codec (with no Codec.setDefault call), and own Lucene104Codec instances with default settings, should see no change (the executed controls). Indexes whose segments use several codecs, and custom-named codecs, were not checked and need a compatibility review.

// MergePolicy.MergeContext
default CompoundFormat getCompoundFormat() {
  return null; // unknown: isMerged falls back to the segment's codec
}
// IndexWriter (CachingMergeContext delegates to it)
public CompoundFormat getCompoundFormat() {
  return config.getCodec().compoundFormat();
}
// MergePolicy.isMerged
CompoundFormat cf = mergeContext.getCompoundFormat();
if (cf == null) cf = info.info.getCodec().compoundFormat();
return delCount == 0
    && cf.useCompoundFile(size(info, mergeContext), this) == info.info.getUseCompoundFile();

Trade-offs

• A new experimental method on the public MergeContext interface, and so a public IndexWriter#getCompoundFormat().
• A MergeContext implemented outside Lucene keeps today's behaviour until it overrides the method; any wrapper must delegate it.

What the checks validated

• Stock 645d4fe14a20: the repro fails with assertions on and off. With the fix: it passes both ways.
• 19 merge-related core suites (TestLogMergePolicy, TestTieredMergePolicy, TestIndexWriterMergePolicy, TestCachingMergeContext, TestIndexWriterForceMerge, TestSizeBoundedForceMerge, TestForceMergeForever, TestMergePolicy, TestFilterMergePolicy, TestNoMergePolicy, TestUpgradeIndexMergePolicy, TestTemporalMergePolicy*, TestSoftDeletesRetentionMergePolicy, TestOneMergeWrappingMergePolicy, TestThreadedForceMerge, TestMergeCarryOverFromDisk, TestAddIndexes, the repro): 158 tests (7 skipped), 0 failures, assertions on and off.
• All core org.apache.lucene.index.* tests: 215 suites, 1,396 tests (56 skipped), 0 failures, assertions on and off.
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest and renderJavadoc of :lucene:core pass.
• The independent reproduction's 4-test JUnit, including the opposite direction (raised doc threshold, LogDocMergePolicy, 70,000-doc compound segment), passes 4 of 4 with the three fixed classes, assertions on and off; it fails 2 of 4 on stock.
• Not run: other modules, the full core suite, x86, MergeContext implementations outside Lucene, mixed-codec indexes.

If the context is the IndexWriter, or the package-private CachingMergeContext around it, ask w.getConfig().getCodec().

Trade-offs

Makes the repro pass (it was used as the root-cause check), but relies on instanceof against a package-private class and silently changes nothing for any other context. Not proposed.

For example onto IndexWriterConfig, where the merge and isMerged can both read them.

Trade-offs

Removes this bug class and the JVM-wide mutation of the default codec that the MIGRATE.md recipe causes with the default codec, but reverses part of #15295's API shortly before 11.0. Not run.

A javadoc and MIGRATE.md change only.

Trade-offs

A workaround, not a fix: it changes every writer in the JVM, and a writer with its own instance (a compression mode, a per-field subclass) would have to keep the same settings on its own instance and on the shared one, which is global and fragile. Not recommended.

Method: semi-formal (code reading witnessed by a tool, then an executed test). A hunt unit built a census of merge-policy hooks: for each hook, which code answers it. Reading the compound-file decisions raised the hypothesis; the unit's class-file call-site scanner (written on the JDK class-file API) listed the three calls of CompoundFormat#useCompoundFile and the codec each one asks: the two IndexWriter sites (mergeMiddle, addIndexesReaderMerge) ask the merged segment's codec, and MergePolicy#isMerged asks the existing segment's SegmentInfo#getCodec.

A public-API JUnit then showed the rewrite after a reopen on stock main, with two controls (the Codec.forName instance with the same settings; an own instance with default settings) that pass, and a shadow isMerged that asks the writer's codec made it pass. Independent checkers reproduced it with their own tests; the minimal one-method test above came last.