Upgrading an index with a single old segment drops the soft-deleted documents SoftDeletesRetentionMergePolicy keeps

SoftDeletesRetentionMergePolicy "allows to carry over soft deleted documents across merges"; applications use it to keep recently deleted or updated documents, for example as change history. Upgrading such an index with new IndexUpgrader(dir, config, false).upgrade(), or with UpgradeIndexMergePolicy wrapped around the retention policy as that class's javadoc recipe shows, drops every soft-deleted document of an old segment that the delegate merge policy declines to merge. With TieredMergePolicy as the delegate (the default, and the one in the example), that is always so on 7.5.0 and later for a lone old segment without hard deletes, the usual shape of a force-merged index. An index written by 10.5.2 with 13 documents (10 visible, 3 soft-deleted and retained by a match-all query) comes out with 10; with two old segments it keeps all 13. There is no error, and the upgrade keeps only its new commit, so only a backup made before the upgrade still has them; the visible documents, as seen by readers that apply soft deletes, do not change. Cause: UpgradeIndexMergePolicy.findForcedMerges rewrites the old segments its delegate did not select with its own new OneMerge(newInfos), and OneMergeWrappingMergePolicy, the base class of the retention policy, only wraps merges it returns itself. Data loss since 7.4.0, when soft deletes arrived; the leftover merge itself dates from 2011.

• Soft deletes: with IndexWriterConfig.setSoftDeletesField("soft"), softUpdateDocument, or updateDocValues on the soft-deletes field, hides a document by giving it a doc value in that field instead of marking it deleted. Readers obtained from the IndexWriter, and SoftDeletesDirectoryReaderWrapper, skip it (a plain DirectoryReader does not); a merge drops it unless the merge policy keeps it.
• SoftDeletesRetentionMergePolicy: a OneMergeWrappingMergePolicy that wraps each merge so that soft-deleted documents matching a retention query stay in the merged segment (SoftDeletesRetentionMergePolicy.java:36-46).
• OneMergeWrappingMergePolicy: "A wrapping merge policy that wraps the OneMerge objects returned by the wrapped merge policy" (OneMergeWrappingMergePolicy.java:23-25). Its four segment-based merge-selection methods return wrapSpec(in.find*Merges(...)) (:44-83); findMerges(CodecReader...), the addIndexes path, is inherited from FilterMergePolicy unwrapped, a separate gap not claimed here.
• UpgradeIndexMergePolicy and IndexUpgrader: the upgrade policy turns forceMerge into "rewrite every segment written by another version"; IndexUpgrader.upgrade() installs it around the configured policy (IndexUpgrader.java:180), keeps only the last commit (:181), runs forceMerge(1) (:196) and commits (:204).

1. The delegate is asked first. UpgradeIndexMergePolicy.findForcedMerges collects the old segments (UpgradeIndexMergePolicy.java:86-93) and asks its delegate, here the retention policy around TieredMergePolicy, to merge them (:101-102).
2. The delegate (TieredMergePolicy) declines a lone old segment without hard deletes. On the writer the upgrade has just opened, the segment is not pooled yet, so IndexWriter.numDeletesToMerge returns only hard deletes ("this is safe!", IndexWriter.java:6449-6455); TieredMergePolicy sees one segment with no deletes and returns nothing (logging "already merged" only with an InfoStream, TieredMergePolicy.java:835-847). With a retain-all query it would decline even with exact counting, so this estimate is not the defect; for a query that keeps only some soft-deleted documents it is a second ingredient, since exact counting would leave reclaimable deletes and TieredMergePolicy would merge (and the wrapper wrap) the segment itself (code reading).
3. The upgrade policy merges the leftovers itself. It logs "does not want to merge all old segments, merge remaining ones into new segment" (:117-125, only with an InfoStream) and adds spec.add(new OneMerge(newInfos)) (:136), a plain merge created outside the delegate. Its other methods still delegate; only the wrapForMerge decoration of this merge is bypassed.
4. The retention wrapper never sees it. Only merges returned through wrapSpec are wrapped. IndexWriter.mergeMiddle calls merge.wrapForMerge(reader); the plain merge returns the reader unchanged, and soft-deleted documents are kept only for a wrapped reader (IndexWriter.java:5497-5501). The new segment holds the live documents only.

With two old segments, TieredMergePolicy merges them itself, that merge comes back through wrapSpec, and every retained document survives. A probe logged the delegate's view (numDeletesToMerge=0 delCount=0 softDel=3), the plain MergePolicy$OneMerge returned for the segment, and maxDoc=10 soft=0 after the upgrade.

In every loss case the 10 visible documents (through readers that apply soft deletes) are unchanged and the index is consistent; only the retained soft-deleted documents disappear, also for a plain DirectoryReader, which could still find them before the upgrade. With a LogMergePolicy delegate, the delegate merges a lone non-compound-file old segment itself, wrapped; it declines a small compound-file one, whose compound-file flag already matches what the writer would write (MergePolicy.java:750-758), and that one loses the documents.

• Who. Applications that keep soft-deleted documents with SoftDeletesRetentionMergePolicy (opt-in; for example change history for replication or recovery, or recent versions for a time window) and upgrade the index format with IndexUpgrader(dir, config, deletePriorCommits), or with UpgradeIndexMergePolicy around the retention policy and forceMerge. Main's MIGRATE.md (11.0) suggests UpgradeIndexMergePolicy to "upgrade segment formats immediately". For a soft-deletes index the config constructor is the only way to run IndexUpgrader: the default constructors and the command line cannot open it.
• Trigger. Deterministic once an old segment has no hard deletes and the delegate does not select it. With TieredMergePolicy as the delegate (the default) that is always so on 7.5.0 and later for a lone old segment (a force-merged or small one-segment index), and for a partially upgraded index that holds one such segment; on 7.4.0 only for a non-compound-file one. With a LogMergePolicy delegate only a lone compound-file old segment is declined. Any retention query that matches at least one soft-deleted document can be affected, not only match-all; for partial retention the fresh writer's hard-deletes-only estimate is part of the trigger. No scale, concurrency or special data.
• Effect. Silent loss of the documents the retention policy exists to keep, permanent for the index: the upgrade keeps only its new commit, so the previous commit is deleted; only a backup or copy made before the upgrade still has them. The visible documents (through readers that apply soft deletes) and the hard deletes are unchanged and the index is consistent; no deleted document comes back. Anything that later reads the retained history, including a plain DirectoryReader search, which does not apply soft deletes, finds it missing.
• How rare. A judgement, not measured: the combination is uncommon, IndexUpgrader is optional, no Lucene test combines these classes, and Elasticsearch and OpenSearch (default branches, code reading) do not call Lucene's IndexUpgrader or UpgradeIndexMergePolicy.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<2> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.index.TestUpgradeIndexMergePolicyRetainsSoftDeletes.testLoneOldSegmentKeepsRetainedSoftDeletes(TestUpgradeIndexMergePolicyRetainsSoftDeletes.java:56)

No javadoc sentence literally promises that an upgrade keeps retained soft deletes; the case rests on consistency.

• Contracts. SoftDeletesRetentionMergePolicy: "allows to carry over soft deleted documents across merges ... allows to control when soft deletes are claimed by merges" (SoftDeletesRetentionMergePolicy.java:37-42). UpgradeIndexMergePolicy: "All other methods delegate to the base MergePolicy given to the constructor", "forceMerge does no longer really merge; it is just used to 'forceMerge' older segment versions away", and its recipe iwc.setMergePolicy(new UpgradeIndexMergePolicy(iwc.getMergePolicy())) puts the configured policy inside (UpgradeIndexMergePolicy.java:27-31, :36-42). IndexUpgrader(Directory, IndexWriterConfig, boolean) upgrades "using an IndexWriter using the given config" (IndexUpgrader.java:154-158) and its class javadoc expects a user policy ("The MergePolicy set by IndexWriterConfig may also reorder documents", :51-52). IndexUpgrader is not @lucene.experimental.
• The inconsistency. With the same tool, config and documents, two old segments keep 13 and one keeps 10; a lone segment with one hard delete keeps all its soft-deleted documents and one without loses them. With a LogMergePolicy delegate the outcome also depends on whether the lone old segment is a compound file.
• History. The leftover merge is from LUCENE-3082 (2011, trunk 600a50bb2ae; 3.2 via 535d974518b), six years before OneMergeWrappingMergePolicy (LUCENE-7688, 2017, master 6f9acb51549; 6.5.0 via 180614c1491) and seven before soft deletes (LUCENE-8233, 2018, 7.4.0); none of LUCENE-8233's 15 comments mentions upgrading. No evidence was found that this interaction was considered or later addressed.
• Tests. No upstream test combines IndexUpgrader or UpgradeIndexMergePolicy with SoftDeletesRetentionMergePolicy; TestUpgradeIndexMergePolicy runs the generic merge-policy tests over a plain TieredMergePolicy.
• Upstream remark. On LUCENE-8264 (#9311) a committer (@s1monw) pointed to UpgradeIndexMergePolicy for upgrades and to OneMergeWrappingMergePolicy "similar to what we do in the soft delete case in SoftDeletesRetentionMergePolicy", presenting them as composable.

The intended-behaviour verifier built the strongest case for "not a bug": OneMergeWrappingMergePolicy only promises to wrap the merges its delegate returns, so the leftover merge was never in scope; IndexWriterConfig.setSoftDeletesField says documents are retained "if the merge policy modifies the live docs of a merge reader" (IndexWriterConfig.java:551-552), and the writer's top policy, UpgradeIndexMergePolicy, does not; a direct user can put the retention policy outermost, which works; @mikemccand wrote on #8574 that the upgrade policy "was really designed for one-off usage via the IndexUpgrader tool"; the merge-policy classes involved are @lucene.experimental; and only invisible documents are lost.

Why it fails: IndexUpgrader forces the losing nesting itself while saying it uses the given config, and its default constructor and command line cannot open a soft-deletes index, so the tool offers no safe way; the UpgradeIndexMergePolicy recipe nests the same way; the IndexWriterConfig sentence refers to the policy the user configured, not one the tool put around it; "one-off usage via the IndexUpgrader tool" is exactly this case; and @lucene.experimental permits API changes, not silently dropping documents a configured policy keeps. The verifier, the independent reproducer and cross-model reviews rated the case moderate. The cross-model fact check found this rebuttal mostly fair but noted that "using the given config" and "All other methods delegate" carry limited contractual weight (IndexUpgrader already replaces the merge and deletion policies); the strongest argument is the behavioural inconsistency. Verdict: a defect; a maintainer may still call it a composition gap.

Two fixes were applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF): the three repro classes (11 tests; stock: 4 failures, assertions on and off) and 14 closest upstream classes in core, backward-codecs (which runs IndexUpgrader on real old-version indexes) and misc. Either fix makes every repro pass. Which one, or another design, is the maintainers' call.

MergePolicy.wrapMerge returns the merge unchanged by default ("decorates a merge that a policy wrapping this one created itself"); FilterMergePolicy delegates it; OneMergeWrappingMergePolicy applies its operator after its delegate's; UpgradeIndexMergePolicy passes its leftover merge through it. Patch: optionB-mergepolicy-wrapmerge-hook.diff, +22/-1 in four files.

// MergePolicy: by default a policy does not decorate merges created around it
public OneMerge wrapMerge(OneMerge merge) { return merge; }
// FilterMergePolicy
@Override public OneMerge wrapMerge(OneMerge merge) { return in.wrapMerge(merge); }
// OneMergeWrappingMergePolicy (and so SoftDeletesRetentionMergePolicy)
@Override public OneMerge wrapMerge(OneMerge merge) { return wrapOneMerge.apply(in.wrapMerge(merge)); }
// UpgradeIndexMergePolicy.findForcedMerges, the leftover merge
spec.add(in.wrapMerge(new OneMerge(newInfos)));   // was: spec.add(new OneMerge(newInfos))

Trade-offs

• A new public expert method on the @lucene.experimental MergePolicy; any decorating policy can opt in. BPReorderingMergePolicy (misc) would need its own override, not written here, to reorder this merge.
• MergePolicy is widely subclassed: experimental status lowers compatibility expectations, but a clash with an existing downstream method of the same name, and the order in which mixed custom decorators apply, deserve review; the 450 tests cover no outside policies.
• No CHANGES entry, javadoc build or full check was run; no performance effect expected (one call per upgrade merge).

What the checks validated

• Stock: the three repro classes run 11 tests with 4 failures, assertions on and off.
• With B: 11 of 11 pass with assertions on and off. With the 14 closest classes (TestUpgradeIndexMergePolicy, TestOneMergeWrappingMergePolicy, TestSoftDeletesRetentionMergePolicy, TestSoftDeletesDirectoryReaderWrapper, TestPendingSoftDeletes, TestFilterMergePolicy, TestMergePolicy, TestTieredMergePolicy, TestIndexWriterForceMerge, TestIndexWriterMergePolicy, TestSizeBoundedForceMerge; TestIndexUpgradeBackwardsCompatibility, TestEmptyIndexBackwardsCompatibility; TestBPReorderingMergePolicy): 17 classes, 450 tests, 76 skipped, 0 failures. TestFilterMergePolicy checks that FilterMergePolicy overrides every MergePolicy method.
• :lucene:core:checkGoogleJavaFormat passes with the patch and the compact test.

UpgradeIndexMergePolicy walks its FilterMergePolicy chain and applies each OneMergeWrappingMergePolicy's operator to its own merge, innermost first, as wrapSpec does for returned merges; OneMergeWrappingMergePolicy gets a package-private wrapOneMerge(OneMerge). Patch: optionA-wrap-leftover-merge-like-delegate.diff, the finder's sketch, +24/-1 in two files.

private OneMerge wrapLikeDelegate(OneMerge merge) {
  ArrayDeque<OneMergeWrappingMergePolicy> wrappers = new ArrayDeque<>();
  for (MergePolicy p = in; p instanceof FilterMergePolicy f; p = f.in) {
    if (p instanceof OneMergeWrappingMergePolicy w) wrappers.push(w);
  }
  while (wrappers.isEmpty() == false) merge = wrappers.pop().wrapOneMerge(merge);
  return merge;
}
// findForcedMerges: spec.add(wrapLikeDelegate(new OneMerge(newInfos)));

Trade-offs

• No public API change, but it only reaches OneMergeWrappingMergePolicy subclasses through FilterMergePolicy.in; another decorator in the chain, such as BPReorderingMergePolicy, stays bypassed (by reading only). It reaches into another class's operator, which a maintainer may find too clever.

What the checks validated

• The three repro classes 11 of 11 pass (assertions on); the same 17 classes, 450 tests, 76 skipped, 0 failures. Earlier the finder's build passed its 2 tests and 19 upstream merge-policy tests, and the independent reproducer's and the adjudication's runs passed 6 of 6 and 2 of 2 with it.

Open an IndexWriter with new SoftDeletesRetentionMergePolicy(field, query, new UpgradeIndexMergePolicy(new TieredMergePolicy())), run forceMerge(1) and commit (IndexUpgrader also sets KeepOnlyLastCommitDeletionPolicy, which is already the default deletion policy; set it only if your config uses another one). The retention wrapper then also wraps the upgrade policy's own merge.

Trade-offs

• Users must know about it; IndexUpgrader itself cannot be configured this way.

What the checks validated

• testRetentionPolicyOutermost and the independent reproducer's and devil's advocate's controls keep all 13 documents (and 12 or 11 under partial retention).

Warn in IndexUpgrader and UpgradeIndexMergePolicy that decorating delegates do not see the leftover merge and that a retention policy must be outermost.

Trade-offs

• Cheapest, but IndexUpgrader still cannot upgrade such an index safely. Not a fix either: counting soft deletes for unpooled segments in numDeletesToMerge, since with a retain-all query TieredMergePolicy would still decline the segment.

Method: code reading (non-formal tier), in a hunt unit that compares Lucene's index tools by round-tripping indexes through them (a randomized differential test).

• While modelling the IndexUpgrader leg (which tools may change which documents), the finder read UpgradeIndexMergePolicy.findForcedMerges and noticed that the merge it creates for old segments its delegate declines is a plain OneMerge, which a wrapping delegate never sees.
• It wrote fixture indexes with the released 10.5.2 and 9.12.3 jars (13 documents, 3 soft-deleted, one or two segments) and a public-API JUnit around new IndexUpgrader(dir, config, false): one segment loses the 3 retained documents, two keep them. A shadow build with the fix sketch passed.
• The randomized differential itself did not find it: it builds no old-version segments and no retention policies.
• Found 2026-10-26 on main a9779ef3c193; then checked by an independent reproducer, a devil's-advocate verifier, reviews by a model from another family (reading only), an adjudication, a follow-up (history, prior reports, security) and an audit.