An un-delete with tryUpdateDocValue during a merge returns success, then the merge silently drops the document

With a soft-deletes field, a soft-deleted document stays in its segment, where it can be restored, until a merge reclaims it, and IndexWriter#tryUpdateDocValue(reader, docID, <soft-deletes field = null>) is the documented way to un-delete it. The javadoc promises that while the segment "has not been merged away" the update succeeds and returns a sequence number > 0, and otherwise the call returns -1. If a merge of that segment is already running, the call succeeds and an NRT reader shows the document again, but the merge took its readers while the document was soft-deleted and left it out, and the carry-over of changes made during the merge skips documents the merge dropped. When the merge commits the document is gone, with no exception, and the next commit persists the loss, although its sequence number is above the un-delete's. The same happens with a SoftDeletesRetentionMergePolicy whose query does not match the document; a retention query that keeps it makes the un-delete survive. Every release with the feature is affected, 7.4.0 (LUCENE-8298, 2018) through 10.5.2, and main. Returning -1 for an un-delete while the segment is merging makes the repro pass, and the core index package still passes with it.

• Soft delete: with IndexWriterConfig#setSoftDeletesField("soft"), a document whose soft doc-values field has a value counts as deleted. It stays in its segment, so it can still be restored, until a merge leaves it out (or until the whole segment is dropped once all of its documents are deleted, IndexWriter.java:4128-4139).
• Un-delete: tryUpdateDocValue(reader, docID, new NumericDocValuesField("soft", (Long) null)) removes the value. The javadoc says: "This can be used to un-delete a soft-deleted document" (IndexWriter.java:1714-1715).
• Retention: a SoftDeletesRetentionMergePolicy keeps soft-deleted documents that match its query in the merged segment; without it a merge drops every soft-deleted document.
• Carry-over: changes made to a segment while it is being merged are re-applied to the merged segment when the merge commits, through the merge's doc map from old to new docIDs. A document the merge left out maps to -1.

1. The call accepts a merging segment. tryUpdateDocValue (IndexWriter.java:1721) hands its change to tryModifyDocument (:1791-1829), which applies it whenever the segment is still listed in segmentInfos (:1819) and has pooled reader state (:1820-1821, true for a segment of an NRT reader from this writer), and returns the next sequence number (:1824); otherwise it returns -1 (:1828). A segment being merged stays listed until the merge commits, and nothing here checks mergingSegments (unlike dropDeletedSegment, :1837).
2. Readers see the un-delete. The reset lands in the segment's pending updates, so the next NRT reader counts the document as live: 3 documents.
3. The merge had already dropped it. The merge took its readers while the document was soft-deleted; mergeMiddle passes each one through merge.wrapForMerge(reader) (:5497), where only a retention policy could mark it live again. With no retention, or a query that does not match, the document is not written to the merged segment and maps to -1.
4. The carry-over skips it. commitMergedDeletesAndUpdates (:4485-4556) maps every change through the doc map and skips -1: residual doc-values updates (:4644-4647) and the on-disk numeric, binary and sorted-numeric diffs (:4734-4737, :4766-4769, :4805-4808); hard deletes use the same test (:4847). The reset has nowhere to go.
5. The merge commits and the loss is permanent. The source segment that held the un-delete is replaced by a merged segment that never contained the document. No exception; the merge lines of IndexWriter's info stream that the diagram probe kept end with "0 new deletes since merge started" (the probe filters the stream; no warning is logged on this path, by reading). CheckIndex is clean, and even a raw commit-point reader that ignores soft deletes finds no copy.

With a retention query that keeps the document, the merge writes it (still soft-deleted) into the merged segment, its doc-map entry is not -1, and step 4 carries the reset over: the un-delete survives. An un-delete made before the merge takes its readers also survives, and a tryDeleteDocument hard delete of a live document, made at the same point, is carried over.

• Who is hit. Soft deletes are opt-in (IndexWriterConfig#setSoftDeletesField, default null); the un-delete is an Expert try* call. No caller of tryUpdateDocValue was found in Elasticsearch, OpenSearch or Solr (code search on default branches, not exhaustive; nothing was run downstream). An unmerged 2018 Elasticsearch prototype used this un-delete, treated -1 as "retry" and kept rollback candidates with a retention query; it may have avoided this bug if that query always kept every document it restored, which was not established.
• The window is long. Under the default ConcurrentMergeScheduler merges run in the background; the loss is shown for un-deletes that land after the merge took readers without the document and before it commits, which includes the time spent writing the merged segment. The earlier time, after the merge is registered and before it takes its readers, was not characterized. It is not a tight race: the repros hold the merge at three different points and fail every time (20 of 20 and 60 of 60 iterated runs).
• Silent and permanent. The call reports success, NRT readers confirm it, and nothing signals the loss later. No copy is left in the current merged segment or the current commit point; only a reader opened before the merge committed, or an older commit kept by a deletion policy, may still hold it. In practice the application must notice and re-index it from its own source of truth.
• Sequence numbers mislead. commit()'s sequence number describes "which changes are in the commit point and which are not" (IndexWriter.java:117-120), and commit()'s javadoc says "All sequence numbers <= this value will be reflected in the commit" (:4208-4210); here it is above the un-delete's, yet the commit lacks it.
• How common was not measured. The un-delete is niche (its own IndexWriterConfig javadoc still says it does not exist), and no one has reported the loss since 7.4.0; the registry rates the likelihood rare.
• Workarounds until fixed: a SoftDeletesRetentionMergePolicy whose query keeps every document the application may restore; or wait until the merges that were running when the document was restored have finished, then check that it is live and re-index it if not. A commit() alone is not a sync point: merges can keep running after it (LiveIndexWriterConfig.java:467-470).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected seqNo == -1 or numDocs == 2 after the merge, actual seqNo=7 numDocs=1
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.index.TestTryUndeleteDuringMerge.testUndeleteDuringMergeIsDropped(TestTryUndeleteDuringMerge.java:55)

• The method's contract. tryUpdateDocValue (IndexWriter.java:1708-1715): "If the provided reader is an NRT reader obtained from this writer, and its segment has not been merged away, then the update succeeds and this method returns a valid (> 0) sequence number; else, it returns -1 and the caller must then either retry the update and resolve the document again. ... This can be used to un-delete a soft-deleted document".
• A running merge is not "merged away". TestSoftDeletesRetentionMergePolicy#testDeleteDocWhileMergeThatIsSoftDeleted (:544-593) calls tryDeleteDocument from the merged-segment warmer, on the soft-deleted document that the merge drops, and asserts "seqId was -1", seqNo != -1 (:582). Its comment: this "requires special logic inside IndexWriter#carryOverHardDeletes since docMaps are not created for this document". For a delete, the merge dropping the document is the requested result; for an un-delete it is the opposite, and no logic or test covers that case. Upstream's tryUpdateDocValue retry helper doUpdate (used for the un-delete in testUndeleteDocument and also for soft deletes) retries only on -1, "retry if we just committing a merge" (:781-792).
• Sequence numbers. The class javadoc ties commit()'s sequence number to the changes the commit contains (IndexWriter.java:117-120), and commit()'s javadoc says "All sequence numbers <= this value will be reflected in the commit" (:4208-4210).
• Precedent. LUCENE-4521 (#5587): a try* call that reported success but whose change was not persisted was fixed as a bug. The try* contract itself comes from LUCENE-4203 (#5275): success while the segment is not merged away, failure otherwise.
• Nothing documents the revert. No javadoc, CHANGES or MIGRATE text says an acknowledged un-delete may be undone by a merge that is already running.

The strongest case for "intended", made by the intended-behaviour checker, the refute-lens review and the adjudication: CHANGES 7.4 for LUCENE-8298 says the null value "allows to undelete a soft-deleted document unless it's been claimed by a merge" (lucene/CHANGES.txt:7390-7393). Lucene uses "claim" for a merge's decision to reclaim deletes (SoftDeletesRetentionMergePolicy.java:42), and a merge takes that decision when it wraps its readers, so a running merge that does not retain the document has already claimed it. The same commit edited the carry-over for resets and kept the skip for dropped documents. IndexWriterConfig#setSoftDeletesField still says "Currently there is no API support to un-delete a soft-deleted document" (IndexWriterConfig.java:557-559). The try* methods are Expert, and the behaviour is eight years old.

Why it fails: at its strongest the CHANGES line says the un-delete may be impossible once a merge has claimed the document. It does not say the call may return success, show the document to readers and then be reverted; the API has a signal for "impossible", -1, and the same change's commit message states the limit in the javadoc's own words, "unless it's merged away". The IndexWriterConfig sentence came with LUCENE-8233, a month before the un-delete existed, and contradicts the tryUpdateDocValue javadoc: it is stale. A maintainer may still say that retention is the intended way to keep a soft-deleted document restorable, and that carrying a reset onto a document the merge left out is impossible without retaining it, aborting the merge or re-running merge work; that supports "return -1" or "document it", not "make it survive". Verdict: not intended.

One fix was applied and tested in a scratch worktree at ea57360912ed; the others are sketches, not run.

tryUpdateDocValue marks the call as an un-delete when one of its fields is the soft-deletes field with a null value, and tryModifyDocument returns -1 for it while mergingSegments contains the segment, so the caller gets the documented failure signal and retries. Calls without a soft-field reset, and tryDeleteDocument, are unchanged (for documents the merge keeps, the carry-over applies them). A call that combines the reset with other fields is refused whole. A real patch should also say this in the tryUpdateDocValue javadoc and fix the stale IndexWriterConfig#setSoftDeletesField sentence.

// tryUpdateDocValue: is this an un-delete (null value on the soft-deletes field)?
boolean undelete = false;
for (DocValuesUpdate update : dvUpdates) {
  undelete |= update.hasValue() == false && update.field.equals(config.getSoftDeletesField());
}
return tryModifyDocument(readerIn, docID, undelete, (leafDocId, rld) -> { /* unchanged */ });

// tryModifyDocument(readerIn, docID, refuseWhileMerging, toApply): refuse while merging
if (refuseWhileMerging && mergingSegments.contains(info)) {
  return -1;
}
if (segmentInfos.indexOf(info) != -1) { /* unchanged */ }

Trade-offs

• It also refuses un-deletes that would have survived. With a retain-all retention query, stock returns 11 and the un-delete survives; with the fix the call returns -1, and a retry after the merge (fresh reader, document resolved again) succeeds with sequence number 12 and the commit has the document. Without retention, the retry finds the document gone (maxDoc 2), so the caller learns it must re-index instead of believing the restore worked.
• Retries wait for the merge. A caller that retries on -1 in a loop, like upstream's doUpdate test helper, spins until the merge commits.
• Scope. It refuses for as long as the segment is registered for a merge, including the time before the merge takes its readers; whether an un-delete in that early part would have survived was not tested. A call that combines the soft-field reset with other fields is refused whole, and because the check looks only at the field and the null value, it also refuses that reset on a live document whose soft field is already empty.
• Cost. One set lookup per un-delete call. Not benchmarked.

What the checks validated

• This page's repro: 2 of 3 fail on stock, 3 of 3 pass with the fix, through gradle with -Ptests.asserts=true and false, and with the JUnit runner (-ea, -da).
• The 18 closest core suites (soft deletes, try*, doc-values updates, merges, TestIndexWriter, TestIndexWriterReader, TestAddIndexes, TestCheckIndex and others): 380 tests, 13 skipped, 0 failed, with assertions on (seed 1A2B3C4D) and off (seed 77AA77AA).
• The whole core org.apache.lucene.index.* package: 215 suites, 1,397 tests, 56 skipped, 0 failed (seed 1A2B3C4D, assertions on). :lucene:core:checkGoogleJavaFormat passes.
• Earlier: the finder's contract test fails 2 of 4 on stock and passes 4 of 4 with the fix; the security probe's loss test fails at the expected step with the fix (a vacuity check).

Return -1 only if a merge holding the segment has already taken its readers and the document is deleted in the merge's retention-applied reader.

Trade-offs

Keeps retain-all un-deletes working during merges, but the merge would have to keep its wrapped readers' live docs (today mergeMiddle keeps the wrapped reader local, IndexWriter.java:5494-5536), and the time before the readers are taken needs care. More code and state.

When an un-delete targets a document a running merge dropped, abort or re-run that merge so the document is merged as live.

Trade-offs

Keeps today's success semantics, but wastes the merge's work and interacts with forceMerge and merge-abort handling; not explored.

Say in the tryUpdateDocValue javadoc that an un-delete during a merge can be reverted when the merge commits, and point to a retention query as the way to avoid it.

Trade-offs

Cheapest, but callers that trust > 0 still lose data silently.

Method: model-based testing against an executable reference model (the hunt's "semi-formal" tier), a hunt unit (soft deletes), 2026-10-26, main ea57360912ed.

• The model tracks every document as live, soft-deleted, hard-deleted or retained, per segment, and compares the real writer after each step through NRT readers, commit points, SoftDeletesDirectoryReaderWrapper and CheckIndex. Its file:line table maps each model element to the Lucene code.
• The operation MERGE_CONC holds a forced merge inside OneMerge#wrapForMerge after the merge took its readers and runs sub-operations meanwhile. A TRY_REVIVE sub-operation (the tryUpdateDocValue un-delete) produced a document that the model keeps live after the merge but the implementation drops (model check merge-drops-revived-doc).
• The finder minimised it by hand to a latch-driven JUnit on the stock jars. No model checker, SMT solver or prover was involved. To stay off known bugs the model never un-deletes hard-deleted documents (a separate registered bug) and runs try* calls only right after a refresh.