A tryUpdateDocValue un-delete is ignored or lost by later term-based operations

With a soft-deletes field, un-delete a document with tryUpdateDocValue(reader, doc, soft_delete=null), the way its javadoc suggests, and then touch it again by term before the writer writes the pending update (for example at the next refresh, flush or commit): the later operation does not see the un-delete. In the repro the soft delete gets sequence number 5, the un-delete 7 (success) and a second soft delete by term 8, yet a fresh reader and the commit still show 1 live copy of the document instead of 0. With softUpdateDocument as the later call the commit holds 2 live copies of the same key, and deleting the other document of its segment drops the revived document altogether. Lucene raises no exception, and CheckIndex passes on the committed index in the two tests that run it. The repro needs two documents and the opt-in soft-deletes field, and every release from 7.4.0 to 10.5.2 has it.

Lucene can soft-delete a document instead of removing it. You pick a doc-values field with IndexWriterConfig.setSoftDeletesField("soft_delete"); a document that has a value in that field counts as deleted, and readers from the writer hide it. You soft-delete by term with updateDocValues(id:x, soft_delete=1), or with softUpdateDocument, which marks the old copy and adds a new one.

To bring a soft-deleted document back, you remove its value with the "Expert" method tryUpdateDocValue(reader, docID, soft_delete=null). Its javadoc says: "This can be used to un-delete a soft-deleted document". It addresses the document by its doc ID in an NRT reader of the same writer and returns a sequence number (> 0 means success). The IndexWriter javadoc says sequence numbers express "the effective order in which each change was applied".

Two documents, id:x and id:y, no merges:

What you get: 1 live copy of x, in the fresh reader and in the commit. The step-5 delete never reached x. If you open an NRT reader between steps 4 and 5, you get 0. (Sequence numbers 3 and 6 are not hidden operations: the commit's and the reader's full flushes reserve them.)

Two more versions of step 5 show what this does to real data:

• Duplicate key. Step 5 is softUpdateDocument(id:x, new x, soft_delete=2), the usual way to replace a record when you use soft deletes. You expect 1 live x (the new one) and get 2, in the commit too. The same happens when every document is indexed with softUpdateDocument, as the soft-deletes javadoc prescribes.
• Lost document. Start from a segment that holds y and a soft-deleted x, un-delete x, then deleteDocuments(id:y) and commit. You expect x to be live and get no x at all. This also happens with a plain default writer config and a larger segment.

For each segment the writer keeps a bitmap of live documents and a deleted-document count, and next to them a queue of doc-values changes that are not written yet.

1. The un-delete only joins the queue. tryUpdateDocValue hands its change to the segment with rld.addDVUpdate(updates) (IndexWriter.java:1775), which just appends it (ReadersAndUpdates.java:153-164). The bitmap still says "x deleted". Its twin tryDeleteDocument is different: it clears the live bit at once (IndexWriter.java:1694).
2. The bitmap catches up only when the queue is written. Writing calls onDocValuesUpdate (ReadersAndUpdates.java:429), which re-applies the soft-delete field to the bitmap and the count (PendingSoftDeletes.java:151-167). That happens, for example, at the next NRT reopen, flush, commit or close, when a merge starts, or under RAM pressure.
3. Term-based updates consult the bitmap (the duplicate key). A buffered update by term only touches documents the bitmap calls live: acceptDocs = segState.rld.getLiveDocs() (FrozenBufferedUpdates.java:321, used at :327 and :341). x looks deleted, so the step-5 soft delete, or the soft-delete half of softUpdateDocument, skips it. Even the next reopen resolves the buffered step 5 (IndexWriter.java:596) before it writes the queued un-delete (:608).
4. The count still counts x (the lost document). A segment is "fully deleted" when getDelCount() == maxDoc (PendingDeletes.java:219-221), and the count includes the soft-deleted x (:243-246). After y is deleted the segment counts 2 of 2 deleted and is dropped (IndexWriter.java:6750-6752, :6726-6733), and the queued un-delete goes with it.

So the outcome depends on whether something wrote the queue between the two calls, which is why it can look intermittent.

A related, contested variant (C2). The same lag applies to two soft deletes by term with no un-delete: updateDocValues(id:x, soft_delete=1) and then soft_delete=2 commit x with value 2, but with 1 if an NRT reader was opened in between. Lucene's own tests expect term updates to skip documents whose soft delete is already written, so which answer is right when nothing was written is a question for the maintainers; the fix below does not change it. No javadoc promises refresh independence for this case.

• Who is hit. Applications with a soft-deletes field that change a document's soft-delete value with tryUpdateDocValue (to un-delete it, the javadoc's stated use, or to soft-delete it) and then, before the pending update is written (for example by the next NRT reopen, flush, commit or merge, or a RAM-triggered write), either touch the same document by term (updateDocValues, updateNumericDocValue, softUpdateDocument) or delete the other live documents of its segment (deleteDocuments, tryDeleteDocument). A plausible pattern: "restore this record, then replace it", that is, an un-delete followed by softUpdateDocument on the same key.
• What breaks. Two live documents with the same key (K1c); a later delete or update that silently misses (K1; K1b: a numeric or binary update by term keeps the old value); a revived document that disappears from the commit (K2). The mirror case also exists: after a soft delete with tryUpdateDocValue, a term-based numeric update reaches the soft-deleted copy only if no reader was opened in between.
• Opt-in feature, default settings otherwise. Soft deletes are opt-in and tryUpdateDocValue is an "Expert" public method. Default codec. K1 also fails under SoftDeletesRetentionMergePolicy; K2 needs a merge policy that drops fully deleted segments, which is the default; it was also run with a plain default writer config.
• No scale needed, deterministic. Two documents and a handful of calls; the same result on all 6 seeds tried and in 10 of 10 iterations. The writer config is fixed; the test framework still randomizes the directory and the codec, so this is tested robustness, not a proof.
• Silent and durable. Lucene raises no exception and no Lucene-internal assertion fires under -ea (only the test's own assertions report it); CheckIndex passes on the K1c and K2 commits; the wrong state is committed. Anything that writes the pending update between the two calls hides it, so it can look intermittent.
• Reach. A code search found no tryUpdateDocValue callers in Elasticsearch, OpenSearch or Solr, so the known exposure is custom applications using the expert API (the search says nothing about private code). How often applications hit the window was not measured.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: live x after un-delete (seq 7) then soft delete (seq 8) expected:<0> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.index.TestTryUndeleteThenSoftDelete.testSoftDeleteAfterTryUndeleteIsApplied(TestTryUndeleteThenSoftDelete.java:55)

• The method's contract. tryUpdateDocValue (IndexWriter.java:1707-1720): for an NRT reader of this writer whose segment was not merged away "the update succeeds and this method returns a valid (> 0) sequence number"; a null value removes the existing value, and "This can be used to un-delete a soft-deleted document since this method will apply the field update even if the document is marked as deleted." CHANGES.txt:7390-7393 (LUCENE-8298): "This allows to undelete a soft-deleted document unless it's been claimed by a merge." Nothing is merged in the repro.
• The ordering contract. IndexWriter.java:117-118: a sequence number "expresses the effective order in which each change was applied". The skipped delete has sequence number 8, after the un-delete's 7. The javadoc does not define which documents a later term operation selects, but a later call that behaves as if the earlier, successful one had not happened appears inconsistent with that order.
• What term operations promise. updateDocValues (IndexWriter.java:2060-2061): "Each field update is applied to the set of documents that are associated with the Term". softUpdateDocument (:1918-1919): "first updating the document(s) containing term". Neither excepts a just-revived document.
• Soft equals hard. IndexWriterConfig.java:543-545: a soft-deleted document "is treated as if it has been hard-deleted". The hard twin, tryDeleteDocument, changes the live docs at once, so later term operations see it.
• No upstream test issues a term operation while an un-delete is pending. TestSoftDeletesRetentionMergePolicy#testUndeleteDocument (:595-625) opens a new NRT reader right after the un-delete (:620); the only two helpers that call tryUpdateDocValue (TestSoftDeletesRetentionMergePolicy.java:781-792, TestMixedDocValuesUpdates.java:589-599) open a fresh reader before every call. The nearest case, testMixedSoftDeletesAndHardDeletes (:696-745), may issue a buffered delete (:735) right after a try soft delete (:721); that is not one of this bug's forms (no un-delete, and term hard deletes are not filtered by the soft live docs).
• History. LUCENE-8290 (#9337, committed 2018-05-03) moved the soft-delete bookkeeping to write time "to be consistent with what is actually updated on disk", before un-delete existed. LUCENE-8298 (#9345, by @s1monw, +1 from @mikemccand and @jpountz; 2018-05-09) added un-delete without revisiting that; its thread does not discuss later operations issued while the reset is pending. #8398 (LUCENE-7344, open) is the same design family for delete-by-query.

Strongest case for "intended": IndexWriterConfig.java:557-559 says "Currently there is no API support to un-delete a soft-deleted document" and tells you to re-index instead; the same javadoc says documents "must be indexed via IndexWriter#softUpdateDocument" (IndexWriterConfig.java:547-548), and the repro uses addDocument. The tryUpdateDocValue NOTE says it "can only update documents visible to the currently open NRT reader" (IndexWriter.java:1717-1719). Term operations resolve against the written live docs, and sequence numbers order operations without defining their target set. LUCENE-8290 deliberately keeps the live docs in sync with what is on disk, so term operations see the written state, and an application using the Expert try* API could be expected to refresh after it. The independent reviewer from another model family (OpenAI GPT-5.6) built this argument itself and rated it moderate. The cross-model fact check of this page rated it moderate for K1/K1b/K1c, weak for K2, strong for C2 and moderate for the mirror case.

Why it fails: that IndexWriterConfig paragraph predates the feature (it came with LUCENE-8233 and LUCENE-8298 never updated it); the later method javadoc, the CHANGES entry and the upstream un-delete test all support un-delete through tryUpdateDocValue. No javadoc states a refresh precondition; the call reports success with a sequence number that orders it first, and the result then follows hidden buffer state (the same calls are right after any refresh). The hard-delete twin is visible at once, and soft deletes are documented to behave like hard ones. Indexing with softUpdateDocument changes nothing: K1, K1c and K2 were reproduced that way. "Visible" in the NOTE contrasts the reader's segments with documents indexed after it (its next sentence); the same javadoc says the update applies "even if the document is marked as deleted", and the upstream helper finds the soft-deleted document through an IncludeSoftDeletesWrapper. The lost-document form is more than a visibility question: a successful, pending un-delete is discarded with its segment because the deleted count still counts x. The reviewer reached "real, high confidence" for the same reasons. Verdict: not intended for the un-delete forms (K1, K1c, K2).

Where "intended" has a point: C2. TestSoftDeletesDirectoryReaderWrapper#testReuseUnchangedLeafReader (:94-165) expects a term-based soft update to leave an already soft-deleted, written document alone. C2 asks what two term-based soft deletes should give when nothing was written in between; no API promises refresh independence for this case. The study's verifiers split 4 "new bug" / 2 "intended" on a related form that un-deletes by term (updateDocValues(term, soft=null)); the soft=1-then-soft=2 form in the test was added later as adjudication evidence and was not voted on by itself.

For the un-delete forms the gap is that a successful try* change to the soft-deletes field reaches neither the live docs nor the deleted count until it is written. A fix either makes it visible at once (option B, validated) or writes it at once (option A, the study's diagnostic). C2 needs a separate decision. Option B was built and validated in a scratch worktree at ea57360912ed (removed afterwards); the others are diagnostics or untested ideas.

tryUpdateDocValue hands its resolved update to a new ReadersAndUpdates.addTryDVUpdate, which still queues it for writing and, if it touches the soft-deletes field, also applies it to the pending live docs and the pending soft-delete count, the way tryDeleteDocument applies its delete. When the pending updates are written later, onDocValuesUpdate applies the newest value of each document again: if the try update is still the newest, nothing flips (applySoftDeletes counts only bits it changes); if a later update of the same document exists, as in K1, the newer value wins, which is the intended result. +33/-1 lines in IndexWriter.java, ReadersAndUpdates.java, PendingDeletes.java and PendingSoftDeletes.java.

// IndexWriter.tryUpdateDocValue
-              rld.addDVUpdate(updates);
+              rld.addTryDVUpdate(updates);

// ReadersAndUpdates
public synchronized void addTryDVUpdate(DocValuesFieldUpdates update) throws IOException {
  if (reader == null) {
    getReader(IOContext.DEFAULT).decRef(); // pass a reader to initialize the pending deletes
  }
  addDVUpdate(update);
  pendingDeletes.onTryDocValuesUpdate(update);   // no-op in PendingDeletes
}

// PendingSoftDeletes
@Override
void onTryDocValuesUpdate(DocValuesFieldUpdates update) throws IOException {
  if (this.field.equals(update.field)) {
    pendingDeleteCount += applySoftDeletes(update.iterator(), getMutableBits());
    assert assertPendingDeletes();
  }
}

Trade-offs

• It brings back, for the try* path only, the eager bookkeeping that LUCENE-8290 removed for all updates because a retention merge policy could "see a deleted document but not it's value on disk". Updates pending when a merge starts are written before the merge reads the segment (IndexWriter.java:5310), and pending updates are written before an NRT reader is opened (:608). Merge selection can still see the difference: SoftDeletesRetentionMergePolicy.numDeletesToMerge (:154-183) runs its retention query on the on-disk values while the count already includes a pending try soft delete, so it can over-count reclaimable documents until the update is written (read, not tested).
• Not covered (read, not tested): a try update that arrives while a merge of its segment is running is carried over to the merged segment with plain addDVUpdate (IndexWriter.java:4531-4540), not addTryDVUpdate, so on the merged segment the window can open again until the update is written. A complete fix would also have to handle that, or document it. There is no concurrent test for try updates during a merge, merge carry-over, abort or rollback.
• NRT readers that are already open keep their point-in-time live docs; option B changes what later writer operations resolve against.
• No new doc-values generation per call (unlike option A); the added work is the in-memory bit update and, when the segment has no pooled reader yet, a defensive getReader. A small cost probe (1,000 rounds of a try soft-field change plus a term-based numeric update, in-memory directory, 3 runs of 3 rounds) found no clear slowdown: steady-state rounds took 57-155 ms with option B and 37-219 ms on stock, so the ranges overlap. Nine samples are not a benchmark.
• tryUpdateDocValue on a document that is also hard-deleted sets its live bit back: that is the separate bug LFV-008 (applySoftDeletes ignores hard deletes), now hit inside the try call with assertions on (stock: at the next NRT open); with assertions off both fail at the next NRT open (executed by the fact check). If LFV-008's fix lands, the new call should pass the hard live docs too (not tested together).
• Does not fix C2 or the sister bug LFV-009 (an earlier buffered update overwriting a try update).

What the checks validated

• The repro: K1, K1c, K2 and the control pass with assertions on and off; only C2 still fails (it has no try* call).
• The study's original 15-test repro (10 bug tests including K1b, K2b, K2c and K2d, plus 5 controls): 15/15 pass with -ea and -da (stock: 10 of 15 fail). The reviewer's own repro: 2/2 pass. The edge test: 5/5 pass.
• :lucene:core:test on 22 targeted suites (the repro, the edge test, TestPendingSoftDeletes, TestPendingDeletes, TestSoftDeletesRetentionMergePolicy, TestSoftDeletesDirectoryReaderWrapper, TestMixedDocValuesUpdates, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestSortedNumericDocValuesUpdates, TestIncrementalDocValuesUpdates, TestDocValuesFieldUpdates, TestMergeCarryOverFromDisk, TestIndexWriterMergePolicy, TestTryDelete, TestIndexWriterDelete, TestIndexWriterReader, TestIndexingSequenceNumbers, TestIndexWriter, TestStressNRT, TestCheckIndex, TestAddIndexes): 408 tests, 1 failure (C2), with assertions on and with -Ptests.asserts=false.
• --tests 'org.apache.lucene.index.*': 215 test classes, 1,404 tests as Gradle counts them, 1 failure (C2), 56 skipped.
• :lucene:core:check -x test passes (google-java-format, ecjLint, forbiddenApis) after tidy re-wrapped one javadoc sentence of the patch and a few test lines.
• After two comment corrections on 2026-10-26 (no code change): the changed files are google-java-format clean, and the repro (-ea, -da), the edge test and the study's 15-test repro give the same results.

After a try update that touches the soft-deletes field, call rld.writeFieldUpdates(...) and checkpoint() inside tryUpdateDocValue (+18 lines in IndexWriter.java). Partly checked: on ea57360912ed K1, K1c and K2 pass and C2 fails; in the study (main 7cecf79523) its 15-test repro and 19 upstream soft-deletes tests passed. Not run against the module test suites here.

Trade-offs

• A new doc-values generation (files) per call. In the same cost probe steady-state rounds took 404-1,184 ms against 37-219 ms on stock.
• It writes only updates whose delete generation is at most completedDelGen (ReadersAndUpdates.java:870-882), so while another thread is still applying an older packet, the new try update is not written (other eligible pending updates may be) and the window stays open (read, not tested).

Term-based updates would then reach soft-deleted documents too. K1, K1c and C2 pass, K2 still fails.

Trade-offs

Not a fix: it breaks TestSoftDeletesDirectoryReaderWrapper#testReuseUnchangedLeafReader (its assertSame on the unchanged leaf fails; the stock run passes), so it changes intended semantics.

If the maintainers agree that a later term-based soft update should skip a document that an earlier, unwritten soft delete already covers (what testReuseUnchangedLeafReader expects once it is written), term-resolved soft-field updates must reach the live docs before later resolution.

Trade-offs

Needs the eager hook for those updates too, which undoes LUCENE-8290's move, plus tracking inside one packet, because a packet's updates are handed to the segment only after all of them are resolved (FrozenBufferedUpdates.java:351-357). A design question for the issue.

State on tryUpdateDocValue and setSoftDeletesField that a try* soft-field change is not seen by term-based operations, and can be lost with its segment, until the pending update is written.

Trade-offs

The cheapest change, but it contradicts the sequence-number contract and makes K2 documented data loss. Independent of any fix, the stale IndexWriterConfig.java:557-559 paragraph should be updated.

Method: model-based randomized testing (the study's non-formal baseline); formal tools re-found it after being told about it.

• A 2026-10-22 study of Lucene by AI tools covered the IndexWriter/NRT layer because an earlier bug there (now #16900) was the study's starting point. So the area was hinted; this bug was not.
• An AI tool wrote a model-based testing harness: an executable reference model of the writer, random operation sequences, and a comparison of every reader against the model. Its model-value comparison flagged this bug 704 s into a run on its own, and after 5 s when combined with the operations of the sister bug LFV-009.
• TLA+ (the TLC model checker), Alloy, P, Dafny, JBMC and OpenJML checks then re-found it with hints; the TLC model also pinned the root cause of the lost-document form.
• Independent AI verifiers reproduced it and voted "new bug"; this hunt re-verified it on ea57360912ed, ran the release history and the prior-art search, had it reviewed adversarially, and validated a fix.