Removing a document's soft-delete value brings a hard-deleted document back

Turn on soft deletes, hard-delete a document, then remove that document's soft-delete value (a null doc-values update, the documented way to un-delete a soft-deleted document): the hard-deleted document is live again. In the repro, with b soft-deleted, hard-deleting a and removing a's soft value leaves the live documents [a, c] instead of [c], with no error. A merge in the same writer session copies it into the merged segment, which no longer records the hard delete, and CheckIndex does not flag the commit. When no other document in the segment is soft-deleted, the soft-delete count goes to -1 instead: with assertions on the writer closes on an AssertionError, and with assertions off that refresh throws, and so did every later refresh, commit and close in the tested sequence. Soft deletes are opt-in, three documents are enough, and every release since 7.4.0 has it.

Turn on soft deletes with IndexWriterConfig#setSoftDeletesField("soft_delete"). A document can then be deleted in two ways:

• Hard delete: deleteDocuments(id:a). The document is gone; merges drop it.
• Soft delete: give the document a value in the soft_delete doc-values field, for example updateDocValues(id:b, soft_delete=1). Readers skip it, but a retention merge policy can keep it around.

And there is one way back: remove the soft value again with a null doc-values update by doc ID, tryUpdateDocValue(reader, doc, soft_delete=null), which applies "even if the document is marked as deleted"; its javadoc says this "can be used to un-delete a soft-deleted document". A null update by term, updateDocValues(term, soft_delete=null), removes values too, but it is resolved against the current live docs, so it skips documents whose delete has already been applied.

The rules: the setSoftDeletesField javadoc says "Hard-deleted documents won't be considered as soft-deleted even if they have a value in the soft-deletes field", and the null update is documented as the way to bring back a soft-deleted document. Nothing allows it to bring back a hard-deleted one.

Three documents a, b, c (doc 0, 1, 2) in one committed segment, NoMergePolicy (c stays live, so the segment never becomes fully deleted and dropped):

What you get: live [a, c]. a is back, and a search for id:a finds it. The writer's soft-delete count says 0, although b is still soft-deleted. Nothing fails.

The documented un-delete does the same: soft-delete b and d, un-delete b, then hard-delete b before the next refresh. You expect live [a, c] and get [a, b, c]: a "restore, then purge" brings the purged document back.

Each segment keeps two sets of live bits, the hard live docs and the combined live docs (hard and soft deletes together, which readers and merges use), plus a soft-delete count of documents that are soft-deleted but not hard-deleted.

1. At the next refresh the writer first applies the buffered hard delete of a (IndexWriter.java:586-596). a is cleared in both sets; the count stays 1 (that is b).
2. Then it writes the pending doc-values updates (IndexWriter.java:608) and applies a's null update to the live docs (ReadersAndUpdates.java:429 -> PendingSoftDeletes.onDocValuesUpdate, PendingSoftDeletes.java:151-158).
3. The branch for a removed value reasons "the bit is clear, so this was a soft delete: set it, and count one soft delete less": if (bits.getAndSet(docID) == false) newDeletes--; (PendingSoftDeletes.java:143). It never looks at the hard live docs.
4. So a's combined bit is set again, which makes a live for NRT readers and for merges, and the count drops from 1 to 0.

A merge in the same writer session copies a into the new segment, which no longer records the hard delete, and CheckIndex does not flag it. If no other document in the segment is soft-deleted, the count goes to -1 instead, and the writer starts throwing. Timing matters: if a refresh happens between the null update and the hard delete, the delete sticks, and a term-based null update that is in the same buffered packet as the hard delete correctly skips the deleted document.

• Who is hit. Applications with a soft-deletes field that issue a null update of that field for a document that is hard-deleted, or gets hard-deleted, before the update is written (the next NRT refresh, commit or merge): by doc ID with tryUpdateDocValue, or by term while the document is still live. Natural ways to get there: restore, then purge (un-delete a soft-deleted document, then hard-delete it before the next refresh); clearing the soft value by term as a "make sure it is live" step and hard-deleting the document after a flush; or one code path un-deleting by doc ID while another hard-deletes by term.
• Opt-in, otherwise ordinary. Soft deletes are off by default. With them on, nothing else is special: three or four documents and public API only; the same result with every seed tried (4, on the earlier 7-test copy) and with the default codec forced.
• Silent and permanent with a retention policy and default merging. With SoftDeletesRetentionMergePolicy over the default TieredMergePolicy and the default ConcurrentMergeScheduler: after forceMerge(1) the commit has live [a, c, x] instead of [c, x]; with no forceMerge and 30 more one-document commits, the commit has 7 segments and a is live. CheckIndex is clean both times.
• Loud when there is no slack. If the segment has no other soft-deleted document, the count goes to -1. With assertions on that is an AssertionError ("illegal pending delete count: -1"), a tragic event that closes the writer. With assertions off, that reader open throws IllegalArgumentException: invalid softDelCount=-1; the exception is not tragic, but in the tested sequence the next open, a commit and a close threw it too, and rollback() then discarded the work since the last commit (other repairs were not tried). With the plain default merge policy the tiny repro index lands here: when the NRT reader was opened, the small segments had been merged and b reclaimed (one segment, 0 deletions).
• A commit without a merge keeps the hard delete on disk but records a soft-delete count that is too low; CheckIndex reports actual soft deletes: 1 but expected: 0 (and fails on it in every release run from 8.11.0 on).
• A corrupt merged segment in a longer, heavily configured sequence found by the test harness (index sort, SoftDeletesRetentionMergePolicy over a two-segments-per-tier TieredMergePolicy, SerialMergeScheduler, addIndexes, a term-based null update of two live documents and a delete-by-query that hard-deletes both): the merge run by commit() fails an assertion (numMerged=4 vs mergeState.segmentInfo.maxDoc()=6) with -ea; with -da the commit returns and reading stored fields from a new NRT reader throws IndexOutOfBoundsException (the committed index was not reopened or checked). The same fix removes both. It is secondary and should not lead an issue.
• Reach. A code search found no tryUpdateDocValue callers and only test-file callers of updateDocValues in Elasticsearch and OpenSearch, so the known exposure is custom applications (the search says nothing about private code). Every release from 7.4.0 to 10.5.2 is affected.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<0> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.index.TestSoftValueRemovalRevivesHardDeletedDoc.testHardDeletedDocStaysDeleted(TestSoftValueRemovalRevivesHardDeletedDoc.java:48)

• The contract. IndexWriterConfig#setSoftDeletesField (IndexWriterConfig.java:561-563): "If soft-deletes are enabled, documents can still be hard-deleted. Hard-deleted documents won't be considered as soft-deleted even if they have a value in the soft-deletes field." The tryUpdateDocValue javadoc (IndexWriter.java:1712-1715) and CHANGES.txt:7390-7393 (LUCENE-8298, 7.4.0) describe a null value as the way to un-delete a soft-deleted document.
• The count. LUCENE-8341 (#9388) defines the recorded count as "the number of documents that are soft deletes but not hard deleted". SegmentCommitInfo.setSoftDelCount rejects negative values (SegmentCommitInfo.java:384-388) and CheckIndex rejects a mismatch (CheckIndex.java:4945-4947). The hard-delete path keeps this invariant (PendingSoftDeletes.delete, :50-66, takes a soft-deleted document out of the count at :60); the value-removed branch does not.
• Upstream tests. TestSoftDeletesRetentionMergePolicy.testMergeSoftDeleteAndHardDelete says "hard-delete should supersede the soft-delete" (:663); testUndeleteDocument (:595) un-deletes only a soft-deleted document. None of the tests added with LUCENE-8298 calls a delete API, so the combination was never tested.
• Foreseen in review. In the LUCENE-8298 review (#9345, 2018-05-09) a committer (@jpountz) wrote that "the ability to un-hard-delete is a bit scary to me because of the documents that got marked as deleted because they failed indexing". The patch was revised and approved; the merged branch still has no hard-delete check.
• Precedent. LUCENE-8381 (#9428; JIRA fix version 7.4.1, which was never published, so first released in 7.5.0): hard-deleted documents were counted as soft-deleted in merges, and this was fixed as a bug. The reporter first wrote that SoftDeletesRetentionMergePolicy "brings to live documents that soft-deleted then hard-deleted"; a maintainer replied that "it won't bring a deleted document back to live it will just loose the hard delete bit". Same invariant (hard deletes win), different code, and not a visible revival.
• Timing. If an NRT reader is opened between the null update and the hard delete, the hard delete sticks (study control 3). A documented behaviour would not depend on hidden buffer state. And in two of the failing tests the hard delete is the later operation, yet it loses.

Strongest case for "intended": tryUpdateDocValue is an Expert API, and its javadoc says it "will apply the field update even if the document is marked as deleted"; read broadly, "deleted" includes hard deletes and "apply" includes making the document live. In the core case the null update is also the later operation (higher sequence number), so "the later change wins" can be argued there. IndexWriterConfig.java:557-559 even says "Currently there is no API support to un-delete a soft-deleted document", so null soft updates could be called unsupported. And IndexWriterConfig.java:547-549 says that with soft deletes "documents must be indexed via" softUpdateDocument, while the repro uses addDocument.

Why it fails: the same javadoc sentence names its purpose, un-deleting a soft-deleted document, and setSoftDeletesField says soft values do not affect hard-deleted documents; "apply the field update" means writing the value, not undoing a hard delete. The sequence-number reading covers one order only: in testUndeleteThenHardDelete and testTermApiOnly the hard delete is the later operation and still loses, although sequence numbers express "the effective order in which each change was applied" (IndexWriter.java:117-118). The term API itself skips deleted documents (it resolves terms against the live docs), so the result depends on buffer timing. softUpdateDocument with a term that matches nothing just adds the document, upstream's soft-delete tests index with addDocument too, and IndexWriterConfig.java:562 allows hard deletes. The "no API support" sentence is stale against LUCENE-8298, and the control test, the documented un-delete, passes. Even if reviving were allowed, a soft-delete count of -1, exceptions on the following opens, commits and closes, and a commit that CheckIndex rejects are broken invariants, not API behaviour. testTermApiOnly removes the soft value of a document that never had one (absent before and after) and still wedges the writer. An independent reviewer from another model family (OpenAI GPT-5.6) built a similar argument and rated it weak; a second cross-model check rated it weak overall and moderately plausible only for the core case. Verdict: not intended, a defect.

The value-removed branch treats every clear live bit as a soft delete. The fix makes it skip documents that are hard-deleted at the moment the update is applied. A check at call time is not enough, because the hard delete may still be buffered, or not yet issued, then. One option was built and validated on a scratch worktree at ea57360912ed (removed afterwards); the others are untested.

onDocValuesUpdate passes the hard live docs to applySoftDeletes, and the branch sets the bit and lowers the count only for a document that is not hard-deleted. +12/-2 lines, PendingSoftDeletes.java only. It is the study verifier's patch, unchanged.

static int applySoftDeletes(DocIdSetIterator iterator, FixedBitSet bits, Bits hardLiveDocs) ... {
  ...
  } else {
    if ((hardLiveDocs == null || hardLiveDocs.get(docID)) && bits.getAndSet(docID) == false) {
      newDeletes--;
    }
  }

// onDocValuesUpdate
FixedBitSet mutableBits = getMutableBits();
pendingDeleteCount += applySoftDeletes(iterator, mutableBits, hardDeletes.getLiveDocs());

Trade-offs

• A null soft update of a hard-deleted document still removes the doc value, but no longer changes liveness or the count. The documented un-delete of a soft-deleted document is unchanged (the control passes).
• Not measured: hardDeletes.getLiveDocs() hands out a read-only snapshot and drops the writable copy (PendingDeletes.java:107-112). If a writable hard live-docs copy exists at that moment, the next hard delete in that segment copies the bitset once (maxDoc bits, PendingDeletes.java:63-81); otherwise nothing extra is copied. An accessor without that side effect would avoid it.
• The two-argument applySoftDeletes stays for its other two callers, which pass plain doc-ID iterators and only ever clear bits.
• It does not change the related bugs LFV-009 and LFV-010: their repros fail exactly as before with it.

What the checks validated

• The repro: 8/8 pass with assertions on and off; with the fix stashed, 7 of 8 fail.
• 21 nearby suites in :lucene:core (soft deletes, pending deletes, doc-values updates, TestIndexWriter, TestIndexWriterReader, TestIndexWriterDelete, TestStressNRT, TestCheckIndex, TestAddIndexes and more): 405 tests, 0 failures; with -Ptests.asserts=false: 406 tests, 0 failures.
• All of :lucene:core:test: 747 test classes, 8,873 tests, 322 skipped, 0 failures.
• :lucene:core:check -x test passes (google-java-format, ecjLint, forbiddenApis); tidy did not change the patch.
• With the patched class shadowed in the standalone harness: the merge-policy probe (4/4), the harness-found sorted-merge case, and the cross-model reviewer's own test pass.

Give applySoftDeletes a Bits hardLiveDocs parameter and pass null from the other two callers, instead of adding an overload.

Trade-offs

Same behaviour as A; touches two files, and no overload hides the difference.

Make tryUpdateDocValue return -1, or drop the reset, when the target is hard-deleted.

Trade-offs

Misses the failing cases: when the try call runs, the hard delete is still buffered (it is applied later, during IndexWriter.java:586-596, before the update is written at :608) or not issued yet (testUndeleteThenHardDelete), and testTermApiOnly never calls try*.

State that a null soft update can revive hard-deleted documents.

Trade-offs

Contradicts IndexWriterConfig.java:561-563 and leaves the negative count, the wedged writer and the CheckIndex failure. Separately, the stale "no API support to un-delete" sentence (IndexWriterConfig.java:557-559) could be fixed with any option.

Method: model-based randomized testing (the study's non-formal baseline), with later re-finds by formal tools that were told the bug class.

• A 2026-10-22 study of Lucene by AI tools covered the IndexWriter/NRT layer because an earlier bug there (now #16900) was its starting point. The experiment's prompt scoped it to IndexWriter and NRT readers and listed a broad operation alphabet that named soft deletes, tryUpdateDocValue and doc-values updates of the soft-deletes field among many others; its only bug hint was a file name pointing at the earlier bug's class. It was not told about this bug or its symptom.
• An AI tool wrote a model-based testing harness: random IndexWriter operation sequences against an executable reference model. Lucene's own -ea assertions plus CheckIndex flagged this bug 4.2 s into a smoke run, and 51.3 s and 21.7 s into the two base hunt.
• TLA+, Alloy, P, Dafny, JBMC/JPF and OpenJML models re-found it later, with the bug class given as a hint.
• Independent AI verifiers reproduced it and voted "new bug". This hunt re-verified it on ea57360912ed, added the merge and retention-policy cases, ran the release history and the prior-art search, had it reviewed adversarially by OpenAI GPT-5.6, and validated the fix.