IndexWriter#hasDeletions() returns false after a writer restart when the only deletions are committed soft deletes

IndexWriter#hasDeletions() returns true if buffered changes or segments in the writer's reader pool have deletions (these checks count soft deletes), and otherwise if any committed segment has SegmentCommitInfo#hasDeletions(), which is delGen != -1: the segment has a live-docs file, so only hard deletes count. A soft delete never writes a live-docs file; since 7.4.0 the commit records it as a per-segment soft-delete count instead. A writer opened on an existing index has nothing buffered and nothing pooled, so with a soft-deletes field configured and only soft deletes committed it answers false, while its own getDocStats() counts the deleted documents (maxDoc 2, numDocs 1 in the repro). The answer turns true, with no change to the index, once DirectoryReader.open(writer) loads the segment into the pool (reopened writer with NoMergePolicy; with the default merge policy that open merges the soft-deleted document away), and addIndexes(Directory) of such segments gives the same false. No Lucene code calls the method, so searches, merges, live docs and index files are unaffected: a minor, deterministic wrong answer in a public statistics method, in every release since 7.4.0 and on main. The one-line fix info.getDelCount(softDeletesEnabled) > 0, the count numDeletedDocs(info) already uses, passes the repro and the whole core test suite.

// IndexWriter.java:1444-1460
/**
 * Returns true if this index has deletions (including buffered deletions). Note that this will
 * return true if there are buffered Term/Query deletions, even if it turns out those buffered
 * deletions don't match any documents.
 */
public synchronized boolean hasDeletions() {
  ensureOpen();
  if (bufferedUpdatesStream.any() || docWriter.anyDeletions() || readerPool.anyDeletions()) {
    return true;                          // :1451-1452  buffered and pooled: count soft deletes
  }
  for (final SegmentCommitInfo info : segmentInfos) {
    if (info.hasDeletions()) {            // :1455  delGen != -1: hard deletes only
      return true;
    }
  }
  return false;
}

1. The commit. One segment _0: maxDoc 2, hard-delete count 0, soft-delete count 1, delGen -1 (no live-docs file). SegmentCommitInfo#getDelCount(boolean includeSoftDeletes) adds the two counts (SegmentCommitInfo.java:453-455).
2. The pooled check counts soft deletes. ReaderPool#anyDeletions() (ReaderPool.java:156-163) is true if a pooled segment has rld.getDelCount() > 0, and PendingDeletes#getDelCount() is info.getDelCount() + info.getSoftDelCount() + numPendingDeletes() (PendingDeletes.java:242-246). A buffered (unflushed) soft-delete update makes the buffered check docWriter.anyDeletions() true.
3. The committed-segment check does not. SegmentCommitInfo#hasDeletions() is return delGen != -1; under the javadoc "Returns true if there are any deletions for the segment at this commit." (SegmentCommitInfo.java:319-322).
4. A reopened writer has an empty pool. The probe of the reopened writer prints bufferedUpdatesStream.any()=false docWriter.anyDeletions()=false readerPool.anyDeletions()=false; _0 info.hasDeletions()=false info.getDelCount(true)>0=true, so hasDeletions() returns false.
5. The same writer counts the delete elsewhere. numDeletedDocs(info) uses info.getDelCount(softDeletesEnabled) for an unpooled segment (IndexWriter.java:891-903, :898), and getDocStats() sums it (:6824-6834): maxDoc 2, numDocs 1.
6. Pooling flips the answer. After DirectoryReader.open(writer) (reopened writer with NoMergePolicy) the segment is pooled, readerPool.anyDeletions() is true and hasDeletions() returns true; the index files and the commit are unchanged. With the default merge policy, the probe saw the segment merged away while the reader opened (new segment _1, maxDoc 1, the soft-deleted document reclaimed), and then false is correct.
7. addIndexes(Directory) copies the segment as it is, without pooling it: the receiving writer reports maxDoc 2, numDocs 1 and hasDeletions() false.

• Who is hit. Applications that configure a soft-deletes field (opt-in, IndexWriterConfig#setSoftDeletesField) and call IndexWriter#hasDeletions() on a writer opened on an existing index whose committed segments still hold soft-deleted documents (as after a restart), or after addIndexes(Directory...) of such segments. No timing and no scale are needed; assertions on or off.
• Default config is not affected. Without soft deletes every deletion writes a live-docs file and the committed-segment check is right; the hard-delete control passes on every version run.
• What goes wrong. A public statistics method gives a wrong answer, and the answer for an unchanged index changes when an NRT reader is opened (with NoMergePolicy on that writer). An application that asks it after a restart whether deletes need attention (reclaiming them, reporting them) is told there are none.
• What does not go wrong. No Lucene main code calls IndexWriter#hasDeletions() (the other hasDeletions() calls are reader- or SegmentCommitInfo-level). Searches, live docs, merges and index files are unchanged; soft-deleted documents stay hidden from writer readers and from SoftDeletesDirectoryReaderWrapper. Upstream tests call the method only without soft deletes (TestTryDelete, TestTieredMergePolicy, and an assert in TestFunctionRangeQuery in lucene/queries).
• Not affected either: the writer that made the soft delete keeps the segment pooled and answers true, even with setReaderPooling(false), after a merge that retains the soft-deleted document, and for a document added already soft-deleted.
• Impact class: minor (a wrong return value with no in-tree effect); priority low.

NoMergePolicy on the first writer only keeps the soft-deleted document on disk: with the default merge policy a merge reclaimed it before the reopen, and then false is correct (measured while minimizing). The reopened writer uses the default config plus the same soft-deletes field (a writer without it cannot open this index).

The same failure, by the repro harness: main 645d4fe14a20 and the 10.6.0 jars, assertions on and off. The hunt's longer 3-test class (the NRT-reader flip and a hard-delete control) fails the same way on 7.4.0, 7.7.3, 8.0.0, 8.11.4, 9.0.0, 9.12.3, 10.0.0 and 10.5.2. Use this one test as the regression test too.

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on
TestIndexWriterHasDeletionsSoftDeletes > testCommittedSoftDeleteAfterWriterReopen FAILED
    java.lang.AssertionError: hasDeletions() vs getDocStats() maxDoc=2 numDocs=1 expected:<true> but was:<false>
        at org.apache.lucene.index.TestIndexWriterHasDeletionsSoftDeletes.testCommittedSoftDeleteAfterWriterReopen(TestIndexWriterHasDeletionsSoftDeletes.java:43)

# repro harness, the same message: main -ea, main -da, 10.6.0 -ea, 10.6.0 -da (1 run, 1 failed each)
# with fix A: the test passes (gradle; harness -ea and -da)

• Contract. IndexWriter.java:1445-1447: "Returns true if this index has deletions (including buffered deletions). Note that this will return true if there are buffered Term/Query deletions, even if it turns out those buffered deletions don't match any documents." The only approximation it documents is a false positive.
• Soft deletes are deletes for the writer. IndexWriterConfig.java:544-547: "If a document is marked as soft-deleted, the document is treated as if it has been hard-deleted through the IndexWriter API ... index readers obtained from the IndexWriter will reflect all deleted documents in its live docs." CHANGES.txt:7388-7389 (LUCENE-8233): "Soft deletes are accounted for inside the index writer".
• The method disagrees with itself. Its buffered and pooled checks count soft deletes, the committed-segment loop does not, so for one unchanged index the answer depends on whether a reader happened to be pooled. No javadoc says so.
• The same writer disagrees. getDocStats() and numDeletedDocs(info) count the soft delete for the same unpooled segment.
• History: an oversight. LUCENE-8233 (2018) made the writer account for soft deletes and did not touch the method. LUCENE-8341 (2018) changed its pooled check (readerPool.anyPendingDeletes() to readerPool.anyDeletions()) and the unpooled branch of numDeletedDocs (info.getDelCount() to info.getDelCount(softDeletesEnabled)), but left the loop; its commit message says the point is that "the number of document can be read from commit points without opening a full blown reader". The discussions of #9280, #9388 and #9247 never mention hasDeletions.

Strongest case for "intended": hasDeletions means hard (live-docs) deletions, like SegmentCommitInfo#hasDeletions() and a plain DirectoryReader.open(dir), which shows soft-deleted documents as live; and the writer does the same on purpose in numDeletesToMerge: // if we don't have a pooled instance let's just return the hard deletes, this is safe! (IndexWriter.java:6454), a fallback a reviewer on #13324 also called deliberate.

Why it fails:

1. The method's own pooled and buffered checks already count soft deletes, so "hard only" does not describe the method.
2. numDeletesToMerge returns a merge-policy-dependent count (a retention policy keeps soft-deleted documents) that needs a reader to compute, so under-counting is its safe direction. hasDeletions depends on no policy, its javadoc documents only false positives, and the exact soft count has been in the commit, readable in constant time, since LUCENE-8341.
3. A writer without the soft-deletes field cannot even open such an index (IllegalArgumentException), so every writer that can see the segment is soft-aware.
4. No javadoc, test, CHANGES or MIGRATE entry, issue, PR or list thread found calls this behaviour intended.

What remains are impact arguments (no in-tree caller), not intent arguments. Verdict: not intended, a minor defect; maintainers may still prefer a javadoc change or deprecation (option C) to the one-line fix.

Two code fixes were run; A is recommended. Workaround for users today: DocStats s = w.getDocStats(); boolean committedDeletes = s.maxDoc > s.numDocs; counts unpooled soft deletes (IndexWriter.java:6824-6834; it does not see buffered, unapplied deletes), or open an NRT reader first.

Use the same expression as numDeletedDocs(info) (IndexWriter.java:898): constant time per segment, reads the hard and soft counts stored in the commit, so it pulls no reader and depends on no merge policy.

--- a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
@@ -1452,7 +1452,7 @@ public class IndexWriter
       return true;
     }
     for (final SegmentCommitInfo info : segmentInfos) {
-      if (info.hasDeletions()) {
+      if (info.getDelCount(softDeletesEnabled) > 0) {
         return true;
       }
     }

Trade-offs

For hard deletes A asks delCount > 0 where the old code asked delGen != -1. The writer writes a live-docs file only with pending deletes (PendingDeletes.java:171-172), and SegmentReader asserts that no live-docs file means delCount == 0 (SegmentReader.java:100-105); the suites below found no difference. If a reviewer wants no change at all for hard deletes, take B.

What the checks validated

• Scratch worktree of main 645d4fe14a20 (removed afterwards), seed DEADBEEF: stock, the test fails through gradle; with A it passes.
• The repro and the 8 closest suites with A, -Ptests.asserts=true and false: TestIndexWriterHasDeletionsSoftDeletes, TestTryDelete, TestTieredMergePolicy, TestSoftDeletesRetentionMergePolicy, TestSoftDeletesDirectoryReaderWrapper, TestPendingSoftDeletes, TestIndexWriterDelete, TestAddIndexes, TestIndexWriter: 233 tests, 8 skipped, 0 failures each time.
• All of :lucene:core:test with A (assertions on): 8,898 tests in 749 classes, 322 skipped, 0 failures.
• Earlier, with A shadowed in: the hunt's 3 tests, the independent reproducer's 8 and the 6 written for a cross-model review all pass.

if (info.hasDeletions() || (softDeletesEnabled && info.getSoftDelCount() > 0)). Behaviour without soft deletes is unchanged by construction.

-      if (info.hasDeletions()) {
+      if (info.hasDeletions() || (softDeletesEnabled && info.getSoftDelCount() > 0)) {

Trade-offs

Slightly longer; a second notion of "has deletions" next to numDeletedDocs.

What the checks validated

The repro passes with B shadowed in, assertions on and off (repro harness). Not run through the suites.

Document that committed soft deletes count only once the segment is pooled, or deprecate the method (no in-tree caller).

Trade-offs

Leaves the method inconsistent with getDocStats() and with its own pooled check; not recommended.

• Method (non-formal): model-based randomized testing in a hunt unit on soft deletes. Random scripts of soft updates, doc-values soft deletes, hard deletes, flushes, NRT reopens, commits, forced merges, addIndexes, writer reopens and rollbacks ran on the stock jars; after each observation an exact executable reference model of live, soft-deleted, hard-deleted and retained documents was compared with the writer.
• Signal. After a writer reopen the model expects hasDeletions() true whenever a deleted document is committed; the note "hasDeletions-after-reopen expected=true got=false anyHard=false" fired 86 times in the first 150 scripts and 1,341 times in 1,500. A later bounded-exhaustive run hit it too.
• Reduction. The finder wrote a public-API JUnit (3 tests, one control); it was later minimized to the one test above.
• The finder labelled it semi-formal; the registry records non-formal, because the first signal came from random scripts.