deleteDocuments(Query) rewrites the query on each segment alone, so it can delete a different set than search returns

IndexWriter.deleteDocuments(Query...) and updateDocuments(Query, docs) do not run the query over the whole index the way a search does. When the buffered delete is applied, FrozenBufferedUpdates.applyQueryDeletes builds a new IndexSearcher over each segment, rewrites the query there and deletes that segment's matches. For per-document predicates (a term, a range) that changes nothing. For queries whose matches depend on the other documents of the index (kNN's top k, the default FuzzyQuery's top 50 terms and other top-N rewrites, CommonTermsQuery, MoreLikeThisQuery, score ranges) each segment picks its own matches, so the delete removes more or fewer documents than IndexSearcher.search returns for the same query: on two segments a kNN query with k = 2 deletes 4 documents and a default FuzzyQuery deletes 60 where search finds 50, while MoreLikeThisQuery and a score range delete none of the 6 and 4 hits. With the same documents committed in one segment every delete equals the search, so the segment layout alone decides what is deleted, silently and permanently once committed. It is as old as the API (2.4.0, 2008): executed on 22 releases from 2.4.0 to 10.5.2, on 10.6.0 (branch_10_6 at RC1 4ae946e5a9b) and on main, all affected, and every release tag since 2.4.0 has the same per-segment code. updateDocuments(Query, ColumnBatch) (since 10.5.0) uses the same delete path (by reading).

• Segment: an index is a set of independent segments; each commit or flush can add one, and merges combine them. Users do not choose the layout; the default merge policy changes it over time.
• Rewrite: before a query runs, IndexSearcher.rewrite turns it into a primitive query using the searcher's reader. kNN does its real work here (it finds the top k), and a FuzzyQuery picks its top 50 terms here.
• Delete packet: deleteDocuments(Query) only buffers the query. It is applied later (at flush, commit, NRT reopen or merge) by FrozenBufferedUpdates.applyQueryDeletes.

1. applyQueryDeletes loops over the segments the packet applies to and, for each one, runs new IndexSearcher(readerContext.reader()), searcher.rewrite(query) and createWeight (FrozenBufferedUpdates.java:397-400 on main).
2. So every reader-dependent decision is made per segment: kNN's top k over that segment's vectors (AbstractKnnVectorQuery.rewrite, :75-118), the top 50 fuzzy terms among that segment's terms (FuzzyQuery.java:34-36, 56), CommonTermsQuery's high/low frequency split against that segment's maxDoc (CommonTermsQuery.java:113, 154-155), MoreLikeThis's minDocFreq and idf against that segment (MoreLikeThis.java:634-660), and the BM25 statistics behind a score range.
3. The deleted set is the union of the per-segment results. An independent check that replaces "search on the index" with "union of one-segment searches" as the expected result passes all 9 of its tests.
4. One call can even be resolved several times: once for the segments that already exist, and once more for each segment flushed from documents buffered before the call (a segment-private packet always covers exactly one segment). So for ordinary query deletes the writer never builds one searcher over all segments (an exact MatchAllDocsQuery is not resolved at all: it becomes deleteAll()). One segment is not always safe either: in the 2016-cell matrix, when the documents are still buffered at the call and more are added after it, the one flushed segment also holds the later documents, and the default fuzzy query deletes 49 of its 50 search hits, top-terms(5) 2 of 5, kNN k = 2 none of 2 and a score range 31 of 36.

With the same documents committed in one segment, every delete equals the search. On three segments the first probe saw kNN k = 2 delete 6 documents and the default fuzzy query 90 where search finds 50. In a bounded-exhaustive check (every contiguous layout of 8 documents, every set partition of 6), search returned one fixed set per query on every layout, while the kNN, top-2 fuzzy, top-terms and score-range deletes mismatched on 127 of 128 and 202 of 203 layouts; the per-document controls (a term, a point range, a default prefix, a fuzzy query whose 1024-term cap is never reached, and a boolean of a term and a range) matched in all 1655 checks.

• Who is hit. Applications that delete or update by a query whose matches depend on other documents: kNN ("delete the k nearest", or replace them with updateDocuments(Query, docs)), a parsed or typed fuzzy term in a delete-by-query feature, top-N MultiTermQuery rewrites, CommonTermsQuery, MoreLikeThisQuery, and score ranges (FunctionRangeQuery, FunctionMatchQuery).
• Lucene's own Luke does this. Its "Delete Docs" button asks "Are you sure to permanently delete the documents?", then passes the current search query to IndexWriter.deleteDocuments and commits. By reading (Luke itself was not run), a user who searched name:lucenex~2 on such an index and saw 50 hits would lose 60 documents.
• Default configuration. Default IndexWriterConfig, codec and merge policy; indexes with two or more segments, the normal case, when a segment's own top k or top terms differ from the index's. The JUnit pins the layout with NoMergePolicy; the independent reproducer also fails with the default TieredMergePolicy, and the 2016-cell matrix shows the same through an NRT reader, through updateDocuments(Query, docs) and with the last segment still buffered.
• Silent and permanent. No error, with assertions on or off. Over-deletion is data loss once committed; under-deletion leaves documents that a retention or erasure delete was meant to remove.
• Not affected by this. Per-document predicates. Executed controls: a term, a point range, a default prefix, a fuzzy query whose 1024-term cap is never reached and a boolean of a term and a range matched search in all 1655 layout checks; a term, a point range, a prefix, a wildcard, the same fuzzy query and a vector-similarity threshold matched in all 576 control cells of the matrix. Term-set and field-value function queries were not executed; they are per-document by reading. This covers this mechanism only: stale doc values in delete-by-query (#8398) is a separate, known bug.
• How likely. Rare: a ranking or statistics query is an unusual delete predicate, no upstream test uses one, and the searches below found no prior report.
• Workaround. Search a near-real-time reader and delete the hits by a unique id term (or try tryDeleteDocument per hit, an expert best-effort call that returns -1 when it cannot delete); deleting by id removes exactly the search hits. Pre-rewriting a fuzzy query on a whole-index searcher also works, but not a kNN query: the rewritten kNN query throws "This DocAndScore query was created by a different reader" when the delete is applied and closes the writer (a separate, known bug).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: live docs after deleteDocuments(k=1) expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.index.TestDeleteByKnnQuery.testDeleteMatchesSearchOnTwoSegments(TestDeleteByKnnQuery.java:52)

• Contract. deleteDocuments(Query...): "Deletes the document(s) matching any of the provided queries" and "@param queries array of queries to identify the documents to be deleted" (IndexWriter.java:1875, 1879); updateDocuments(Query, docs): "take a query instead of a term to identify the documents to be updated" (:1613-1614); updateDocuments(Query, ColumnBatch) (since 10.5.0): "takes a query instead of a term to identify the documents to be deleted" (:1560-1561). No restriction on the query kind; the only special case is MatchAllDocsQuery, turned into deleteAll().
• The query classes define matches over the whole reader. KnnFloatVectorQuery: "@param k the number of documents to find"; FuzzyQuery: "Only the top terms are used". On #11838 (2022) a Lucene developer explained that kNN does its work in rewrite to "match the global top k across segments, as opposed to matching k docs per segment. This means the number of documents it matches doesn't depend on the number of segments".
• Nothing documents per-segment resolution. No javadoc, package doc, MIGRATE.md or CHANGES.txt entry says delete queries are resolved per segment or must be per-document predicates; the only "per segment" wording is the class comment of the package-private BufferedUpdatesStream, about resolving deletes to docIDs.
• Untested, not decided. Upstream tests delete by query only with per-document predicates.
• Lucene's own usage. Luke's "Delete Docs" passes the current search query to deleteDocuments: it assumes a delete matches the search.
• History. Per-segment resolution is as old as the API (LUCENE-1194, 2008); its discussion and the later rewrites of the resolution code (LUCENE-6688, LUCENE-7868, LUCENE-8242) never discussed reader-dependent queries, which mostly arrived later (fuzzy with 50 terms in 4.0.0, CommonTermsQuery in 4.1.0, kNN in 9.0.0); the two that existed in 2008, FuzzyQuery's top-1024-terms rewrite and contrib MoreLikeThisQuery, are not mentioned in the LUCENE-1194 discussion.

The intended-behaviour checker's strongest case (rated moderate by every checker): per-segment, deferred resolution is the original, deliberate design, and one delete is resolved in several packets, so the writer never has a whole-index view. On #8398 (stale doc values in delete-by-query) a committer wrote that the resolution code "has a hard assumption that segments are write-once as far as queries resolving to docIDs is concerned" and leaned "towards requiring/documenting" a limitation; another suggested "Maybe we should just document the limitation". Query#rewrite depends on the searcher and matching is per leaf, so each deleted document does match the query as rewritten on its own segment; kNN, MLT, CommonTerms and score ranges are ranking tools, not predicates; kNN search is approximate anyway, and the writer exposes no snapshot of the index at the time of the call; an exact way to delete what a search returned is a delete by a unique id.

Why it fails: nothing tells users, the queries are accepted silently, and the result is permanent. It fails in the cleanest case (everything committed, then one delete), where search returns one fixed set on every layout and only the layout changes what is deleted. #8398 is still open as a bug (the same committer opened with "This is indeed a bug"). Luke relies on delete matching search, and for kNN there is no general way to keep the query as the delete query and get the search set: pre-rewriting it on the whole index throws when the delete is applied. Verdict: a real, undocumented, silent defect; the fix may legitimately be documentation.

Two patches were written and run in a scratch worktree at a9779ef3c193. Neither is presented as the obvious fix: the checkers expect upstream to answer with documentation first.

javadoc-per-segment-note.diff, +18 lines in IndexWriter.java, javadoc only: a NOTE on deleteDocuments(Query...) (which excepts MatchAllDocsQuery, turned into deleteAll()) and a pointer from both updateDocuments(Query, ...) overloads (Iterable and ColumnBatch). Revised after the fact checks; the first version lacked the MatchAllDocsQuery exception, the 'collect every hit' condition and the ColumnBatch pointer.

 * <p><b>NOTE</b>: each query (other than a {@link MatchAllDocsQuery}, which deletes all
 * documents) is resolved against each segment on its own, with an {@link
 * org.apache.lucene.search.IndexSearcher} over that one segment, not against the whole index.
 * Queries whose matches depend on the other documents of the index or on index statistics, such
 * as {@link org.apache.lucene.search.KnnFloatVectorQuery} (the top k), {@link
 * org.apache.lucene.search.FuzzyQuery} with its default top-terms rewrite, or a range over
 * scores, can therefore delete more or fewer documents than {@link
 * org.apache.lucene.search.IndexSearcher#search(Query, int)} returns for the same query (with
 * {@code n} large enough to collect every hit), depending on how the index is split into
 * segments. To delete exactly the hits of a search, search a near-real-time reader and delete the
 * hits by an id term or with {@link #tryDeleteDocument(IndexReader, int)}.

Trade-offs

Smallest change; it describes what every release since 2.4.0 does; and it is in line with the tentative suggestions on #8398 ("would lean towards requiring/documenting", "Maybe we should just document the limitation"; about stale doc values, and still open). It leaves the silent wrong delete for anyone who does not read the note, and Luke's "Delete Docs" would need its own change (delete the hits by docID through an NRT reader). The note could also say that the internal searcher uses default settings (for example the default Similarity), which a related check found undocumented.

What the checks validated

Revised note, on a clean extraction of a9779ef3c193: ./gradlew :lucene:core:tidy formatted it; ./gradlew :lucene:core:check -x test with both patches and this doc's test: BUILD SUCCESSFUL (format check, ecj lint, forbidden APIs); ./gradlew :lucene:core:renderJavadoc: BUILD SUCCESSFUL, and the rendered IndexWriter.html holds the three notes with all six links resolved (the extracted text and link targets are saved). The first version passed the same checks. It changes no behaviour, so the repro still fails, by design.

rewrite-once-per-packet.diff, +49/-33 in FrozenBufferedUpdates.java. applyQueryDeletes first collects the segments the packet applies to (the same two skip checks), wraps their readers in one new MultiReader(readers, false), rewrites and weights each query once on an IndexSearcher over it, and then runs the scorer on each leaf with the unchanged docID-limit and index-sort logic.

try (MultiReader reader = new MultiReader(readers, false)) {
  final IndexSearcher searcher = new IndexSearcher(reader);
  searcher.setQueryCache(null);
  for (int i = 0; i < deleteQueries.length; i++) {
    final Query query = searcher.rewrite(deleteQueries[i]);
    final Weight weight = searcher.createWeight(query, ScoreMode.COMPLETE_NO_SCORES, 1);
    for (int s = 0; s < readers.length; s++) {
      final Scorer scorer = weight.scorer(reader.leaves().get(s));
      // ... unchanged: delete the matches below the packet's docID limit

Trade-offs

• Partial. It fixes the clean cases on segments that exist when the delete is applied: everything committed, then a delete (the repro's case), and the NRT and updateDocuments paths. It does not fix documents buffered before the call (still resolved in their own one-segment private packet: in the matrix the "last segment still buffered" path keeps all 28 of its mismatching BM25 cells), documents added after the call (20 of 30 remain), kNN after an earlier delete in the same writer (8 of 30 remain, a separate known bug about stale live docs), or the non-default-Similarity cells. A full fix needs one resolution point over all segments, including the buffered ones: a design change.
• Inside a private packet, top k is still chosen among all documents of the flushed segment, including those added after the call (then cut by the docID limit): unchanged from today.
• Cost: one MultiReader and one rewrite per packet instead of one rewrite per segment; no benchmark was run. Concurrency with merges was not reviewed beyond the test runs.
• A pre-rewritten kNN query still throws when applied and closes the writer (a separate, known bug).

What the checks validated

• Gradle, scratch worktree with the patch, seed DEADBEEF: this doc's test plus the whole core org.apache.lucene.index test package, TestKnn* and TestFuzzyQuery: 219 classes, 1536 tests, 0 failures (56 skipped), with assertions on and off; the 3 tests that fail on stock pass.
• The repro runner, stock main with the patched class shadowed: the finder's 7 tests pass (-ea, -da) and the independent reproducer's 9 tests pass; without the shadow the finder's class still fails 6 of 7.
• The 2016-cell matrix (BM25, 3 segments): the committed, NRT and updateDocuments paths drop from 28 of 30 mismatching cells to 0; the "last segment still buffered" path stays at 28 of 30; the "documents added after the call" path goes from 28 to 20. Per-document controls: 0 mismatches before and after. Identical with assertions off.
• Not run: the full Lucene test suite, nightly tests, benchmarks.

Throw from deleteDocuments for queries whose matches depend on other documents. Not run.

Trade-offs

There is no general way to tell this from a Query; a list of known classes (kNN, top-N rewrites, CommonTerms, MLT, score sources) would be fragile and would break code that accepts today's behaviour.

Resolve the query once against all segments, including the buffered documents, at the delete's point in time. Not run.

Trade-offs

Correct by construction, but it needs an NRT-like view of the buffered documents (or a flush) per delete: a much larger change to the write path.

Method: Differential Query Execution, the database-testing idea of running one predicate as a SELECT and as a DELETE, aimed at Lucene's write path. Non-formal by the hunt's rules: the first signal was an executed probe.

• Lead. In a cross-model ideation round, an independent model from another family (OpenAI GPT-5.6, testing-research framing) proposed comparing deleteDocuments(Q) with search(Q). Its probe on the stock main jar printed MISMATCH on 3 segments for kNN k = 2 (6 deleted vs 2 searched), the default FuzzyQuery (90 vs 50), TopTerms(5) (15 vs 5) and CommonTermsQuery (13 vs 40), and passed on 1 segment and for the TermQuery control (also on 10.5.2 and 9.12.3). A planning verifier re-ran every number.
• Hunt. A finder ran a Differential Query Execution matrix: 21 query families (6 per-document controls) x 6 write paths (committed, NRT, updateDocuments, last segment buffered, after an earlier delete, followed by later documents) x 1/3 segments x 2 merge policies x 4 Similarities = 2016 cells, then a bounded-exhaustive layout check (every contiguous layout of 8 documents and every set partition of 6, 4303 checks), and wrote the public-API JUnit.
• Checks. An independent reproducer, an executed review by a model from another family, an intended-behaviour devil's advocate, an adjudication and a credibility audit (section "Independent checks").