LatLonPoint.nearest reports an exact total hit count that includes deleted documents

LatLonPoint.nearest(searcher, field, lat, lon, n) finds the n indexed points closest to a location. It skips deleted documents when it picks the hits, but it takes its total from PointValues#getDocCount(), an index statistic that still counts deleted documents, and labels that total TotalHits.Relation.EQUAL_TO (exact). After an ordinary delete or update, and until a merge drops the old copy, the hits are right but the "exact" total is too high by the number of non-live documents that carry the field; it can exceed reader.numDocs(). Core has had it since nearest moved there in 10.0.0: all 12 10.x releases up to 10.5.2, 10.6.0 (by reading) and main ea57360912ed.

• LatLonPoint.nearest: a public static method that walks the BKD tree of a LatLonPoint field and returns the n closest points as TopFieldDocs. Its javadoc says it is "functionally equivalent to running MatchAllDocsQuery with a LatLonDocValuesField#newDistanceSort", but faster.
• TotalHits: the value plus a relation. EQUAL_TO means the value is the exact hit count; GREATER_THAN_OR_EQUAL_TO means it is a lower bound.
• Deleted but not merged: a delete, or the old copy left by updateDocument or softUpdateDocument, only clears the document's live bit. The document stays in its segment, and in the segment's statistics, until a merge rewrites the segment without it.

1. LatLonPoint.java:421-430 loops over the segments. For each one it adds points.getDocCount() to the total (:425): "the total number of documents that have indexed at least one point" (PointValues.java:495-496), deletions not subtracted.
2. The same loop collects each segment's live docs (:428), but only for NearestNeighbor, whose visitor skips every document whose live bit is clear (NearestNeighbor.java:156). So the hits are right.
3. :442 returns new TotalHits(totalHits, TotalHits.Relation.EQUAL_TO).

softUpdateDocument with a soft-deletes field does the same (maxDoc 4, numDocs 3, "4 hits"), and so does a reader that hides documents through getLiveDocs(). For a fixed field and reader, the wrong number does not depend on lat, lon or n: the total is summed before the search runs.

• Who is hit. Callers of the public, non-experimental static LatLonPoint.nearest that read TopFieldDocs.totalHits: a displayed result count, a page count, or any logic that trusts an EQUAL_TO total.
• Default, not opt-in. No setting is needed: the default IndexWriterConfig, codec, merge policy and IndexSearcher show it after one updateDocument. Tests use NoMergePolicy only to keep the deleted document in place.
• Size of the error. The total is too high by the number of non-live documents that carry the field in the searched segments. An update deletes (or soft-deletes) every document that matches its term, so with unique ids it adds one: 100 updates of 1,000 documents give "1100 hits" (executed). The old copy counts as long as a segment holds it: usually until a merge, longer under a SoftDeletesRetentionMergePolicy. The default TieredMergePolicy targets at most 20% deleted documents (TieredMergePolicy.java:121-124); that is a goal for choosing merges, not a hard cap. For a dense field at 20% deleted the total is about 25% above the live count, and for a sparse field the relative error can be larger. This is arithmetic, not measured.
• What the damage looks like. A total that is too high and claims to be exact. Possible application effects (hypothetical): a count larger than the results that can be served, or a loop that pages until it has totalHits results.
• Reach. In-tree, the only non-test caller is the JMH benchmark (NearestNeighborBenchmark.java:109). A code search of the default branches of Elasticsearch, OpenSearch and Solr (2026-10-25, repeated 2026-10-26) found no call; code search has limits, and private applications are unknown. So it mainly affects applications that use Lucene directly.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected exactly 2 hits (or a lower bound <= 2), got 3 hits
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.search.TestNearestTotalHitsDeletedDoc.testDeletedDocNotCounted(TestNearestTotalHitsDeletedDoc.java:50)

• Contract. TotalHits.java:30-32: the relation says "Whether value is the exact hit count, in which case relation is equal to EQUAL_TO, or a lower bound"; :38 "The total hit count is equal to value". TopDocs.java:29: "The total number of hits for the query." Weight.java:182: an exact count "Counts the number of live documents that match".
• Deleted documents are not hits. nearest filters them out of the hits (NearestNeighbor.java:156), so a total above reader.numDocs() is wrong under any definition of a hit.
• Lucene's own count shortcuts. TermQuery.java:256, PointRangeQuery.java:353 and FieldExistsQuery.java:247-251 ("No deletions: we can use points or terms doc count as a proxy") use the statistic as a count only when hasDeletions() == false; FieldExistsQuery.java:261-270 also accepts the case where every document has the field ("All docs in the leaf (live or deleted) have the field. Return the count of live docs."). Among these count paths nearest is the outlier. They are precedents, not a universal rule: PointValues.getDocCount(IndexReader, field) (PointValues.java:132-147) computes the same deletion-blind sum, but as a documented statistic, not as a hit count.
• Tests. TestNearest is the only core test that calls nearest; its deletion tests (:43, :76) check the hits only. A sandbox test (TestFloatPointNearestNeighbor.java:69) calls it once and reads scoreDocs only. No test reads its totalHits.
• History. The getDocCount sum came with the method in 2016 (LUCENE-7069, sandbox), when totalHits was a plain number documented as the total number of hits; EQUAL_TO came from LUCENE-8430 (2018), which converted every TopDocs producer; LUCENE-10266 (#897, 2022) moved it to core with the counting loop and the EQUAL_TO return unchanged. No review thread, CHANGES or MIGRATE entry discusses the total.
• Framing for the issue. Base it on the EQUAL_TO contract and on "exact total > reader.numDocs()", not only on the MatchAllDocsQuery equivalence (see the devil's advocate). The multi-valued oddity (documented duplicates can make scoreDocs.length exceed totalHits) is a side issue.

The strongest "works as designed" case (built by the works-as-designed review and the devil's-advocate verifier; the cross-model reviews rate it moderate): nearest is a specialized BKD walk, not a Query, and the TopDocs javadoc scopes that class to hits "returned by IndexSearcher#search(Query,int)" (TopDocs.java:26); its own javadoc never says what totalHits holds; the total can be read as the field's physical cardinality, the deletion-blind sum that PointValues.getDocCount(IndexReader, field) also returns; the cheap O(segments) total has been there since 2016; an exact live count could need a pass over the points; LeafReader#searchNearestVectors documents its TotalHits as "the number of documents visited during the search" (LeafReader.java:249-252); and the MatchAllDocsQuery equivalence never pinned the total (sparse field, no deletions: nearest says 3, the equivalent 4).

Why it fails: nearest itself builds a TotalHits and labels it EQUAL_TO, which that record defines as the exact hit count, with no exception for callers outside IndexSearcher; nothing redefines TotalHits for LatLonPoint#nearest, and its javadoc presents it as functionally equivalent to a query; the physical-cardinality reading already has a documented home (PointValues.getDocCount) that does not claim to be a hit count; a total above the number of live documents is wrong under both readings (all live documents, or live documents with the field); and a lower bound marked GREATER_THAN_OR_EQUAL_TO keeps the O(segments) cost. Verdict: not intended, a defect.

Two fixes were built and checked; the recommended one keeps exact totals in the common case where every document has a location. Counting exactly in every case is possible but costs a pass over the points.

fix-dense-exact.diff, +19/-4 in LatLonPoint.java (with the two tests: fix-dense-exact-with-tests.diff). If every document of the segment has a point, the live documents with a point are exactly numDocs(); FieldExistsQuery.java:261-270 applies the same rule (and :233-234 for norms). Otherwise at most numDeletedDocs of the docCount documents are deleted, so docCount - numDeletedDocs is a valid lower bound.

int docCount = points.getDocCount(); // index statistic: ignores deletions
if (reader.hasDeletions() == false) {
  totalHits += docCount;
} else if (docCount == reader.maxDoc()) {
  // every document has the field, so the live documents that have it are exactly numDocs
  totalHits += reader.numDocs();
} else {
  // only a lower bound is cheap to compute
  totalHits += Math.max(0, docCount - reader.numDeletedDocs());
  exactTotalHits = false;
}
// after the loop:
// relation = exactTotalHits ? EQUAL_TO : GREATER_THAN_OR_EQUAL_TO

Trade-offs

• A sparse field with deletions gets a lower bound ("3+ hits" in the sparse case, where 3 is the true count) instead of an exact number. The bound can be loose: it subtracts every deleted document, also those without a point, so it can be 0 when most deleted documents have no point.
• It assumes a reader whose numDocs(), hasDeletions() and getLiveDocs() agree, as FilterLeafReader.java:37-38 asks of wrappers; an inconsistent wrapper is not repaired.
• Not benchmarked; the added work is a few integer reads per segment for standard segment readers, which cache numDocs() (IndexReader#numDocs() may be O(maxDoc) in other readers).
• Open draft #16847 edits the same lines (it widens int totalHits to long), so a PR may need a rebase.

What the checks validated

• Effect on the probe cases: "2 hits", "10 hits" and "1000 hits", all EQUAL_TO; the no-deletion control is unchanged.
• Gradle, clean checkout of ea57360912ed with the patch, seed DEADBEEF: the repro, the edge test, the finder's, the independent reproduction's, the default-config and the hidden-docs tests, TestNearest and every TestLatLon* suite in document and search: 28 classes, 255 tests, 0 failures, 19 skipped (18 nightly-only, 1 @Ignore).
• Assertions on and off, recorded: the same 28 classes through the repro harness on the ea57360912ed jars with the patched class, -ea and -da: 236 passed, 0 failed, the same 19 skipped. (A second gradle run meant for assertions off gave the same counts, but its log does not record the flag.)
• ./gradlew :lucene:core:check -x test (google-java-format, ecj lint including javadoc lint, forbidden APIs, licenses; error-prone skipped) passes with the patch and the two new tests. The same repro failed 2 of 2 in that checkout without the patch.

For segments with deletions add max(0, getDocCount - numDeletedDocs) and report GREATER_THAN_OR_EQUAL_TO. The simplest change.

Trade-offs

Any deletion turns the total into a lower bound, even where the exact number is free: "2+ hits", "10+ hits", "1000+ hits" in the probe cases.

What the checks validated

With the harness's shadow classes: the finder's, the independent reproduction's (4 run, 0 failed), the cross-model reviewers' and the default-config repros pass, and so does TestNearest (5 run, 0 failed); it also fixes 10.0.0's own file and the hidden-docs probe. The new repro passes; the edge test's two-segment case fails as designed (it expects EQUAL_TO).

For sparse segments with deletions, count the live documents with the field, for example with an IntersectVisitor over all points that checks the live docs. On a multi-valued field the visitor sees a document once per value, so the count must de-duplicate doc ids.

Trade-offs

Exact in every case, but O(points) per such segment and call, which works against the method's purpose.

Say that totalHits counts documents that have the field, including deleted but unmerged ones.

Trade-offs

Keeps an EQUAL_TO label on a number that is not the hit count, which contradicts TotalHits.

Method: code reading (the hunt's non-formal tier). The hunt unit on counts and totalHits made a grep inventory of every Weight#count override and every TotalHits(..., EQUAL_TO) producer, checking each against the contract "TotalHits never claims EQUAL_TO for a wrong count". LatLonPoint.nearest stood out because it labels an index statistic exact.

• The finder turned it into a JUnit test with a no-deletion control and ran it on the stock main jar with assertions on and off, and on 10.0.0 and 10.5.2.
• Before the candidate was filed, an independent reviewer from another model family (OpenAI GPT-5.6) tried to refute it, wrote its own committed-reader test, and voted real.
• Then an independent reproducer with two oracles, a second GPT-5.6 review with a works-as-designed lens, a devil's advocate for intended behaviour and an adjudication, who re-ran the independent reproduction's test with a new seed.