Range facet totals count the hits that the fastMatchQuery skipped

LongRangeFacetCounts, DoubleRangeFacetCounts and Long/DoubleRangeOnRangeFacetCounts accept an optional fastMatchQuery: only hits that match it are checked against the ranges. In the single-valued and range-on-range counting loops each segment first adds its whole hits.totalHits() to the dimension total, then subtracts only the checked hits that have no value or fall in no range, so the hits the fastMatchQuery rejected stay in FacetResult.value. With the values 0, 10 and 20, the range [0, 5] and a fastMatchQuery that matches only the document with 0, the range count is 1 but the total is 3 instead of 1. On main, a segment where the fastMatchQuery has no scorer is skipped before the add, so the same documents in two segments give 1: the total depends on the segment layout. The multi-valued path of the same classes counts each checked document and gives 1. Per-range counts, hits and drill-down are correct. Every release tested from 4.7.0 (LUCENE-5418, 2014) to 10.5.2, and main, is affected (the releases not run have the same code), range-on-range since 9.5.0. Counting each checked document instead of the bulk add fixes it; the whole lucene/facet test suite passes with that change.

• Range facets: LongRangeFacetCounts and DoubleRangeFacetCounts count, per range, the collected hits whose value (from field doc values or a LongValuesSource / DoubleValuesSource) falls in it; LongRangeOnRangeFacetCounts and DoubleRangeOnRangeFacetCounts do the same for indexed ranges (LongRangeDocValuesField).
• Dimension total: FacetResult.value, the "Total number of documents containing a value for this path" (FacetResult.java:30-34). For range facets that is the hits in at least one range: upstream testBasicLong collects 101 documents and asserts value=22.
• fastMatchQuery: optional; "we first test this Query to see whether the document should be checked for matching ranges" (FacetCountsWithFilterQuery.java:41-44). createIterator(hits) intersects the segment's hits with it and returns null when it has no scorer in the segment (:59-90, :75-78).

1. The single-valued loops skip a segment whose iterator is null, then add the whole segment's hit count: totCount += hits.totalHits(); (RangeFacetCounts.java:140 for field doc values, LongRangeFacetCounts.java:150, DoubleRangeFacetCounts.java:177, RangeOnRangeFacetCounts.java:100).
2. The loop visits only the documents of the intersected iterator. Checked documents with no value, or in no range (LongRangeCounter.finish(), LongRangeCounter.java:163-167), are subtracted: totCount -= missingCount (RangeFacetCounts.java:190-191, LongRangeFacetCounts.java:165-166, DoubleRangeFacetCounts.java:192-193, RangeOnRangeFacetCounts.java:123).
3. A hit the fastMatchQuery rejects is never visited, so it is added and never subtracted. The total becomes "documents in some range" plus "rejected hits of segments where the fastMatchQuery has a scorer".
4. A segment where the fastMatchQuery has no scorer is skipped before the add, so its rejected hits are not counted: the total depends on how documents are spread over segments. A term query has no scorer in a segment that lacks its term; a point range query has none in a segment with no points for the field or whose minimum and maximum values lie outside the range (PointRangeQuery.java:252-275), but a segment whose values straddle the range without a match gets an empty scorer, and its rejected hits are counted. A doc-values range query returns no scorer only when the segment lacks the field or a doc-values skip index rules the range out (SortedNumericDocValuesRangeQuery.java:124-132, :182-186); in the diagram's probe (no skip index) it had a scorer in both segments, so every rejected hit was counted.
5. The multi-valued loops count each checked document (totCount++, RangeFacetCounts.java:168, :180; LongRangeFacetCounts.java:199, :211; DoubleRangeFacetCounts.java:226, :238) and are right. The field path switches every segment to the multi-valued loop when one hit segment is truly multi-valued (RangeFacetCounts.java:95-106, :126).

Values 0, 10 and 20; MatchAllDocsQuery; one range [0, 5]; fastMatchQuery = TermQuery(tag:in), which matches only the document with 0.

Every range count is 1. The same numbers on main with assertions on and off, on 10.5.2 and on 9.12.3. Larger examples from the checks: 10 documents, range [0, 2], fastMatchQuery [0, 4]: value 8 where 3 is right; a fastMatchQuery that matches only an out-of-range document: every range count 0 and value 9.

• Who is exposed. Applications that pass a fastMatchQuery to LongRangeFacetCounts or DoubleRangeFacetCounts (with a LongValuesSource / DoubleValuesSource, or field doc values when no hit segment is truly multi-valued) or to LongRangeOnRangeFacetCounts / DoubleRangeOnRangeFacetCounts, and read FacetResult.value, for example to show "N results within 10 km". Lucene's own DistanceFacetsExample passes a bounding box as the fastMatchQuery (DistanceFacetsExample.java:215-231); on java-user@ (2022) gsmiller described it as extra filtering "to only documents that might match one of the ranges being faceted on", such as a price filter around price ranges.
• Default vs opt-in. The fastMatchQuery is optional (the constructors without it pass null). RangeFacetCounts, LongRangeFacetCounts, DoubleRangeFacetCounts and FacetCountsWithFilterQuery are @lucene.experimental; the range-on-range counters carry no such tag. Nothing else needs a non-default setting: IndexWriterConfig, codec and searcher can be the defaults (the repro sets NoMergePolicy only to pin the segment layout).
• Scale. Any index size. The total is too high by the hits the fastMatchQuery rejects in segments where it has a scorer, at most all main-query hits. In 60 random indexes, 2,160 of 4,800 counter checks were overcounted, by up to 1,269.
• Silent. No exception and no log line, and the per-range counts look right. Nothing is stored wrong, so a fix needs no reindex.
• Not affected. Per-range counts, childCount, hits, scores and drill-down; the multi-valued paths; MatchingFacetSetsCounts; any counter without a fastMatchQuery over an ordinary FacetsCollector (a sampled collector is a separate case, see the fix). No use of these classes was found on the default branches of three large downstream search engines (9 code searches); that is "no known use", not proof.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<1> but was:<3>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.facet.range.TestLongRangeFacetFastMatchTotal.testFastMatchQueryDoesNotInflateTotal(TestLongRangeFacetFastMatchTotal.java:57)

• Contract. FacetResult.java:30-34: "Total number of documents containing a value for this path ... it will only be counted once in this value." FacetCountsWithFilterQuery.java:41-44: "we first test this Query to see whether the document should be checked for matching ranges. If this is null, all documents are checked." LongRangeFacetCounts.java:76-79: "only documents passing the filter are checked for the matching ranges, which is helpful when the provided LongValuesSource is costly per-document, such as a geo distance." LongRangeOnRangeFacetCounts.java:27-31: the class counts "all the documents in the FacetsCollector (or that match a fast match query) that contain ranges that "match" the provided ranges". CHANGES.txt:12787-12792 (LUCENE-5418): a fast-match filter "to avoid computing the value for documents that are out of bounds".
• Tests. testBasicLong asserts value=22 for 101 collected documents (TestRangeFacetCounts.java:73-107). The randomized tests that pass a fastMatchQuery assert only per-range counts; testCustomDoubleValuesSource asserts value=3 with a fastMatchQuery that wraps MatchAllDocsQuery and rejects nothing (:1671-1700); the demo test's three documents all lie inside the bounding box. No test asserts the total under a fastMatchQuery that rejects hits, and the fix breaks none.
• History. LUCENE-5371 (2013) made the total subtract the hits in no range. LUCENE-5418 (2014) added the fast-match filter in front of the older bulk add; its 16 comments, and the 80 texts of #11901 (range-on-range, 2022), never mention the total. The field doc-values loop (since 9.0.0) and range-on-range (since 9.5.0) put the add after the null check from the start, so their totals have depended on segment layout since they were added; #12408 (9.11.0 / 10.0.0), a lazy-initialisation change, did the same for the LongValuesSource / DoubleValuesSource loops, which until then counted every rejected hit.
• Precedent. The 2021 dev@ thread "FacetResult#value semantics?" (mikemccand: "we really should strive to count each hit only once") led to LUCENE-9952, 9953 and 9964, which fixed double counting in other classes.

The strongest "intended" case: the classic range classes are @lucene.experimental; the totCount field comment says "Total number of hits" (RangeFacetCounts.java:49, RangeOnRangeFacetCounts.java:45) and the code seeds it with hits.totalHits(); the bulk add has stood since 2014; and the range-on-range javadoc calls the fastMatchQuery a way "to quickly discard hits using some heuristic" (LongRangeOnRangeFacetCounts.java:63).

Why it fails: the code already subtracts the checked hits in no range, so the total is not "all hits" in any configuration (the comment is stale since LUCENE-5371, and testBasicLong asserts 22 for 101 hits); no reading gives 3 in one segment and 1 in two for the same documents; the same class gives 1 on its multi-valued path; @lucene.experimental is about API stability, not wrong numbers. Verdict: a defect (adjudication: confirmed, high; the cross-model review rated the intended case weak). It lowers the severity: only the aggregate is wrong.

Over an ordinary FacetsCollector without a fastMatchQuery every hit is checked, so totals do not change; with one, the total becomes the checked documents in some range, which the multi-valued loops and MatchingFacetSetsCounts already compute. Side effect, executed: RandomSamplingFacetsCollector keeps each segment's unsampled totalHits() but passes only the sampled documents, so a sampled range facet today reports value=950 next to a range count of 50 (1,000 documents, range [0, 499], sample of 100), even without a fastMatchQuery; with the fix it reports 50, consistent with the sampled count. After amortizeFacetCounts, which divides the total by the sampling rate and caps it at the live documents, that is 1,000 today and 500 with the fix (read, not run). No upstream test combines sampling with range facets, but callers may rely on the raw or the amortized value, so a maintainer may want to decide it explicitly or handle sampling in a separate change. The fix conflicts textually with open PR #16439. Only query-time numbers change, so no reindex is needed.

In RangeFacetCounts.java:140, LongRangeFacetCounts.java:150, DoubleRangeFacetCounts.java:177 and RangeOnRangeFacetCounts.java:100, drop totCount += hits.totalHits(); and add totCount++ as the first statement of the loop; the existing totCount -= missingCount stays. The totCount comment (RangeFacetCounts.java:49, RangeOnRangeFacetCounts.java:45) becomes "Number of checked documents that fall in at least one range." +6/-8 in four files.

// RangeFacetCounts, LongRangeFacetCounts, DoubleRangeFacetCounts, RangeOnRangeFacetCounts
-      totCount += hits.totalHits();
       for (int doc = it.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; ) {
+        totCount++; // count checked documents only: the fastMatchQuery may skip some hits
         ... // unchanged: count the value, or missingCount++
       }
     ...
     totCount -= missingCount; // unchanged

Trade-offs

• One increment per checked document instead of one add per segment; no benchmark was run.
• Changes sampled range totals as described above (a consistency gain, but a visible change that may deserve its own test, CHANGES note or separate change).

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 2 of 2 on stock (assertions on and off) and passes 2 of 2 with the fix; the whole :lucene:facet:test, 40 suites and 308 tests, 0 failures, with assertions on (7 skipped) and off (8 skipped), using the test before one over-long line was wrapped (the same code); :lucene:facet:check -x test (google-java-format, ecj lint and forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test 2 of 2 and the finder's, the reproducer's and the devil's advocate's tests 12 of 12 pass. Earlier: TestRangeFacetCounts and TestRangeOnRangeFacetCounts 42 of 42, and the same change patched into 4.7.0 and 9.5.0 fixes the release probes there.

Count the visited documents and subtract hits.totalHits() - checked after each segment loop.

Trade-offs

Same numbers as counting each checked document, with more code; not run.

Describe FacetResult.value with a fastMatchQuery as an upper bound.

Trade-offs

Not recommended: the bound would depend on the segment layout and on whether the fastMatchQuery has a scorer per segment, which no caller can use.

Drop the subtraction too, so the total is every collected hit.

Trade-offs

Not recommended: it would not depend on the segment layout, but it contradicts the FacetResult.value javadoc and testBasicLong (22 of 101), a larger contract change; not run.

Method: code reading (the hunt's "non-formal" tier), a hunt unit (document and field factories), 2026-10-26.

• The finder was wiring RangeOnRangeFacetCounts into the unit's bounded-exhaustive differential check of BinaryRangeDocValues / QueryType.matches. That harness passes no fastMatchQuery, so it could not see the bug; the finder noticed the bulk add while reading the counting loop, confirmed it with a probe and a public-API JUnit on the stock jars, and found the same pattern in the classic range counters.
• About two hours later a hunt unit re-found the same root cause, by its own record independently, with an SMT model (linear integer arithmetic, Z3) of the counting loops: obligation O7c (a fastMatchQuery that excludes hits makes the single-valued total differ from the documents in a range) is satisfiable, so the model has such a case. Three properties hold in the model, because their negations are unsatisfiable: O7a (without a fastMatchQuery the single-valued total equals the documents in a range), O7b (the multi-valued total equals the documents in a range) and O7d (the overcount is exactly the number of excluded hits). A 24-seed randomized differential against a brute-force oracle agreed. That candidate was withdrawn as a duplicate.
• Then independent AI tools reproduced, reviewed and judged it (see the checks).