Sampled facet counts collapse to 0 when a dimension skips its drill-down terms

When a query matches more documents than sampleSize, RandomSamplingFacetsCollector counts facets on a random sample and amortizeFacetCounts scales each sampled label count back up, sampled / samplingRate, capped at what its javadoc calls "the upper bound for each facet value": the document frequency of the label's drill-down term. That was a true bound when the cap was written in 2014, because every drill-down term was always indexed. Since 8.9.0 (LUCENE-9385) FacetsConfig.setDrillDownTermsIndexing can skip those terms: under NONE every amortized count of the dimension is 0, under FULL_PATH_ONLY or DIMENSION_AND_FULL_PATH every parent label of a hierarchical dimension is 0, and a label that is the full path of some documents and a parent of others can be capped below its true count (in the repro, 500 instead of 1000). The dimension total stays amortized, so one FacetResult reports 1000 next to children of 0, with no error. Taxonomy facets fail this way on every tested release from 8.9.0 to 10.5.2 and on main, sorted-set facets on every tested release from 9.2.0. The method already holds the dimension's DimConfig; choosing the bound from its drillDownTermsIndexing fixes it, and the whole facet test suite passes with that change.

• Facet sampling: RandomSamplingFacetsCollector(sampleSize, seed) (or createManager) keeps about sampleSize of the matching documents when there are more; getSamplingRate() is sampleSize / totalHits. Facet counts computed over it are counts in the sample.
• Amortizing: amortizeFacetCounts(result, config, searcher) turns sampled counts into estimates of the full counts: "you can amortize the sampled counts by calling this method. Uses the FacetsConfig and the IndexSearcher to determine the upper bound for each facet value" (RandomSamplingFacetsCollector.java:215-219).
• Drill-down terms: for each facet path, FacetsConfig.build can add terms such as iMod10/3 to an indexed field ($facets by default) so that DrillDownQuery can filter on a label. The counts themselves come from doc values (taxonomy ordinals or sorted-set values), not from these terms.
• DrillDownTermsIndexing (8.9.0, LUCENE-9385), per dimension: ALL (default: the dimension, every sub-path and the full path), ALL_PATHS_NO_DIM, DIMENSION_AND_FULL_PATH, FULL_PATH_ONLY, and NONE ("Index no drill down terms") (FacetsConfig.java:74-106).

1. amortizeFacetCounts returns its input when totalHits <= sampleSize (RandomSamplingFacetsCollector.java:222-224). Otherwise, for each label it builds the path dim/.../label (:231-238), reads max = reader.docFreq(new Term(dimConfig.indexFieldName, fullPath)) (:239), computes (int) (sampled / samplingRate) (:240) and keeps Math.min(max, estimate) (:241). It never looks at dimConfig.drillDownTermsIndexing, although it holds the DimConfig (:228).
2. FacetsConfig.indexDrillDownTerms (FacetsConfig.java:419-460, called for taxonomy, sorted-set and association fields at :400, :501, :530) adds no term under NONE (:421), only the full path under FULL_PATH_ONLY (:423-427), and the bare dimension plus the full path under DIMENSION_AND_FULL_PATH (:434-438). Only ALL_PATHS_NO_DIM and ALL add every sub-path (:439-452).
3. So the docFreq is 0 for every label under NONE, 0 for every parent label of a hierarchical dimension under the two full-path modes, and only the documents that stop at the label for a mixed-depth label. Each of those stops being an upper bound: a docFreq of 0 always caps a returned label below its real count, and a partial docFreq does so when it is below both the estimate and the label's count in the query (it counts the whole index, the facet count only the query).
4. The dimension total is capped by reader.numDocs() instead (:245-250) and stays amortized, so the result contradicts itself: total 1000, children 0.

The amortized dimension total is 1000 in every row. Not affected: ALL, ALL_PATHS_NO_DIM, and the full-path modes on a flat dimension or at the leaf level, where every document with the label carries its term.

• Who is exposed. Applications that sample facets with RandomSamplingFacetsCollector, call amortizeFacetCounts, and configure the dimension with NONE (the "count but never drill down" case LUCENE-9385 was built for), or with FULL_PATH_ONLY / DIMENSION_AND_FULL_PATH on a hierarchical dimension. Taxonomy facets and sorted-set doc values facets, flat and hierarchical, were run. Association fields are written through the same drill-down helper, but association facets sum values and the method's javadoc covers counting Facets only, so whether amortizing them applies was neither established nor run.
• Default vs opt-in. Two independent opt-ins on one dimension. ALL (the default), ALL_PATHS_NO_DIM and the full-path modes on flat dimensions are unaffected. FacetsConfig is @lucene.experimental.
• Scale. Every query with more hits than sampleSize. Under NONE every returned label comes back 0; under the full-path modes every returned parent label of a hierarchical dimension does, and a mixed-depth label is capped whenever its partial docFreq is below the estimate. 1000 documents are enough; in the layouts tested more documents do not shrink the error (the cap is 0, or a fixed fraction of the true count).
• Silent. No exception or log line, and the dimension total looks right. An application that hides zero-count values shows an empty facet.
• How likely. Rare: a read-only code search found no use of RandomSamplingFacetsCollector or setDrillDownTermsIndexing in Elasticsearch, OpenSearch or Solr (default branches), and nothing has been reported since 8.9.0 (2021). A 2024 java-user@ thread describes applications that sample facets and use the amortized counts; nobody in it mentions DrillDownTermsIndexing.
• Possibly wider, not run. amortizeFacetCounts accepts any FacetResult. Counting implementations whose labels are not written as FacetsConfig drill-down terms (StringValueFacetCounts, LongRangeFacetCounts, DoubleRangeFacetCounts) may find no term to cap with, in any indexing mode. Nothing here executed that, and fix A would not change it (an unconfigured dimension reads the default ALL).
• Not affected. Exact (unsampled) counts; which documents match; drill-down, beyond what the chosen mode already implies (these modes make a DrillDownQuery on an omitted path find nothing, by design); the index; queries with at most sampleSize hits. Nothing is stored wrong, so a fix needs no reindex.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: amortized count of dim/x expected:<2> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.facet.TestAmortizeWithoutDrillDownTerms.testAmortizedCountWithDrillDownTermsIndexingNone(TestAmortizeWithoutDrillDownTerms.java:57)

• Contract. RandomSamplingFacetsCollector.java:215-219: "you can amortize the sampled counts by calling this method. Uses the FacetsConfig and the IndexSearcher to determine the upper bound for each facet value." The class javadoc (:38-40): "since the counts of the top-facets is based on the sampled set, you can amortize the counts". A bound below the number of sampled documents with the label, or below its true count, is not an upper bound.
• No stated precondition. DrillDownTermsIndexing (FacetsConfig.java:74-106) and its setter (:216-226) describe only which terms are indexed. The facet package overview presents drill-down terms as something that "can be used at search time for drill-down", with NONE as a choice, and its sampling section states no dependency (lucene/demo/.../facet/package-info.java:231-243, :377-388). CHANGES.txt and MIGRATE.md say nothing either.
• Tests. TestRandomSamplingFacetsCollector.java:135-149 asserts, in the default configuration, that every amortized count is min(5 * sampled, numDocs / 10.f) within 1.0: the estimate capped at about the label's document count (numDocs / 10.f stands in for it; the slack is there because numDocs need not divide by 10). No upstream test combines sampling with a non-default DrillDownTermsIndexing.
• History. LUCENE-5476 (#6539, 2014) added the cap after a design-review suggestion: "The docFreq of the category drill-down term is an upper bound - and could be used as a limit"; then every prefix of every path was indexed. LUCENE-9385 (#10425, PR #25, 2021) made the terms optional; its discussion wanted "to decouple the idea of indexing data required for facet counting separate from the ability to actually drill down" (@gsmiller), and it never mentions sampling. It changed only FacetsConfig.java and two drill-down tests.
• Self-contradiction. The same FacetResult amortizes the dimension total to 1000 while every child is 0; the exact unsampled counts are 100 per label.
• Precedent. LUCENE-5522 (#6585, 4.7.1), drill-down terms missing for association facets, was fixed as a bug.

The strongest "intended" case: amortized counts are only estimates; FacetsConfig is @lucene.experimental; the javadoc says the method uses the FacetsConfig and the IndexSearcher, so it obviously needs the indexed terms; whoever drops the drill-down terms also drops the statistic the heuristic uses, and under NONE a DrillDownQuery on the label finds 0 documents too, so 0 is "consistent". Two facts strengthen it: the facet package overview describes the usual workflow as finding the top facets on the sample and then computing exact counts for those over all matching documents (lucene/demo/.../facet/package-info.java:383-385), so amortization is a convenience; and in the five years since 8.9.0 nobody has reported this, and no known user combines both opt-ins.

Why it fails: the method estimates the unsampled count (the class javadoc, and the upstream test compares with it), which is 100 per label under NONE, and the same call amortizes the total to 1000 next to children of 0. A bound below the sampled count is not a bound, no document states the precondition, and the method already holds the DimConfig it would need to check. @lucene.experimental allows API change, not silently wrong numbers, and NONE is exactly the "count but never drill down" case LUCENE-9385 was built for. The tests check only bounds, not exact estimates. The exact-recount workflow and the missing reports argue for a low priority or a documentation fix, not for the numbers being right. Verdict: a defect (adjudication: confirmed, high; the independent reproduction, the cross-model review and the intended-behaviour check real; the intended case rated moderate by the cross-model reviews, moderate-to-weak by the independent reproduction, weak by the intended-behaviour check). A maintainer may still prefer a documentation fix; it is offered below.

The default and the complete modes keep exactly today's docFreq cap; in the other modes the plain estimate sampled / samplingRate is kept, bounded by totalHits. That bound can bind: the 2014 design review argued the estimate essentially cannot exceed totalHits, but LFV-437 shows amortized counts up to twice too high (1999 hits, sampleSize 1000, every hit sampled, dimension total 3996). The switch is exhaustive, so a new DrillDownTermsIndexing value will not compile until someone picks its bound. The javadoc names the bound. Workaround until a fix ships: compute sampled / getSamplingRate() yourself for such dimensions. Open draft PR #16847 rewrites the same lines (long doc counts), so whichever lands second needs a trivial rebase. Only query-time numbers change; no reindex.

RandomSamplingFacetsCollector.java:228 and :239: decide once per dimension whether the indexing mode guarantees that every document with a label carries the label's term, and use the docFreq only then; otherwise use totalHits. +17/-2 lines, javadoc included (the finder's sketch, with the javadoc added). This is version 2: after the fact checks the javadoc and the comment name the modes; the code is the same as in version 1, which the checks validated.

    DimConfig dimConfig = config.getDimConfig(res.dim);
+    // the docFreq of a label's drill-down term bounds the label's count only if the indexing mode
+    // guarantees that every document with the label indexes that term
+    final boolean drillDownTermForEveryDoc =
+        switch (dimConfig.drillDownTermsIndexing) {
+          case ALL, ALL_PATHS_NO_DIM -> true;
+          case FULL_PATH_ONLY, DIMENSION_AND_FULL_PATH -> dimConfig.hierarchical == false;
+          case NONE -> false;
+        };
     ...
-      int max = reader.docFreq(new Term(dimConfig.indexFieldName, fullPath));
+      int max =
+          drillDownTermForEveryDoc
+              ? reader.docFreq(new Term(dimConfig.indexFieldName, fullPath))
+              : totalHits;

Trade-offs

• In the other modes the estimate is no longer capped by the label's own document count, so sampling noise can push it above the true count, as for any uncapped estimate.
• Conservative for hierarchical dimensions: a full-path mode guarantees the label's term at no level (any document may have a longer path through the label), so the fix uses totalHits at every level, including leaf labels where today's docFreq cap happens to be right. There it is looser than stock: leaf a/x/2 under hierarchical FULL_PATH_ONLY (sampled 13, true count 100) gives 100 on main and 130 with the fix (diagram probe).
• A path-aware bound (min(totalHits, sum of the docFreqs of all full-path terms under the label)) would cost a terms-dictionary scan per label, can still exceed the true count on a multi-valued hierarchical dimension (one document can carry several paths under the label), and gives nothing under NONE; not proposed.
• The switch expression needs Java 14 or later. Upstream's maintained branches (main, and branch_10x, which requires Java 21) compile it; there is no maintained 8.x or 9.x branch, whose older Java levels would need a classic switch.
• Not covered by any test here: multi-valued hierarchical dimensions, and a query narrower than all documents, where documents outside the query raise the index-wide docFreq.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 5 of 6 on stock with assertions on and off; with the fix, the new test and upstream TestRandomSamplingFacetsCollector pass 7 of 7, on and off, and TestRandomSamplingFacetsCollector passes 20 of 20 iterations; the whole :lucene:facet:test passes, 40 suites and 312 tests, 0 failures, with assertions on (7 skipped) and off (8 skipped); :lucene:facet:check -x test (google-java-format, ecj lint, forbidden APIs) and :lucene:facet:renderJavadoc pass.
• Standalone runner with the fix shadowed: the new test 6 of 6 (-ea and -da), the finder's test 4 of 4, the independent reproducer's test 9 of 9 (5 of 9 fail stock), upstream TestRandomSamplingFacetsCollector 10 of 10 iterations.
• Version 2 (javadoc and comment only), re-validated after the fact checks: applyGoogleJavaFormat leaves it unchanged; :lucene:facet:check -x test and :lucene:facet:renderJavadoc pass; the new test and upstream TestRandomSamplingFacetsCollector 7 of 7 with assertions on and off; the whole :lucene:facet:test, 40 suites and 312 tests, 0 failures, on (7 skipped) and off (8 skipped); standalone: the new test 6 of 6 (-ea, -da), the finder's 4 of 4, the independent reproducer's 9 of 9, upstream 10 of 10 iterations.

Say in the amortizeFacetCounts javadoc and on DrillDownTermsIndexing that the upper bound is the docFreq of each value's drill-down term, so amortization needs ALL, ALL_PATHS_NO_DIM or a flat dimension, and that other modes cap the counts too low, down to 0.

Trade-offs

No behaviour change and the smallest diff, but it leaves a silent 0 for a configuration Lucene accepts; callers must compute the estimate themselves. Not validated (nothing to run).

Throw IllegalArgumentException from amortizeFacetCounts when the dimension's drill-down terms are incomplete.

Trade-offs

Turns a silent wrong number into an error, but breaks existing callers of these modes: a behaviour change for a minor release. Not validated.

Cap every label at totalHits, in every mode. Run only to show why the cap must stay for the default.

Trade-offs

Fixes the new test (6 of 6), but upstream TestRandomSamplingFacetsCollector fails 10 of 10 iterations (for example an amortized count of 1220 against 1172.7, the test's numDocs / 10.f stand-in for the label's 1172 or 1173 documents): in the default configuration the estimate then exceeds the label's true count, which the docFreq cap prevents.

What the checks validated

Executed with the standalone runner on main a9779ef3c193, as a negative control.

Method: code reading (the hunt's "non-formal" tier), a hunt unit ("every count or statistic a module reports equals a brute-force count over the live documents that pass its filters"), 2026-10-26, stock main jars.

• While building the inventory of every place lucene/facet produces a reported count, the hunting AI tool saw that amortizeFacetCounts caps each estimate with reader.docFreq of a drill-down term, and that LUCENE-9385 had made those terms optional per dimension.
• It wrote a public-API JUnit over the DrillDownTermsIndexing values for a flat and a hierarchical dimension (1000 documents, a 10% sample). A read-only review by another model family (OpenAI GPT-5.6, refute lens) pointed out the wider mixed-depth case, which the finder then added and ran (500 instead of 1000).
• Then independent AI tools reproduced, reviewed and judged it (see the checks), and this doc's test added the hierarchical sorted-set case, which had only been read before.