Taxonomy facets: a hierarchical sub-path's total leaves out the documents labeled at that sub-path

For a dimension configured setHierarchical(dim, true) and left single-valued, taxonomy faceting reports a sub-path's FacetResult.value (from getTopChildren(n, dim, path...) and getAllChildren(dim, path...)) as the sum of its immediate children's values. Documents labeled at the sub-path itself are not children, so they are left out. With one document at Publish Date/2010 and one at Publish Date/2010/10 the value of Publish Date/2010 is 1, while getSpecificValue on the same Facets object and a drill-down on the path give 2, and SortedSetDocValues faceting with the same config gives 2. The right number is already computed: the search-time roll-up makes each node count the documents at and below it, and getSpecificValue returns that; only the multiValued + requireDimCount branch of the two methods uses it. FastTaxonomyFacetCounts (dense and sparse counting) and TaxonomyFacetFloatAssociations with a DoubleValuesSource are affected. Child counts, the dimension-level value, getSpecificValue and drill-down are right (on the value-source path, LabelAndValue.count has a separate defect). Every lucene-facet release from 4.7.0 (2014) to 10.6.0 (10.6.0 by reading) and main gives the same numbers; until 8.8.2 the javadoc described the child sum, since 8.9.0 it promises a document count. Returning the rolled-up value fixes it, and the whole lucene/facet test suite passes with that change.

• Hierarchical dimension: FacetsConfig.setHierarchical(dim, true) allows labels with more than one component, such as Publish Date/2010/10. A document may stop at any depth: nothing requires leaf or equal-depth labels.
• Single-valued (the default): each document has at most one label in the dimension. FacetsConfig.build then indexes only the label's own ordinal, not its ancestors (FacetsConfig.java:386-398: ancestors only when multi-valued).
• Roll-up: after counting, rollup() adds every node's descendants into it for each single-valued hierarchical dimension (TaxonomyFacets.java:262-299). getSpecificValue returns that rolled-up number (:537-553).
• FacetResult.value: "Total number of documents containing a value for this path ... counted once" (FacetResult.java:30-35), returned by getTopChildren(n, dim, path...) and getAllChildren(dim, path...) next to the child labels.

1. getAllChildren (TaxonomyFacets.java:342-415) and getTopChildrenForPath (:467-516, used by getTopChildren) walk the path's immediate children with a count above 0 and aggregate their values: dense branch :373-386 and :489-503, sparse hash-map branch :360-371 and :478-488.
2. Only for multi-valued dims do they replace that sum: with the path's own value for requireDimCount (:393-395, :506-508), or with -1 without it. The single-valued branch keeps the sum: "Our aggregateddim value is accurate, so we keep it" (:401).
3. That holds at the dimension level, where every label has at least one component below the dimension. For a sub-path it fails as soon as some matching documents are labeled at the sub-path itself: for counts the value is low by exactly the number of those documents (with a value source, their values are left out of the aggregate), although the path's own rolled-up value is already correct.
4. SortedSetDocValues faceting returns the path's own count for hierarchical dims ("hierarchical dims index all path counts directly", AbstractSortedSetDocValueFacetCounts.java:341-345).

Documents A at Publish Date/2010 and B at Publish Date/2010/10; only setHierarchical("Publish Date", true); MatchAllDocsQuery.

The same numbers on main with assertions on and off, on 10.5.2, 9.12.3 and 9.3.0, in the dense and the sparse counting branch. With a DoubleValuesSource and SUM (A worth 10, B worth 1): sub-path value 1.0, getSpecificValue 11.0.

• Who is exposed. Applications that use FastTaxonomyFacetCounts (both its dense and its sparse counting branch) or TaxonomyFacetFloatAssociations with a DoubleValuesSource, on a hierarchical dimension left single-valued, with mixed-depth labels (a product filed directly under a parent category next to products in sub-categories, a date known only to the year next to full dates), and that show FacetResult.value of a sub-path from getTopChildren(n, dim, path...) or getAllChildren(dim, path...).
• Default vs opt-in. setHierarchical is opt-in configuration (needed for multi-component labels); single-valued is the default. Everything else is default.
• Scale. Any index size; deterministic, no concurrency. For counts the value is low by exactly the number of matching documents labeled at the requested path. With a DoubleValuesSource and SUM it is low by the sum of their values (1.0 instead of 11.0); MAX has no such formula and was not run.
• Silent. No exception or log line, and the child counts look right. Nothing is stored wrong, so a fix needs no reindex.
• Not affected. Child labels and counts, the dimension-level value, getSpecificValue, drill-down hit counts, SortedSetDocValues faceting, multi-valued + requireDimCount dims, and data whose labels all have the same depth (the demo's full dates). On the value-source path, LabelAndValue.count has its own separate defect. A sub-path with no child that has a count is a different case, which the fix does not change: getTopChildren returns null in both taxonomy and SortedSetDocValues faceting, and taxonomy getAllChildren returns null too, while SortedSetDocValues getAllChildren returns the path count with no labels (executed).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<2> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.facet.taxonomy.TestHierarchicalSubPathValue.testSubPathValueCountsDocsLabeledAtThePath(TestHierarchicalSubPathValue.java:56)

• Contract. FacetResult.java:30-35: "Total number of documents containing a value for this path, even those not included in the topN. If a document contains multiple values for the same path, it will only be counted once in this value." The demo user guide repeats it (demo/facet/package-info.java:217-219). setHierarchical only says "has depth > 1 paths" (FacetsConfig.java:170), FacetField needs only one path component (FacetField.java:53-54), and FacetsConfig.build accepts mixed depths (it rejects only a second label in a single-valued dimension: FacetsConfig.java:275-277 calls checkSeen, :233-241). The guide's "path from root to a leaf for the current document" (package-info.java:131) describes one document's label; nothing enforces that it is a leaf of the taxonomy, and the same guide says the hierarchical category list "allows counting any sub-category in the taxonomy, and not just leaf nodes" (:251-255).
• Intent. The current wording came from LUCENE-9953 (#10992, 2021) and the dev@ thread "FacetResult#value semantics?", where mikemccand wrote "we really should strive to count each hit only once" and "the goal here is to say 'if the user were to drill down on this value, how many total hits would THAT query return?'". The drill-down here returns 2.
• Tests. The @Nightly TestSortedSetDocValuesFacets#testRandomHierarchicalFlatMix (LUCENE-10250) indexes single-valued hierarchical dims with random path lengths and expects the inclusive sub-path value; switching SortedSetDocValues to the child sum makes it fail (executed mutation check). No taxonomy test covers mixed-depth labels; TestSortedSetDocValuesFacets#testBasicHierarchical does not test this: its hierarchical dimension c is multi-valued, and its c/buzz documents also have children, so it gives 2 under either reading.
• Consistency. One Facets object gives 2 from getSpecificValue and 1 from FacetResult.value; the same documents give 2 when the dimension is multi-valued + requireDimCount, and 2 with SortedSetDocValues faceting.
• Value sources. "Number of documents" cannot literally describe a weighted 11.0. For value sources the argument is consistency: getSpecificValue and the path's rolled-up value give 11.0, the sub-path value 1.0. Only SUM was executed.
• History. The pre-4.7 facet API returned 2 for the same data (executed on 4.6.1).

The strongest "intended" case: the child sum is a commented choice from LUCENE-5339 (2013); from 4.7.0 to 8.8.2 the javadoc said "Total value for this path (sum of all child counts, or sum of all child values)", exactly what the code computes; the 2021 rewording was written for another class (LongValueFacetCounts) and may have overreached; later refactors (LUCENE-10550, #12966) kept the behaviour; getSpecificValue already gives the inclusive number; and the facet API is @lucene.experimental.

Why it fails: the current javadoc and user guide promise a document count with no exception, and the rewording came with a stated drill-down intent that this result breaks; the comment's reasoning is only valid at the dimension level, where it was written; SortedSetDocValues faceting implements and tests the inclusive count for exactly this data shape; one object returning 2 and 1 for the same path is not a coherent semantics; and reading the guide's "path from root to a leaf" as a leaf-only rule fails too, since nothing enforces it and the same guide promises counts for "any sub-category ... not just leaf nodes". @lucene.experimental is about API stability, not wrong numbers. Verdict: a defect (adjudication: confirmed, high). On code fix versus javadoc fix the case is moderate, not decisive: the child sum was deliberate, documented for seven years and kept through two refactors. A maintainer may still prefer to document the child sum.

The dimension level is unchanged, and so are child counts. The roll-up the fix relies on always runs for these dimensions: every taxonomy class that can see a hierarchical dimension calls it, and association fields cannot be hierarchical. Only a query-time number changes, so no reindex is needed. It is still a compatibility change: the edited code is in package-private base classes, but the value is exposed through the public, non-final FastTaxonomyFacetCounts and TaxonomyFacetFloatAssociations (also in DrillSideways results), and it has been the child sum since 2014, so it needs a CHANGES entry. If the MAX-starts-at-zero association fix is applied too, the two patches touch neighbouring lines of getAllChildren.

Two hunks in TaxonomyFacets.java, after the multi-valued branch of getAllChildren (:393-402) and of getTopChildrenForPath (:505-513). In getAllChildren the variable dimOrd holds the ordinal of the requested path (:344-345); getTopDims calls getTopChildrenForPath with a dimension ordinal, whose parent is the root, so the dimension level keeps today's sum.

@@ -397,6 +397,10 @@ abstract class TaxonomyFacets extends Facets {
         // Our aggregated value is not correct, in general:
         aggregatedValue = missingAggregationValue();
       }
+    } else if (dimConfig.hierarchical && path.length > 0) {
+      // A sub-path of a single-valued hierarchical dim: rollup() already aggregated the documents
+      // labeled with the path itself and those below it into the path's own ordinal
+      aggregatedValue = getAggregationValue(dimOrd);
     } else {
       // Our aggregateddim value is accurate, so we keep it
     }
@@ -510,6 +514,10 @@ abstract class TaxonomyFacets extends Facets {
         // Our aggregated value is not correct, in general:
         aggregatedValueNumber = missingAggregationValue();
       }
+    } else if (dimConfig.hierarchical && parents.get(pathOrd) != TaxonomyReader.ROOT_ORDINAL) {
+      // A sub-path of a single-valued hierarchical dim: rollup() already aggregated the documents
+      // labeled with the path itself and those below it into the path's own ordinal
+      aggregatedValueNumber = getAggregationValue(pathOrd);
     }
 
     return new TopChildrenForPath(aggregatedValueNumber, childCount, q);

Trade-offs

• Changes a number that has been the child sum since 4.7.0; needs a CHANGES entry. One array or hash lookup per call.
• For value sources it returns the rolled-up aggregate, consistent with getSpecificValue: SUM 11.0 in the reproducer's case; for MAX the maximum over the path and its descendants, subject to the separate MAX-starts-at-zero defect (LFV-547: with only negative values both today's and the fixed value are 0). MAX was not executed.
• Relies on rollup() having run, which holds for every caller in the module: FastTaxonomyFacetCounts in both count paths and TaxonomyFacetFloatAssociations with a value source; association fields cannot be hierarchical (FacetsConfig.java:301-303).
• A compatibility change: TaxonomyFacets and its two base subclasses are package-private, but the value is exposed through the public, non-final FastTaxonomyFacetCounts and TaxonomyFacetFloatAssociations, so callers that read today's sub-path value see it change.
• It also changes FacetResult.value in DrillSideways results, which count with the same FastTaxonomyFacetCounts (DrillSideways.java:159, :165). TestDrillSideways and TestParallelDrillSideways pass with the fix (14 + 14 tests, assertions on and off), but none of their documents is labeled at an inner node that also has labeled children, so no DrillSideways test covers the mixed-depth case.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 1 of 3 on stock (assertions on and off) and passes 3 of 3 with the fix; the whole :lucene:facet:test, 40 suites and 309 tests, 0 failures, with assertions on (7 skipped) and off (8 skipped); :lucene:demo:test, 23 tests, 0 failures; :lucene:facet:check -x test (google-java-format, ecj lint and forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test 3 of 3 with assertions on and off; the reproducer's and the reviewer's tests 7 of 7 (dense, sparse and value-source cases); the probe prints value=2 for both sub-path calls. Earlier, a reviewer ran the same logic against four taxonomy test classes (four separate runs, 19 + 4 + 10 + 12 = 45 tests, 0 failures).

Drop the sub-path condition (else if (dimConfig.hierarchical)).

Trade-offs

Simpler; at the dimension level the rolled-up value should equal the child sum, because no document can be labeled at the dimension alone. It also touches the dimension-level path used by getTopDims; not run. The equality holds for counts and for the built-in SUM and MAX; AssociationAggregationFunction is a public extension point, and a custom function for which 0 is not an identity, or that is not associative and commutative, could change the dimension value.

Say in FacetResult.value (or the taxonomy classes) that for single-valued hierarchical taxonomy dims a sub-path's value is the sum of its children.

Trade-offs

Not recommended: keeps taxonomy and SortedSetDocValues faceting disagreeing for the same config, contradicts the drill-down intent behind the 2021 wording, and leaves getSpecificValue and FacetResult.value disagreeing on one object.

Method: bounded-exhaustive translation validation (the hunt's "semi-formal" tier), a hunt unit (FacetsConfig and drill-down indexing matrix), 2026-10-27.

• The harness ran the stock jar against a reference written from the javadoc (the value of a path is the number of matching documents with a label at or below it) over DrillDownTermsIndexing x hierarchical x multiValued x requireDimCount x index-field layout x config style x {taxonomy, SortedSetDocValues} faceting, every multiset of up to 3 small documents. In the run below each case was indexed in one of four layouts (1 or 2 segments, with or without deletes), taken in rotation; single-document cases use one segment without deletes.
• In the maxDocs=3 run with assertions off (685,912,348 checks over 400 configs) the mismatch class "path value is the children's sum, not the path count" appeared 99,920 times, in exactly the 50 taxonomy configs with hierarchical and single-valued settings (requireDimCount on or off, every drill-down indexing mode and field layout) and in no SortedSetDocValues or multi-valued config.
• The finder reduced it to the two-document public-API JUnit; then independent AI tools reproduced, reviewed and judged it (see the checks).