Float association facets with a value source over-count labels that have hits below them

TaxonomyFacetFloatAssociations with a DoubleValuesSource aggregates a per-document value per taxonomy category, and since 9.11.0 each LabelAndValue also carries count, the "Number of occurrences for this label". On a dimension declared hierarchical (and single-valued, the default), every label that has matching documents below it gets a count that is too high: two documents under d/a give a (9.0) with count 4, and one document at d/a/b/x gives 3 at a and at a/b, where FastTaxonomyFacetCounts gives 2 and 1. FloatTaxonomyFacets#updateValueFromRollup calls super, which rolls up the counts of the whole sub-tree, and then its own recursion, which calls the same method, and so super, again for every descendant. This defect leaves values, value order, childCount and leaf counts right, short of an int overflow at extreme depths. Upstream testRollupValues runs this path but compares only toString(), which omits the count. Every release from 9.11.0 to 10.5.2 and main is affected; a one-line change that makes the private recursion roll up values only fixes it, and the whole lucene/facet test suite passes with it.

• Taxonomy facets: each document gets category paths such as d/a/x (FacetField); a side index, the taxonomy, gives every path prefix an ordinal: d, d/a, d/a/x.
• Hierarchical dimension: FacetsConfig.setHierarchical("d", true), "Pass true if this dimension is hierarchical (has depth > 1 paths)" (FacetsConfig.java:170). If it is also single-valued (the default), only the leaf ordinal is indexed per document; the counts and values of d/a are computed at search time by a roll-up from the leaves (FacetsConfig.java:380-398; TaxonomyFacets.java:261-285).
• Value-source association facets: TaxonomyFacetFloatAssociations(taxoReader, config, fc, SUM or MAX, valuesSource) aggregates valuesSource per category. The code says "Hierarchical dimensions are supported when using a value source, so we need to rollup" (TaxonomyFacetFloatAssociations.java:164-165).
• LabelAndValue.count: "Number of occurrences for this label" (LabelAndValue.java:27-28), added by #13414 for 9.11.0 ("Counts are always available in the result when using taxonomy facets", CHANGES.txt:2784).

1. Each hit adds its value and count + 1 at its indexed ordinal, the leaf (TaxonomyFacetFloatAssociations.java:149-161, :158). For d/a/x and d/a/y: x = 1, y = 1, a = 0, d = 0.
2. rollup() calls the virtual updateValueFromRollup(d, a) (TaxonomyFacets.java:281). The float override first calls super.updateValueFromRollup (FloatTaxonomyFacets.java:96), which rolls up the counts of the whole sub-tree in place (TaxonomyFacets.java:242-244, :287-299): a = 2, d = 2. Already right.
3. The override then calls its own rollup(a) for the values (FloatTaxonomyFacets.java:98, :112-122). That loop calls the virtual updateValueFromRollup(a, firstChild) (:117), so the super count pass runs again under a and adds x + y once more: a = 4. Every node with children repeats this, and each extra pass rewrites the counts below it in place, so the extra counts compound with depth: one document at depth D is counted C(D, k) times at level k. A label whose hits all sit at the label itself keeps its right count, because the extra passes add only zeros.
4. Values are aggregated once, by the float recursion, and stay right. getTopChildren and getAllChildren put getCount(ord) into each LabelAndValue (TaxonomyFacets.java:325, :333-334, :410-412).

SUM and MAX give the same counts, and FastTaxonomyFacetCounts on the same hits gives the expected ones. The first two rows were run on main with assertions on and off and on 10.5.2, 10.0.0, 9.12.3 and 9.11.0; the third and fourth on main with assertions off (the fourth with SUM only); the last on main.

• Who is exposed. Applications that aggregate a per-document value per category with TaxonomyFacetFloatAssociations and a DoubleValuesSource (revenue per category, or score-weighted facets with DoubleValuesSource.SCORES), over a dimension declared setHierarchical(dim, true) and not multi-valued, and that read LabelAndValue.count, for example to show "Fiction (2 books, 9.00)". On 9.11.0-9.12.x the deprecated TaxonomyFacetSumValueSource (a subclass; count 4, value 9.0) and the deprecated TaxonomyFacetCounts(OrdinalsReader, ...) (count 4, value 2) are hit too; both were removed in 10.0.0.
• Default vs opt-in. setHierarchical and the value-source constructor are opt-in, and the class is @lucene.experimental. Single-valued is the default for a dimension. The defect is in the roll-up after collection, which reads only the per-ordinal counts; the tests ran under Lucene's randomized codec, IndexWriterConfig and searcher settings and failed with every seed tried, in both the dense and the sparse (hash-table) counter branches.
• Scale. A label's count is inflated whenever the request's hits include documents below it; labels whose hits all sit at the label itself, and requests with no hits, are unaffected. Nothing in the roll-up depends on index size. The overcount compounds with depth: with one document at each level of a 20-level path the largest count, 352,716, is at levels 9 and 10, while the deepest labels are near-exact (21 and 1).
• Silent. No exception and no log line, and the values are right, which makes it easy to miss.
• Extreme inputs. The same double walk visits each sub-tree once per ancestor, so the roll-up costs about the sum of node depths on every request with a hit: one 4,095-level path (Lucene's 8,191-character limit) takes 86.9 ms per request against 0.81 ms with the fix, and 5 paths of depth 4,000 take 412 ms against 3.2 ms (medians of 5 after 2 warm-ups on one shared host, -da; only the ratios matter). The walk covers the dimension's whole sub-tree in the taxonomy, so deep paths indexed by other documents raise every request's cost. At 33 levels with one document per level some counts wrap past 2^31, and those labels vanish from the results although they have hits, so result membership and childCount are wrong too. The fix removes both.
• Not affected (short of the int overflow): values and value-based ranking; childCount; leaf counts; the dimension's own value and its internal count (FacetResult carries no count for the dimension itself); FastTaxonomyFacetCounts; indexed FloatAssociationFacetField and IntAssociationFacetField (they cannot be hierarchical); multi-valued hierarchical dimensions (not rolled up); flat dimensions. Nothing is stored wrong, so a fix needs no reindex. "Values are right" means this defect does not change them; a separate all-negative MAX bug in the same roll-up code does.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: count of d/a expected:<1> but was:<2>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.facet.taxonomy.TestFloatValueSourceRollupCounts.testRolledUpCountOfIntermediateLabel(TestFloatValueSourceRollupCounts.java:56)

• Contract. LabelAndValue.java:27-28: count is the "Number of occurrences for this label". CHANGES.txt:2784 (9.11.0): "GITHUB#13414: Counts are always available in the result when using taxonomy facets." TaxonomyFacetFloatAssociations.java:42-44: a DoubleValuesSource "can dynamically supply a weight from each doc"; :164: "Hierarchical dimensions are supported when using a value source, so we need to rollup". No reading of "number of occurrences" gives 4 for two documents, or 3 for one.
• Consistency. FastTaxonomyFacetCounts on the same hits gives 2; so do the same paths on a dimension that is also multi-valued (ancestors indexed, no roll-up), with the same value 9.0. The count should not depend on an indexing detail or on the facet class.
• Tests. TestTaxonomyFacetValueSource#testRollupValues (:437-485) uses this exact setup (hierarchical dimension, value-source constructor, SUM and MAX) but compares only FacetResult.toString() (:467-469, :479-481); LabelAndValue.toString() prints label and value (LabelAndValue.java:44-47), and equals/hashCode ignore count (:49-60). With a count assertion added, it fails on main: count of 1 (6.0) expected:<2> but was:<4>. TestTaxonomyFacetAssociations#testAggregationCounts (:573-611) asserts counts that mean documents, on a flat dimension only.
• History. #12966 ("always do counts", 2024-04) introduced three pieces together: the base TaxonomyFacets#updateValueFromRollup (counts of the whole sub-tree), the FloatTaxonomyFacets override that calls super, and a private recursion that calls the overridable method again for every child. Before it, the private float roll-up aggregated values directly, the shape the recommended fix restores. In its 41 comments none notices that the counts are rolled up more than once (the one roll-up thread is about sharing updateValueFromRollup between the Int and Float classes). Until #13414 exposed LabelAndValue.count (2024-05) the counts were only checked for > 0, which inflation cannot change short of the int overflow at extreme depths.
• Precedent. #14008, wrong association-facet counts from a different cause, was fixed as a bug in 9.12.1 and 10.1.0 (CHANGES.txt:2141, :2510). This bug reproduces after that fix.

The strongest "intended" case: the classes are @lucene.experimental; LabelAndValue.equals and toString leave count out, so it could look secondary; the base javadoc says "Overrides should probably call this to update the counts" (TaxonomyFacets.java:237-241) and the override does; LUCENE-5536 (#6599, 2014) said association facets should not roll up; and #11282 (counts from association facets) is still open.

Why it fails: the value-source path explicitly supports hierarchical dimensions, and LUCENE-5536 was about indexed association values, before this path and before counts existed. CHANGES promises counts, LabelAndValue itself is not experimental, and @lucene.experimental is about API stability, not wrong numbers. Leaving count out of equals (for backward compatibility, as #13414 says) and of toString explains why tests missed it, not what the number should be. Calling super is exactly what nests the two recursions; nothing documents a depth-dependent count. Verdict: a defect (adjudication: confirmed, high; every verifier rated the intended case weak; the only "not real" vote was a read-only review that could not run Java). It lowers the severity: only the count is wrong.

Counts are then rolled up once (by the base class) and values once (by the private recursion), each walk visiting every node once, so the roll-up is linear again and the extreme-depth cost and overflow disappear (executed: 0.81 ms instead of 86.9 ms for one 4,095-level path; exact counts at 33 and 40 levels). Values are computed exactly as before. The IntTaxonomyFacets half changes nothing on main, where no class rolls it up, but keeps the two copies the same; a 9.x backport needs it for the deprecated TaxonomyFacetCounts. Add count assertions to testRollupValues either way. Workaround until a fix ships: take the counts from FastTaxonomyFacetCounts over the same FacetsCollector; this avoids the wrong counts, not the roll-up cost, which runs in the association facets' constructor. Only query-time numbers change, so no reindex is needed.

FloatTaxonomyFacets.java:117 and IntTaxonomyFacets.java:108: replace the call to the virtual updateValueFromRollup(ord, children.get(ord)) with the value aggregation it performed; the super call in updateValueFromRollup keeps rolling up the counts of the sub-tree once, as the base javadoc asks. +2/-1 in each file.

  private float rollup(int ord) throws IOException {
    ...
    while (ord != TaxonomyReader.INVALID_ORDINAL) {
-      updateValueFromRollup(ord, children.get(ord));
+      // Values only: super.updateValueFromRollup already rolled up the counts of this sub-tree.
+      setValue(ord, aggregationFunction.aggregate(getValue(ord), rollup(children.get(ord))));
       aggregatedValue = aggregationFunction.aggregate(aggregatedValue, getValue(ord));
       ord = siblings.get(ord);
     }
// the same line in IntTaxonomyFacets.rollup(int)

Trade-offs

• The private recursion no longer goes through the overridable hook; only the dimension root does. Both classes are package-private and their in-tree subclasses do not override updateValueFromRollup, so no in-tree code depends on it. The public subclasses are not final and inherit the protected hook: an application subclass that overrides it would see it called only for dimension roots (the classes are @lucene.experimental).
• The IntTaxonomyFacets change is latent on main; drop it for the smallest diff.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 2 of 2 on stock (assertions on and off) and passes 2 of 2 with the fix; testRollupValues with count assertions fails on stock and passes with the fix (with the rest of TestTaxonomyFacetValueSource, 12 of 12, on and off); the whole :lucene:facet:test, 40 suites and 308 tests, 0 failures, with assertions on (7 skipped) and off (8 skipped); :lucene:facet:check -x test (google-java-format, ecj lint and forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test passes with -ea and -da; the cost and overflow probe gives exact counts at every depth and 0.81 ms for the 4,095-level path; on 9.12.3 the same line in its two classes makes this test and both deprecated-class probes pass (4 of 4; 4 of 4 fail stock). An independent reproducer had checked the float half with its own 4 tests.

The finder's sketch: after rollup(childOrdinal) has finished every child sub-tree, add the children's rolled-up counts to ordinal (+9/-1 in FloatTaxonomyFacets.java).

Trade-offs

Same results for the in-tree classes, also linear; it keeps the per-node calls to the protected hook. It goes against the base javadoc ("Overrides should probably call this to update the counts") and duplicates the count logic in the float class; IntTaxonomyFacets would need the same.

What the checks validated

The new test passes (standalone); the whole :lucene:facet:test, 40 suites and 308 tests, passes with assertions on; the finder's harness count mismatches go from 2,977 to 0 with it; 75 upstream facet tests passed with the unit's four fix sketches applied together. (The gradle log label fixA means this alternative fix.)

Applications can read counts from FastTaxonomyFacetCounts over the same FacetsCollector and values from the association facets.

Trade-offs

A second facet pass per request; it avoids the wrong counts but not the roll-up cost; not a fix.

Method: randomized model-based metamorphic and differential testing (the hunt's "non-formal" tier; the finder called it semi-formal), a hunt unit (facet API metamorphic relations), 2026-10-26, stock main jars.

• The finder wrote a Facets API specification and a harness that builds random facet schemas and tiny indexes and checks metamorphic relations. Relation F1d, "LabelAndValue.count equals the number of matching documents under the label, cross-checked with FastTaxonomyFacetCounts", failed on the value-source float association path: 1,490 SUM and 1,487 MAX mismatches in 400 iterations (seed 2, non-negative values), none with the fix.
• A bounded-exhaustive check then ran every sequence of 1-3 documents over the 14 paths of depth 1-3 of one hierarchical dimension (438,872 checks): count mismatches only at labels with children, ratios such as 2/1, 4/2 and 9/3; 0 with the fix (the remaining mismatches are a separate MAX bug).
• The finder reduced it to a public-API JUnit test with two documents. Then independent AI tools reproduced, reviewed and judged it (see the checks).