Taxonomy facets never roll up a hierarchical dimension that gets its config from getDefaultDimConfig

FacetsConfig#getDefaultDimConfig() is a protected hook (LUCENE-5479, 4.8.0) that sets the config of every dimension without an explicit entry; Lucene's only test of it makes those dimensions hierarchical. Indexing follows the default: for a hierarchical, single-valued dimension it stores only each document's leaf category and leaves the parent counts to a search-time roll-up. But TaxonomyFacets.rollup() picks the dimensions to roll up from config.getDimConfigs(), which holds only the explicit entries, so such a dimension is never rolled up: its dimension count and every non-leaf count stay 0, getTopChildren(dim) returns null (or only the depth-1 leaves), and nothing throws, while a DrillDownQuery on the same labels still matches the documents. It affects FastTaxonomyFacetCounts (both constructors) and the value-source aggregation of TaxonomyFacetFloatAssociations, in every release from 4.8.0 through 10.6.0 and on main. The index itself is correct, so a fix in the roll-up needs no reindex.

1. The hook. FacetsConfig.getDefaultDimConfig() (FacetsConfig.java:142-151, protected) supplies the config of every dimension without an explicit entry: getDimConfig(dim) returns the map entry if there is one, else getDefaultDimConfig() (:153-160).
2. Indexing honours it. processFacetFields reads getDimConfig(facetField.dim) (:370); a 3-component path is accepted only because the default says hierarchical (:371-378). It stores the leaf ordinal (:383-384) and adds the ancestor ordinals only when the dimension is multiValued (:386-398). So each document stores one ordinal: 2010/10/15, 2010/10/20 or 2012/1/1.
3. Counting. FastTaxonomyFacetCounts counts each stored ordinal once (FastTaxonomyFacetCounts.java:93, :138), then calls rollup() (:117 for hits, :174 for count-all). The value-source path of TaxonomyFacetFloatAssociations calls the same rollup() (TaxonomyFacetFloatAssociations.java:164-165); its indexed-association constructors, and those of TaxonomyFacetIntAssociations, never call rollup() for any config and are not affected (code reading). DrillSideways builds its taxonomy facets with FastTaxonomyFacetCounts (DrillSideways.java:159, :165).
4. The roll-up skips the dimension. TaxonomyFacets.rollup() ("Rolls up any single-valued hierarchical dimensions", TaxonomyFacets.java:261-285) loops over config.getDimConfigs().entrySet() (:269). getDimConfigs() returns the private map of explicit entries (FacetsConfig.java:228-231). Publish Date has no entry, so the leaf counts (1 each) are never added up: 2010/10, 2010, 2012/1, 2012 and the dimension stay 0.
5. What the caller sees. getTopChildren(10, "Publish Date") collects the dimension's children with a count above 0 (TaxonomyFacets.java:478-503), finds none, and returns null for 0 children (:308-310). getSpecificValue("Publish Date") is accepted with the comment "ok: rolled up at search time" (:540-541) and returns the raw count, 0 (:553). getAllDims drops a dimension whose getTopChildren is null (:571-575). getTopDims sums a single-valued dimension's direct children (:639-650), which were never rolled up, and leaves out a dimension whose sum is 0 (:652): it returns [] where the explicit config gives value=3 2010 (2) 2012 (1) (executed on 5adff78bb168; correct with the fix).
6. Everything else asks getDimConfig. verifyDim (:221-235), getAllDims (:570) and FacetsConfig.build see the default. Besides rollup(), only OrdinalMappingLeafReader.java:136 (LFV-722) enumerates getDimConfigs() in core facets; the sandbox cutter has the same loop (TaxonomyFacetsCutter.java:119) and its builder refuses unconfigured dimensions instead (TaxonomyFacetBuilder.java:48-59).

With some depth-1 labels the result is partial: 5 documents under category/electronics/phones, /electronics/laptops, /electronics/phones, /books/fiction and /toys give value=1 childCount=1 toys (1) instead of value=5 childCount=3 electronics (3) books (1) toys (1). The value-source aggregation (TaxonomyFacetFloatAssociations, DoubleValuesSource.fromLongField("price"), MAX, prices 5, 7, 3) gives null instead of value=7.0 2010 (7.0) 2012 (3.0). Counts under the deepest parent, DrillDownQuery, SortedSetDocValues faceting with the same default, a hierarchical multiValued default and explicitly configured dimensions are correct.

• Who is hit. Applications that subclass FacetsConfig and override getDefaultDimConfig() (the documented hook, 4.8.0 and later) or the public getDimConfig (the workaround the LUCENE-5479 reporter had used, 4.7.0 and later) so that dimensions without an explicit entry are hierarchical and single-valued, and that count such a dimension with taxonomy faceting: FastTaxonomyFacetCounts (both constructors), TaxonomyFacetFloatAssociations with a DoubleValuesSource, and DrillSideways' default taxonomy facets (code reading).
• What goes wrong. Dimension and parent counts read 0, getTopChildren(dim) returns null or only the depth-1 leaves, and getAllDims and getTopDims leave the dimension out. Nothing throws, and DrillDownQuery on the same labels still matches, so the facet panel and the result list disagree.
• Opt-in, no scale needed. Deterministic with ordinary data (3 documents suffice), with assertions on and off.
• Likelihood: rare. All three real overrides found in the recorded GitHub code search (in 2 projects, seecr/meresco-lucene and voyanttools/trombone) use multiValued = true, which is unaffected, and the symptom is easy to see, so a user would likely fall back to explicit setHierarchical calls.
• Not affected. The stock FacetsConfig, explicitly configured dimensions, a hierarchical multiValued default, SortedSetDocValues faceting, drill-down and counts under the deepest parent.
• After a fix. The index is written as designed (an index built with the default config and counted with an explicitly configured copy gives the right counts), so a fix in the roll-up corrects existing indexes at search time.

The test sits in org.apache.lucene.facet.taxonomy next to the module's taxonomy tests and uses only public and protected API. Three documents, one dimension, single thread, deterministic. It uses the count-all constructor to stay short; the hits constructor new FastTaxonomyFacetCounts(taxoReader, config, fc) fails the same way. It compiles from 9.1.0 on (org.apache.lucene.tests). For a release, compile and run the same file against that release's lucene-core, lucene-facet and lucene-test-framework jars.

Pass -Ptests.asserts explicitly: on current main its default is drawn from the seed ("Run with assertions for ~75% of all seeds").

The expected string is what the same documents give with config.setHierarchical("Publish Date", true) (the test with that one change passes on stock main). The same test is the suggested regression test; it passes with the recommended fix.

# gradle, main 5adff78bb168, seed DEADBEEF, -Ptests.asserts=true
TestHierarchicalDefaultDimConfig > testUnconfiguredHierarchicalDimIsRolledUp FAILED
    org.junit.ComparisonFailure: expected:<[dim=Publish Date path=[] value=3 childCount=2
      2010 (2)
      2012 (1)
    ]> but was:<[null]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:C525DFB2C6A10945]:0)
        ...
        at app//org.apache.lucene.facet.taxonomy.TestHierarchicalDefaultDimConfig.testUnconfiguredHierarchicalDimIsRolledUp(TestHierarchicalDefaultDimConfig.java:64)

# the same failure: gradle with -Ptests.asserts=false; the JUnit runner on main (-ea and -da), 10.6.0 (-ea and -da), 10.5.2, 9.12.3 and 9.1.0
# main, 20 iterations (seed 5EED0723): 20 of 20 fail
# control, the same test with explicit setHierarchical, stock main: passes
# main with the recommended fix: passes (assertions on and off)

• The hook is documented for exactly this. getDefaultDimConfig javadoc: "Get the default configuration for new dimensions. Useful when the dimension is not known beforehand and may need different global default settings"; "@return The default configuration to be used for dimensions that are not yet set in the FacetsConfig" (FacetsConfig.java:142-148). CHANGES 4.8.0: "LUCENE-5479: FacetsConfig subclass can now customize the default per-dim facets configuration".
• Lucene's only test of the hook sets hierarchical = true (TestFacetsConfig#testCustomDefault, TestFacetsConfig.java:104-117). It checks only getDimConfig("foobar").hierarchical, never a count; no test combines a custom default with taxonomy counts.
• The rest of the module honours the default. Indexing accepts a 3-component path only because the default is hierarchical (with the stock default build throws "is not hierarchical yet has 3 components", FacetsConfig.java:371-378) and leaves the parents to the roll-up; getSpecificValue(dim) says "ok: rolled up at search time" (TaxonomyFacets.java:541); rollup() says it rolls up "any single-valued hierarchical dimensions"; SortedSetDocValues faceting with the same default counts correctly.
• History: an assumption the hook outdated. The loop dates from LUCENE-5339 (2013), when the only default was the static, flat DEFAULT_DIM_CONFIG, so a hierarchical dimension had to be configured explicitly. LUCENE-5479 (2014) made the default overridable and changed only FacetsConfig and its test. Its discussion (#6542) was about a multiValued = true default, where build stores the ancestors itself and the gap does not show.
• Precedent. LUCENE-5801 (#6863, 2014): OrdinalMappingAtomicReader collected facet fields only from getDimConfigs(); it was reported as "wrong facet data" and fixed.

The strongest case for "intended": FacetsConfig is @lucene.experimental; its class javadoc says "use the setters in this class to change these settings for each dim" (FacetsConfig.java:49-51); the hook's javadoc example is multivalue = true; isDimConfigured's javadoc says an unconfigured dimension uses DEFAULT_DIM_CONFIG (:162-165); enumerating getDimConfigs() keeps the taxonomy's children and siblings arrays lazy, which two tests assert; and the sandbox TaxonomyFacetBuilder (#14237) requires explicit configuration for the same reason.

Why it fails: nothing in core documents or enforces that a hierarchical dimension must be configured explicitly. Indexing silently accepts the paths and relies on a roll-up that never comes, and the result is 0 for a label DrillDownQuery finds in 2 documents. The isDimConfigured javadoc is not older than the hook: it was added in 9.0.0 (LUCENE-9971, #149, 2021) and repeats the pre-2014 assumption. Laziness can be kept: the recommended fix walks the taxonomy only when the default needs a roll-up, and both laziness tests pass with it (it does not cover name-dependent getDimConfig overrides, which only the always-walk alternative does). The sandbox comment shows a contributor who knew the loop's limit and chose to fail loudly rather than miscount; "configure hierarchical dimensions explicitly" is a contract upstream could plausibly choose, but then core should enforce it the same way (option D) instead of returning wrong counts. Verdict: not intended, a defect; every reviewer rated the "intended" case weak (the devil's advocate rated only how upstream may frame it as medium).

Four options. A was validated with gradle on main 5adff78bb168; C was run and breaks two existing tests; B and D were not run. In any case, fix the isDimConfigured javadoc (FacetsConfig.java:162-165): an unconfigured dimension uses getDefaultDimConfig(), not necessarily DEFAULT_DIM_CONFIG.

After the existing loop, read the default through getDimConfig of a name without an explicit entry (a small helper unconfiguredDim() returns "", which a FacetField never has as its dimension, padded with '\u0000' while that name is configured). getDefaultDimConfig() takes no name, so this is exactly the default. If it is hierarchical and single-valued, walk the dimension roots (the children of the root ordinal, the same walk getAllDims does, TaxonomyFacets.java:564-578) and roll up each one without an explicit entry whose getDimConfig(dim) is hierarchical and single-valued. Explicit entries are skipped, so nothing is rolled up twice. The full diff is in the Sources block.

    // Dimensions without an explicit entry are not in getDimConfigs(): they use the default
    // DimConfig, which a FacetsConfig subclass may change (getDefaultDimConfig). Walk the
    // taxonomy's dimensions for them only when that default needs a roll-up, so the
    // children/siblings arrays stay lazy otherwise:
    DimConfig defaultConfig = config.getDimConfig(unconfiguredDim());
    if (defaultConfig.hierarchical && defaultConfig.multiValued == false) {
      if (children == null) {
        children = getChildren();
      }
      ParallelTaxonomyArrays.IntArray siblings = getSiblings();
      for (int dimOrd = children.get(TaxonomyReader.ROOT_ORDINAL);
          dimOrd != TaxonomyReader.INVALID_ORDINAL;
          dimOrd = siblings.get(dimOrd)) {
        String dim = taxoReader.getPath(dimOrd).components[0];
        if (config.isDimConfigured(dim) == false) {
          DimConfig dimConfig = config.getDimConfig(dim);
          if (dimConfig.hierarchical && dimConfig.multiValued == false) {
            updateValueFromRollup(dimOrd, children.get(dimOrd));
          }
        }
      }
    }

Trade-offs

• The probe name works around getDefaultDimConfig being protected in another package; a getDimConfig override that answers flat for "" but hierarchical for some other names would escape the gate. So A is validated for getDefaultDimConfig overrides and for getDimConfig overrides that return the effective default for the probe name, and is incomplete for name-dependent overrides (C covers those).
• When the gate opens, the children and siblings arrays are loaded once per reader, as for any explicitly configured hierarchical dimension today.
• Like the existing loop, it does not compare indexFieldName (rolling up a dimension stored in another field adds zeros).
• No index change and no reindex.

What the checks validated

• Gradle in a scratch worktree of 5adff78bb168, OpenJDK 25.0.3, seed DEADBEEF: stock, the repro fails with assertions on and off; with the fix it passes with assertions on and off.
• The whole facet module with the fix, :lucene:facet:test with assertions on: 40 classes, 307 tests, 0 failures, 7 skipped, including TestTaxonomyFacetCounts (19), TestTaxonomyFacetCounts2 (4), TestTaxonomyFacetValueSource (10), TestTaxonomyFacetAssociations (12), TestFacetsConfig (4) and the new test.
• :lucene:facet:check -x test (google-java-format, ecj lint, forbidden APIs) passes with the fix and the test.
• With the fix applied on 5adff78bb168, the earlier independent tests pass (2 of 2, 3 of 3 including the depth-1 case, 5 of 5 including a getDimConfig override), and the value-source probe gives value=7.0 2010 (7.0) 2012 (3.0) like the explicit config.
• The getTopDims and getAllDims probe gives the explicit config's value=3 2010 (2) 2012 (1) with the fix (stock: []).

Add a public accessor to FacetsConfig that returns getDefaultDimConfig(), and use it in A instead of the probe name.

Trade-offs

Same behaviour for the documented hook, no probe name; an API addition to an @lucene.experimental class.

Replace the getDimConfigs() loop with the getAllDims-style walk (diff in the Sources block). It covers every override, including name-dependent getDimConfig overrides, and the repro passes.

Trade-offs

It loads the children and siblings arrays for every count, even with only flat dimensions, and two existing laziness assertions fail: TestTaxonomyFacetCounts#testLabelWithDelimiter (:451, assertFalse(siblingsLoaded())) and TestTaxonomyFacetValueSource#testChildrenAndSiblingsLoaded (:520, expected:<false> but was:<true>); 29 tests, 2 failed.

What the checks validated

• Gradle on 5adff78bb168: the repro passes; TestTaxonomyFacetCounts and TestTaxonomyFacetValueSource: 2 of 29 fail as above.

Throw an IllegalArgumentException when build or TaxonomyFacets meets an unconfigured dimension whose config is hierarchical and single-valued, and say so in the getDefaultDimConfig javadoc, like the sandbox TaxonomyFacetBuilder (TaxonomyFacetBuilder.java:48-59).

Trade-offs

Fails loudly instead of miscounting, but breaks indexing for applications that use the hook today, and makes the hook unusable for hierarchical taxonomy dimensions, the very case its only test shows.

Method: semi-formal, bounded-exhaustive translation validation of the FacetsConfig option matrix.

• The matrix: drill-down terms indexing (5 options) x hierarchical x multiValued x requireDimCount x index field layout x config style (explicit setters, or a getDefaultDimConfig override) x faceting (taxonomy, SortedSetDocValues): 400 configs. Each was checked on every multiset of up to 2 (and, in a second run, 3) small documents, in 1 or 2 segments, with and without deletes, against a reference written from the javadoc.
• While building the matrix, reading TaxonomyFacets.rollup() showed that it iterates getDimConfigs(). In the run on the stock jars, this defect's mismatches then appeared in exactly the 20 taxonomy configs with hierarchical = true, multiValued = false and the default-override style, and in none of the explicit-setter configs (the run's other mismatches belong to a different finding and to an artifact of the check itself, fixed before the second run): 408,567,314 checks at up to 2 documents, 685,912,348 at up to 3, assertions off.
• The finding was reduced to a public-API JUnit test on the real jars, with an explicit-config control.