getTopDims returns null for a multi-valued dimension the query did not hit

Facets#getTopDims(topNDims, topNChildren) is documented to return the same as the first topNDims entries of getAllDims, which lists only the dimensions that had hits. Taxonomy facets and SortedSetDocValues facets override it with a faster version. For a dimension marked setMultiValued(dim, true) and left at requireDimCount = false (the default, and the only option for association dimensions), the override cannot know the dimension's count and uses the placeholder -1, without checking whether any of the dimension's labels was hit. The admission check value != 0 lets -1 through; a dimension without hits has no result, so the list gets a null entry. Two documents are enough (b=b1 indexed before a=a1): getAllDims(10) = [b], but getTopDims(10, 10) = [null, b] and getTopDims(1, 10) = [null], which drops the only dimension with a hit; with -ea, Lucene's own assert facetResult != null throws. It affects FastTaxonomyFacetCounts, TaxonomyFacetIntAssociations, TaxonomyFacetFloatAssociations, SortedSetDocValuesFacetCounts and ConcurrentSortedSetDocValuesFacetCounts, in every release from 9.2.0 (SortedSetDocValues) or 9.3.0 (taxonomy) to 10.6.0 (10.6.0 by reading) and main. Admitting such a dimension only if one of its children has a hit fixes it; the repro and the whole lucene/facet test suite pass with assertions on and off.

• Dimension: a facet field such as Author or Tag; its labels (children) are the values, such as a1. getAllDims(topN) returns one FacetResult per dimension "that had hits" (Facets.java:53), sorted by value; getTopDims(topNDims, topNChildren) "should be the same as calling getAllDims and then only using the first topNDims" (Facets.java:61-62).
• Multi-valued without requireDimCount: FacetsConfig.setMultiValued(dim, true) lets a document carry several labels of one dimension. Then the dimension's count is not the sum of its labels' counts, and Lucene only knows it if setRequireDimCount(dim, true) indexes it, "which is unusual (default is false)" (FacetsConfig.java:120-124); association dimensions cannot set it at all (FacetsConfig.java:305-307). Without it the dimension's value is reported as -1, by getAllDims too.
• The fast getTopDims: taxonomy facets (TaxonomyFacets.java:594-690, used by FastTaxonomyFacetCounts and the int and float association facets) and SortedSetDocValues facets (AbstractSortedSetDocValueFacetCounts.java:142-235, used by SortedSetDocValuesFacetCounts and ConcurrentSortedSetDocValuesFacetCounts) rank dimensions by value in a priority queue first and compute the top labels only for the winners.

1. The fast getTopDims looks at every dimension of the facet field in the taxonomy or reader, hit or not. For a multi-valued dimension without requireDimCount it sets the value to -1, "a place-holder", without looking at the counts of its children (TaxonomyFacets.java:629-638; AbstractSortedSetDocValueFacetCounts.java:174-183).
2. A dimension enters the queue if its value is not 0 (TaxonomyFacets.java:652; AbstractSortedSetDocValueFacetCounts.java:200). That check was meant to skip dimensions without hits, and does so for the other kinds of dimensions, whose value is 0 then. -1 passes. All -1 dimensions tie, and the tie goes to the smaller name.
3. For each dimension in the queue, createFacetResult builds the result; it "Will return null if there are no children" with a hit (TaxonomyFacets.java:301-310; AbstractSortedSetDocValueFacetCounts.java:362-371). The code asserts the result is not null, "since only dims with non-zero values were considered earlier", and stores it anyway (TaxonomyFacets.java:685-687; AbstractSortedSetDocValueFacetCounts.java:228-232).
4. getAllDims builds the same results and drops the nulls (TaxonomyFacets.java:572-575; AbstractSortedSetDocValueFacetCounts.java:121-125), so the two methods disagree exactly on these dimensions.

Dimensions a and b, both multi-valued, requireDimCount false; doc 1 has b=b1, doc 2 has a=a1; the query matches doc 1.

The first three rows hold for FastTaxonomyFacetCounts (dense and sparse counting), TaxonomyFacetIntAssociations, SortedSetDocValuesFacetCounts and ConcurrentSortedSetDocValuesFacetCounts. The top-10 row was run for all of them on main with assertions on and off, and on 10.5.2, 9.12.3 and 9.3.0; the top-1 row by this doc's test for dense taxonomy counts and SortedSetDocValues facets, and by the fact check for the other three on main (on and off) and 10.5.2. The document order matters only for taxonomy top-1: with a indexed first, taxonomy facets visit b first and the sibling bug LFV-363 returns [a value=-1 childCount=1 / b1 (1)] instead of [null], with no assertion even with -ea. The last row comes from the follow-up's probe on main, taxonomy and SortedSetDocValues facets. With requireDimCount=true, or with single-valued dimensions, the same data gives [b].

• Who is exposed. Callers of Facets#getTopDims on FastTaxonomyFacetCounts, TaxonomyFacetIntAssociations, TaxonomyFacetFloatAssociations, SortedSetDocValuesFacetCounts or ConcurrentSortedSetDocValuesFacetCounts with at least one dimension marked setMultiValued(dim, true) without requireDimCount. That includes every multi-valued association dimension (requireDimCount is rejected for them) and the dimensions of an application that makes new dimensions multi-valued by default through getDefaultDimConfig, which its javadoc gives as an example ("like {@code multivalue = true}"), except SortedSetDocValues hierarchical dimensions.
• When. Deterministically, whenever a query has no hits in at least one such dimension and that dimension ranks into the top topNDims admitted dimensions: always when topNDims is at least the number of admitted dimensions, otherwise only against other -1 dimensions, by name, because a dimension with a value above -1 ranks above it. Queries without hits in some dimensions are the "sparse faceting" case getAllDims is documented for; a filtered query rarely hits every dimension.
• What the caller sees (production, assertions off): a null entry, so code that reads each entry's dim or labelValues throws a NullPointerException (inferred from the returned null); a dimension with hits pushed out when topNDims is smaller than the number of admitted dimensions ([null] instead of [b]), which dropping nulls does not undo; in the extreme every slot ([null, null, null]). With assertions on (tests), an AssertionError inside getTopDims.
• Deleting does not help (taxonomy). The taxonomy is append-only, so a dimension whose documents were all deleted and merged away still gets a null entry; SortedSetDocValues facets drop it once the deletes are merged (follow-up probe on main).
• With the sibling bug LFV-363 (taxonomy only: when an entry replaces the queue's bottom, its ordinal is not updated), the null can instead come back as the hit-less dimension's name over another dimension's labels: [m_hidden value=-1 childCount=1 {r1=1}], where r1 belongs to the dimension that had the hit. The recommended fix here removes that combined case too.
• Default vs opt-in. getTopDims and setMultiValued are opt-in calls; requireDimCount=false is the default. Codec, IndexWriterConfig and searcher do not matter (query-time code; the tests use Lucene's randomized test settings and failed with every seed tried).
• Not affected. getAllDims, getTopChildren, hits and drill-down; requireDimCount dimensions; single-valued dimensions (all executed controls); by the code, SortedSetDocValues hierarchical dimensions (they read the indexed dimension count) and the default Facets#getTopDims of other facet classes (a getAllDims prefix). Nothing is written to the index, so a fix needs no reindex.
• Reach. A read-only code search of the default branches found no call to getTopDims in Elasticsearch, OpenSearch or Solr; the bug reaches applications that use the facet module directly. Other branches and private applications were not searched.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError
       org.apache.lucene.facet.taxonomy.TaxonomyFacets.getTopDims(TaxonomyFacets.java:686)
       org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts.getTopDims(FastTaxonomyFacetCounts.java:40)
       org.apache.lucene.facet.TestGetTopDimsDimWithoutHits.testMultiValuedDimWithoutHits(TestGetTopDimsDimWithoutHits.java:78)

• Contract. Facets.java:59-64: getTopDims results "should be the same as calling getAllDims and then only using the first topNDims"; Facets.java:52-56: getAllDims returns "topN labels for any dimension that had hits". Neither allows a null entry or a dimension without hits.
• The code's own invariant. assert facetResult != null (TaxonomyFacets.java:686, AbstractSortedSetDocValueFacetCounts.java:230), with the comment "should not be null since only dims with non-zero values were considered earlier" (:229). The -1 value itself is supported: "we cannot compute an accurate dim count, and use -1 as a place-holder" (TaxonomyFacets.java:635-636), and getAllDims reports the same -1. Only the admission is wrong.
• Tests. TestSortedSetDocValuesFacets#testBasic (:63-130) has dimension a multi-valued without requireDimCount (:65) and asserts getTopDims(10, 10) equals getAllDims(10), a with value -1 included (:125-130); testCombinationsOfConfig (dimension e, :203-205) asserts getTopDims(i, 10) == getAllDims(10).subList(0, i) for every i (:299-306); TestTaxonomyFacetAssociations#testIntSumAssociation (:211-240) asserts getTopDims(10, 10) equals getAllDims(10) for multi-valued association dimensions (:237-240). Every dimension in those tests has hits.
• History. In the review of #747, which added getTopDims, the reviewer asked to keep "the existing behavior, which is that a FacetResult is not returned when there are no children seen" and to "check for a value of 0 and then skip over that dim": the != 0 check is that suggestion. Nobody noted that these dimensions carry -1, not 0, even without hits. #915 (9.3.0) added the asserts; its description says "No new tests were written".

The strongest "intended" case: Facets is @lucene.experimental; the getTopDims javadoc says "dims should be configured as requiring dim counts if using this functionality to ensure accurate counts are available", so a multi-valued dimension without requireDimCount could be called unsupported; the contract says "should"; and the ranking among -1 dimensions is arbitrary anyway.

Why it fails: the advice is about accurate counts (the -1 value, which getAllDims reports the same way), not about null entries or listing dimensions without hits. It cannot be followed for association dimensions, where FacetsConfig throws "AssociationFacetField cannot requireDimCount", and requireDimCount is documented as "unusual (default is false)". Lucene's own tests use this exact configuration and assert the prefix equality, and the code asserts the null cannot happen. Ties are broken by name the same way in both methods, so [b] is well defined. @lucene.experimental concerns API stability, not wrong results. Verdict: a defect (adjudication: confirmed, high; every verifier rated the intended case at most moderate; the only "not real" vote was a read-only review that could not run Java and agreed on the root cause).

The check uses the same criterion as the per-dimension child count that getAllDims relies on (children with a count > 0, for dense and hash-table counting), so for the affected -1 branch getTopDims admits exactly the dimensions getAllDims lists, with the same -1 value and order. Other kinds of dimensions keep the unchanged value != 0 check, so dimensions with hits whose aggregate is 0 are still dropped (#13191, a different problem). The top labels are still computed only for the dimensions that make the final list. It also removes the combined case with the sibling bug LFV-363 (a hit-less dimension is never admitted), but not LFV-363 itself. Only query-time code changes, so no reindex is needed. Workaround until a fix ships: take the first topNDims entries of getAllDims(topNChildren).

TaxonomyFacets.java:637 and AbstractSortedSetDocValueFacetCounts.java:182: replace -1 with a check that stops at the first child with a count > 0, and use 0 (skipped by the existing admission test) when there is none. +41/-4 in the two files, including two small private helpers.

// TaxonomyFacets#getTopDims (multi-valued dim, no requireDimCount)
-              dimValue = -1;
+              dimValue = hasChildWithHits(dimOrd) ? -1 : 0;

// AbstractSortedSetDocValueFacetCounts#getTopDims
-            dimCount = -1;
+            dimCount = hasChildWithCount(ordRange.iterator()) ? -1 : 0;

// hasChildWithHits: walk the dim's children (dense counts) or the hash table
// (sparse counts) and return true at the first child with a count > 0

Trade-offs

• One extra scan per such dimension: it stops at the first child with a hit; for a dimension without hits it walks all of its children (dense counts), the whole hash table (sparse counts, as getTopChildrenForPath already does per dimension) or its whole ordinal range (SortedSetDocValues). The worst case is the number of hit-less -1 dimensions times the number of their children, or times the number of non-zero hash entries in sparse mode; a maintainer may prefer one pass that marks the dimensions with hits (not implemented). No benchmark was run.
• The deferred top-children computation that LUCENE-10488 added is kept: it still runs only for the dimensions in the final list.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 7 of 8 on stock (assertions on and off) and passes 8 of 8 with the fix; with TestSortedSetDocValuesFacets, TestTaxonomyFacetCounts and TestTaxonomyFacetAssociations: 4 suites, 58 tests, 1 skipped, 0 failures, on and off; the whole :lucene:facet:test, 40 suites and 314 tests, 0 failures, with assertions on (7 skipped) and off (8 skipped); :lucene:facet:check -x test (google-java-format, ecj lint, forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test 8 of 8 (-ea, -da); the independent reproducer's 6 of 6; the history test's 13 of 13; the finder's 2 of 2; the association test's 2 of 2; the follow-up's 14-test probe 14 of 14 (7 fail on stock). A mutant with the taxonomy hash-table branch of the check forced to true fails, of the taxonomy tests, only the sparse-counting test, so that branch is covered (the 3 SortedSetDocValues tests also fail in that run because only the taxonomy source was patched in).
• Not run: a backport to 9.x or 10.x (the same lines exist there), x86, a benchmark.

The finder's sketch: in the -1 branch call getTopChildrenForPath, skip the dimension when its child count is 0, and keep the result for the result phase. +23/-3 in the two files.

Trade-offs

Same results. It computes the top labels of every such dimension, including those that do not make the final list, which partly undoes the LUCENE-10488 optimisation.

What the checks validated

The new test passes 8 of 8 (standalone, -ea and -da); the whole :lucene:facet:test, 40 suites and 314 tests, passes with assertions on. Earlier, the finder's, the independent reproducer's, the history and the probe tests passed with it.

Replace the value != 0 test for all kinds of dimensions.

Trade-offs

Would also cover #13191 (dimensions with hits whose aggregate is 0 are dropped), the goal of open PR #13287, which removes the same taxonomy line. Broader behaviour change; not executed.

Call getAllDims(topNChildren) and take the first topNDims entries, which is what the base Facets#getTopDims does.

Trade-offs

Loses the optimisation of the fast getTopDims. Dropping nulls from the fast result is not enough when topNDims is small; setting requireDimCount is impossible for association dimensions.

Method: bounded-exhaustive translation validation of the facet result API (the hunt's "semi-formal" tier), a hunt unit (taxonomy, SortedSetDocValues and drill-sideways facets), 2026-10-26, stock main jars.

• The harness indexes 12 modelled documents and runs every one of the 2^12 hit subsets, with and without 1,100 padding documents (to reach hash-table counting), over several segment layouts and with deletions. For FastTaxonomyFacetCounts, SortedSetDocValuesFacetCounts and ConcurrentSortedSetDocValuesFacetCounts it checks getTopDims against the first entries of getAllDims and exact brute-force counts.
• It first stopped at an AssertionError inside getTopDims. With assertions off, seed 1 gave 1,584 null entries (496 SortedSetDocValues, 512 concurrent, 576 taxonomy) among 3,504 mismatches in 703,437 checks; the other 1,920 mismatches are the sibling bug LFV-363's mislabelled entries.
• With this fix and the sibling's, 184 of 192 configurations completed with 0 mismatches in 131,356,014 checks; the other 8 stopped at the harness's own setup check before checking anything.
• The finder reduced it to a public-API JUnit test with two documents; then independent AI tools reproduced, reviewed and judged it (see the checks).