Taxonomy getTopDims shows another dimension's children under a multi-valued dimension's name

Facets#getTopDims(topNDims, topNChildren) should return the same as the first topNDims entries of getAllDims. Taxonomy facets compute it with a priority queue of topNDims entries, each holding a dimension's name, ordinal and value. When the queue is full and a dimension beats the bottom entry, TaxonomyFacets#getTopDims reuses that entry and updates its name and value but not its ordinal. A multi-valued dimension's children are computed from that ordinal at the end, so the result carries the pushed-out dimension's children and childCount under the winner's name (with requireDimCount, since 9.3.0, also its value), and the list can fall out of value order. Dimensions are visited newest-created first, so it happens whenever an older multi-valued dimension outranks a newer one already in the queue; with topNDims = 1 a multi-valued winner gets the children of the newest-created dimension with a non-zero value (with requireDimCount, the newest-created one with hits). In the repro, x with 2 hits is returned as x value=1 childCount=1 [y1 (1)]. It needs FacetsConfig#setMultiValued, which multi-valued data requires, and fails with and without requireDimCount. Single-valued dimensions, getAllDims, getTopChildren, hits and SortedSetDocValues facets are correct. Every release from 9.2.0 to 10.6.0 (10.6.0 by reading) and main is affected; one added line fixes it.

• Taxonomy facets: each document gets category labels such as x/x1 through FacetField; a side index, the taxonomy, gives each dimension (x) and each label (x/x1) an ordinal, in order of creation. FastTaxonomyFacetCounts counts a query's hits per ordinal; the association facets aggregate a value instead.
• Multi-valued dimension: FacetsConfig.setMultiValued(dim, true), "Pass true if this dimension may have more than one value per document" (FacetsConfig.java:180-181). Such data needs it: FacetsConfig.build rejects a second value of a single-valued dimension ("is not multiValued, but it appears more than once in this document", FacetsConfig.java:233-241). With setRequireDimCount(dim, true) the dimension's own ordinal is indexed too, so its document count is exact (FacetsConfig.java:190-194, :386-397).
• getTopDims: "Returns labels for topN dimensions and their topNChildren sorted by the number of hits/aggregated values that dimension matched. Results should be the same as calling getAllDims and then only using the first topNDims" (Facets.java:60-62). Taxonomy facets override it with a priority queue of topNDims entries; each DimValue holds dim, dimOrd and value (TaxonomyFacets.java:46-56, :610-614).

1. The loop visits dimensions newest-created first: the taxonomy's children array gives "the category that was added last" and each sibling is "the previous youngest child" (TaxonomyIndexArrays.java:253-255, :269; the loop is TaxonomyFacets.java:620-668).
2. While the queue has room, each dimension with a non-zero value gets a new entry (:652-654). Once it is full, a dimension that beats the bottom entry reuses it: bottomDim.dim = dim; bottomDim.value = dimValue; pq.updateTop(); (:656-662). The entry's dimOrd keeps the ordinal of the dimension that was pushed out.
3. At the end, a single-valued dimension takes its children from a cache keyed by name (:639-651, :678-680): correct. A multi-valued dimension's children are computed from dimValue.dimOrd (:683), and for multi-valued dimensions with requireDimCount the value comes from that ordinal too (without it, the -1 placeholder; :506-512). createFacetResult(topChildrenForPath, dim) labels the result with the winner's name (:685).
4. No entry is created after the queue is full, so the ordinals in the queue stay those of the first topNDims dimensions admitted: the newest-created ones with a non-zero value (:652). That includes multi-valued dimensions without requireDimCount, admitted at -1 even without hits (the sibling bug LFV-368). With topNDims = 1 a multi-valued winner therefore gets the children of the newest-created dimension with a non-zero value; with requireDimCount on every dimension, of the newest-created dimension with hits. A single-valued winner keeps its own children, cached by name.

• x/y: taxonomy ordinals 1 = x, 4 = y. Visit y: the queue gets {y, 4, 1}. Visit x: 2 > 1, the entry becomes {x, 4, 2}. The result is built from ordinal 4: x value=1 childCount=1 [y1 (1)] instead of x value=2 childCount=2 [x1 (1), x2 (1)].
• a, b, c with 3, 2 and 1 hits, created in that order, getTopDims(2, 10): [a value=1 childCount=1 [c1 (1)], b value=2 childCount=2 [b1 (1), b2 (1)]], out of value order.
• Variants the verifiers executed: a name tie between dimensions without dim counts (value -1 each); TaxonomyFacetIntAssociations (dimension a reported with b1 (7)); a multi-valued dimension replacing a single-valued one (value=0 and the loser's child); and, when the pushed-out dimension has no hits (possible only through the sibling bug LFV-368), a null entry with assertions off or an AssertionError at TaxonomyFacets.java:686 with assertions on.

• Who is exposed. Applications that call getTopDims on FastTaxonomyFacetCounts, TaxonomyFacetIntAssociations or TaxonomyFacetFloatAssociations (all three executed; the float class with a value source and with indexed associations) with dimensions declared multi-valued, and ask for fewer dimensions than the code admits: every dimension with a non-zero value (TaxonomyFacets.java:652), which includes multi-valued dimensions without requireDimCount even when they have no hits (:637, the sibling LFV-368). The exact trigger is a multi-valued winner replacing the bottom entry of a full queue. The association facets cannot follow the javadoc's requireDimCount advice (FacetsConfig.build rejects it, FacetsConfig.java:305-307), so for them the value stays the -1 placeholder while the children and childCount are wrong. Tags, authors or genres must be multi-valued, because FacetsConfig.build rejects a second value otherwise; the demo's association dimensions are multi-valued (AssociationsFacetsExample.java:59-61), though the demo does not call getTopDims.
• Default vs opt-in. setMultiValued is opt-in (the default dimension config is single-valued) but required for multi-valued data. requireDimCount, which the javadoc recommends for getTopDims, does not help. The defect is in query-time code that runs after counting, so codec, IndexWriterConfig and searcher should not matter; the test (randomized directory, searcher and codec, plain IndexWriterConfig) failed with every seed tried: the fixed seeds and 10 random iterations, on Linux aarch64.
• Scale. Deterministic for a fixed taxonomy, counts and query; one document with three dimensions is enough. A multi-valued entry is right only when no older multi-valued dimension had to push a newer one out of the full queue; with topNDims = 1 and requireDimCount on every dimension, only when the top dimension is also the newest-created one with hits.
• Silent. The wrong entry has the right name and plausible numbers, and nothing is thrown (except in LFV-368's null-entry case). FacetResult#equals compares the value, the childCount and every child label and count but ignores dim and path (FacetResult.java:72-79), so a comparison with equals misses it only when all of those happen to match.
• Not affected. Single-valued dimensions (looked up by name), getAllDims, getTopChildren, getSpecificValue, hits, drill-down, the index (nothing is written) and SortedSetDocValues facets (looked up by name).
• Reach. getTopDims exists since 9.2.0. GitHub code search of the Elasticsearch, OpenSearch and Solr default branches found 0 hits each for getTopDims, FastTaxonomyFacetCounts and DirectoryTaxonomyReader.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[dim=x path=[] value=2 childCount=2
  x1 (1)
  x2 (1)
]> but was:<[dim=x path=[] value=1 childCount=1
  y1 (1)
]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.facet.taxonomy.TestTaxonomyGetTopDimsMultiValued.testOlderDimWithMoreHits(TestTaxonomyGetTopDimsMultiValued.java:62)

• Contract. Facets.java:60-64: "Returns labels for topN dimensions and their topNChildren sorted by the number of hits/aggregated values that dimension matched. Results should be the same as calling getAllDims and then only using the first topNDims. Note that dims should be configured as requiring dim counts if using this functionality to ensure accurate counts are available". FacetResult: dim is the "Dimension that was requested", value the "Total number of documents containing a value for this path", labelValues its "Child counts" (FacetResult.java:24-40). The base method is getAllDims(topNChildren).subList(...) (Facets.java:69-72). Taxonomy facets inherited it only in development history (before #779/#806 on main and #889 on branch_9x); no release shipped that, since 9.1.0 has no getTopDims and 9.2.0 contains #889.
• Twin. AbstractSortedSetDocValueFacetCounts#getTopDims runs the same queue and the same two-field update (:206-208), but has no ordinal in its entry and looks children up by name (:226, :228): correct.
• Tests. TestTaxonomyFacetCounts#testCountRoot (getTopDims(1, 10) at :677), testGetFacetResultsTwice (:722) and testSparseFacets (:253), and TestTaxonomyFacetValueSource (:219, :702) use single-valued dimensions; testRandom asks for every dimension (:1062). The multi-valued tests of TestTaxonomyFacetCounts (setMultiValued at :390, :430, :481, :533) do not call getTopDims, and TestTaxonomyFacetAssociations calls getTopDims(10, 10) for its 4 multi-valued dimensions, more slots than dimensions. A grep of the test trees finds no test that runs the replacement branch with a multi-valued dimension. The tests compare with FacetResult#equals (for example :678, :722), which ignores dim.
• History. Introduced by the LUCENE-10488 optimisation (#779, #806, backport #889). git log -S 'bottomDim.dimOrd' over the full history finds no commit; nobody raised it in the review threads of 12 related PRs.

The strongest "intended" case: Facets is @lucene.experimental; the javadoc says "should be the same", soft wording, and advises requireDimCount; the finder's first repro needs a three-way tie broken by name; the default single-valued dimensions are fine; and FacetResult#equals ignores the name anyway.

Why it fails: @lucene.experimental is about API stability, not wrong numbers. The recommended requireDimCount setting fails with no tie at all (x/y, a/b/c), and the value and childCount are wrong too, so equals sees it. Multi-valued dimensions are a designed-for input: getTopDims has its own multiValued/requireDimCount branch (TaxonomyFacets.java:629-638), and multi-valued data cannot be indexed any other way. The SortedSetDocValues version of the same algorithm is correct, and the fix costs nothing. Both reviews that rated the intended-behaviour argument called it weak; the devil's advocate rejected it (bug real, high confidence) and the adjudication confirmed the bug with high confidence. Two narrower points stand: without requireDimCount, ranking dimensions by the -1 placeholder and then by name is intended (accurate counts need requireDimCount), so there only the wrong children and childCount are the bug; and the association facets cannot set requireDimCount at all (FacetsConfig.java:305-307), so for them the javadoc's advice cannot be followed.

The reused entry then describes the winner completely and the result phase computes the winner's own children; no extra work. Until a fix ships, applications can call getAllDims(topNChildren) and take the first topNDims entries, which is what the base Facets#getTopDims does. Only query-time results change, so no reindex is needed. Either way, add a test that compares getTopDims with the getAllDims prefix for multi-valued dimensions with fewer slots than dimensions and an older dimension winning, and that compares the dimension name, since FacetResult#equals ignores it. Open PR #13287 (2024, stale) rewrites the same replacement block for zero-valued dimensions and also sets only dim and value; a fix will conflict with it. The sibling LFV-368 (dimensions without hits admitted with the -1 placeholder) needs its own fix in the admission line.

TaxonomyFacets.java:659-662: one added line.

                DimValue bottomDim = pq.top();
                bottomDim.dim = dim;
+               bottomDim.dimOrd = dimOrd;
                bottomDim.value = dimValue;
                pq.updateTop();

Trade-offs

None found: the smallest change, no extra work, and it leaves the rest of the method as it is. It keeps the mutable entry, so a future edit could drop a field again; option B removes that risk.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 2 of 3 on stock with assertions on and off; with the fix, the new test plus TestTaxonomyFacetCounts, TestTaxonomyFacetAssociations and TestTaxonomyFacetValueSource pass (4 suites, 44 tests); the whole :lucene:facet:test passes with assertions on (40 suites, 309 tests, 7 skipped, 0 failures) and off (309 tests, 8 skipped, 0 failures); :lucene:facet:check -x test (google-java-format, ecj lint, forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test 3 of 3 with -ea and -da; the independent reproducer's class 7 of 7 (4 of 7 fail on stock). Earlier, with the same line, the finder's and the devil's advocate's tests passed on main, the release-sweep class passed 9 of 9 on main, 10.5.2 and 9.12.3, and the security probe's contract, no-hit and control tests passed (5 of 6; its informational test, which asserts the stale label, flips as expected).

Replace the add/replace block with pq.insertWithOverflow(new DimValue(dim, dimOrd, dimValue)); and make the three DimValue fields final (+6/-15). The queue's comparator (value, then reverse dimension name, TaxonomyFacets.java:610-614) is exactly the hand-written replacement condition, so the same dimensions are kept.

          if (valueComparator.compare(dimValue, 0) != 0) {
            pq.insertWithOverflow(new DimValue(dim, dimOrd, dimValue));
          }

Trade-offs

Makes a stale field impossible and shortens the method; allocates one small object per dimension with a non-zero value instead of at most topNDims. A larger diff than option A.

What the checks validated

Standalone runner on main a9779ef3c193: the new test 3 of 3, the independent reproducer's class 7 of 7, the finder's 2 of 2; gradle: the whole :lucene:facet:test with assertions on (40 suites, 309 tests, 0 failures) and :lucene:facet:check -x test pass.

Drop dimOrd from DimValue and resolve the dimension's ordinal from its name in the result phase.

Trade-offs

Not validated. One ordinal lookup per returned dimension, and more change than option A for the same effect.

Call getAllDims(topNChildren) and keep the first topNDims entries, or ask getTopDims for at least as many dimensions as the code admits (every dimension with a non-zero value, including multi-valued dimensions without requireDimCount, which are admitted at -1 even without hits). More slots do not avoid the sibling LFV-368's null entries.

Trade-offs

Correct, but gives up the optimisation getTopDims exists for; not a fix.

Method: semi-formal. The hypothesis came from code reading; a bounded-exhaustive translation-validation harness witnessed it, and a public-API JUnit test reproduced it.

• The harness models 12 documents over taxonomy and SortedSetDocValues facets and checks every one of the 2^12 hit subsets per seed, with and without 1,100 padding documents, several segment layouts and deletions, comparing getTopDims with the getAllDims prefix and with exact counts.
• On stock main every one of 8 runs reported wrong taxonomy getTopDims results (seed 1: 1,920 mismatching taxonomy getTopDims results without a null entry among 3,504 mismatches in 703,437 checks; the other 1,584 contain a null entry, in taxonomy and SortedSetDocValues facets, the symptom of the sibling bug LFV-368).
• With this one-line fix and the sibling's fix, 184 of 192 configurations completed with 0 mismatches in 131,356,014 checks; the other 8 stopped at the harness's own setup check (AssertionError: maxDoc) before checking anything.
• The finder then wrote a one-document JUnit test (three tied dimensions); the independent reproducer added the no-tie x/y case used here.