TaxonomyFacetLabels returns ancestor labels a document was never given when its dimension is multi-valued

TaxonomyFacetLabels (since 8.7.0) is documented as a way to "retrieve previously indexed facet labels, allowing you to skip also adding stored fields for these values". For a dimension configured multi-valued plus hierarchical, it also returns every ancestor of each added path (Date/2010/10 reads back as [Date/2010, Date/2010/10]), and for multi-valued plus requireDimCount the bare dimension (Author). FacetsConfig indexes these on purpose so that multi-valued counts are right, and after dedup they cannot be told apart from labels the user added, so this is a documentation/contract defect: the index and the counts are correct, and the fix is a javadoc note.

• Hierarchical dimension (FacetsConfig.setHierarchical): paths may have more than one component below the dimension, such as Date/2010/10.
• Multi-valued dimension (setMultiValued): "this dimension may have more than one value per document" (FacetsConfig.java:180). requireDimCount (setRequireDimCount): "you require accurate counts of the dimension" (:190-193). All three default to false (:113-124).

1. For multiValued && (hierarchical || requireDimCount), FacetsConfig.processFacetFields adds the ordinal of every ancestor of each added path ("Add all parents too"), and removes the dimension's own ordinal unless requireDimCount is set (FacetsConfig.java:386-398). It then sorts and dedupes the ordinals (:403-415).
2. This is intended: TaxonomyFacets.rollup adds child counts into parents only for hierarchical single-valued dims (TaxonomyFacets.java:272), because a document with Date/2010/10 and Date/2010/11 would be counted twice at Date/2010. Multi-valued dims store each ancestor once per document instead (LUCENE-5339, 4.7.0).
3. FacetLabelReader.nextFacetLabel(docId) returns taxoReader.getPath(ord) for every stored ordinal, with no filter (TaxonomyFacetLabels.java:111-140). The reader receives no FacetsConfig (constructor :45-48), and the ordinals carry no record of who added them.
4. So the ancestors come back as labels. The dim-filtered nextFacetLabel(docId, dim) keeps proper descendants of the dimension (:142-150, :207-213): it returns the ancestors too, but never the bare dimension.
5. After dedup, a document given only Date/2010/10 and one given Date/2010 and Date/2010/10 store identical ordinals (executed), so no general reader-side filter can drop only the implicit ones. The dimension setting alone triggers it: the repro's document has one value.

The table lists each document's labels as a set; the reader returns them in ordinal order, and its javadoc says they "may not be in the same order in which they were indexed" (TaxonomyFacetLabels.java:102-103). The requireDimCount rows use non-hierarchical dims, which accept only one-component paths such as Author/Bob (FacetsConfig.java:371-377 throws for more).

• Who. Code that lists a hit's labels (breadcrumbs, tag lists) from this class on a dimension configured setMultiValued plus setHierarchical or plus setRequireDimCount. A document given only a deeper path, such as Date/2010/10, shows ancestors it was never given, deterministically, with assertions on and off; a document that also added those ancestors explicitly sees no difference.
• Opt-in. Every trigger needs non-default FacetsConfig settings, and the class is @lucene.experimental. The default codec, IndexWriterConfig and taxonomy suffice.
• Scale. Date/2010/10 returns 2 labels, Category/Electronics/Phones/Android 3; requireDimCount adds the dimension; shared ancestors of several values come back once.
• Not affected. Counts; drill-down with the default DrillDownTermsIndexing.ALL (DrillDownQuery matches the ancestor in both configurations); the index; single-valued dims. With a non-default drill-down indexing that skips sub-paths, such as FULL_PATH_ONLY, a DrillDownQuery on a returned ancestor label matches nothing (executed). In the Lucene tree only tests use the class; GitHub code search of the default branches of Elasticsearch, OpenSearch and Solr found no use of TaxonomyFacetLabels, DirectoryTaxonomyReader or DirectoryTaxonomyWriter (2026-11-01).

• Input. One dimension, one two-level path, ByteBuffersDirectory, default IndexWriterConfig, no randomness; the package is for locality only.
• Assertions. Pass -Ptests.asserts explicitly: the build picks assertions from the seed, and for DEADBEEF it picks off.
• Releases. The same file fails unchanged on 9.1.0, 9.12.3, 10.0.0, 10.5.2 and 10.6.0; before 9.1 only the LuceneTestCase package differs (the finder's class fails on 8.7.0 to 9.0.0 too).
• Regression test (optional for a javadoc fix): this same method with the expected list set to the documented output, List.of(new FacetLabel("Date", "2010"), new FacetLabel("Date", "2010", "10")); it passes with assertions on and off.

cp $REPRO/TestTaxonomyFacetLabelsMultiValuedHierarchical.java $LUCENE/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/
cd $LUCENE
./gradlew :lucene:facet:test --tests org.apache.lucene.facet.taxonomy.TestTaxonomyFacetLabelsMultiValuedHierarchical -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false (same failure)

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestTaxonomyFacetLabelsMultiValuedHierarchical > testLabelsOfMultiValuedHierarchicalDim FAILED
    java.lang.AssertionError: expected:<[FacetLabel: [Date, 2010, 10]]> but was:<[FacetLabel: [Date, 2010], FacetLabel: [Date, 2010, 10]]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:70C10E831672F2CA]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.facet.taxonomy.TestTaxonomyFacetLabelsMultiValuedHierarchical.testLabelsOfMultiValuedHierarchicalDim(TestTaxonomyFacetLabelsMultiValuedHierarchical.java:56)
1 test completed, 1 failed

• Contract. Class javadoc (TaxonomyFacetLabels.java:27-29). CHANGES 8.7.0: "LUCENE-9444: Add utility class to retrieve facet labels from the taxonomy index for a facet field so such fields do not also have to be redundantly stored". Issue #10484: "A facet field may be included in the list of fields whose values are to be returned for each hit"; a comment by mikemccand (2020-08-01): "We want to make it easy for users to get back their originally indexed facet labels."
• Tests. TestTaxonomyFacetLabels.testBasic sets only setHierarchical and asserts exactly 10 labels for 5 documents with 2 each; TestTaxonomyFacetCounts.testRandom uses a plain FacetsConfig and expects exactly the added labels. No test, review comment or decision covers multi-valued hierarchical or requireDimCount dims.
• Metamorphic. For one document, flipping only setMultiValued (documented as "may have more than one value per document") turns [Category/Electronics/Phones] into [Category/Electronics, Category/Electronics/Phones] (executed).
• Through FacetField, the bare dimension is never a user label: new FacetField("Author") throws "path must have at least one element", so in a field built by FacetsConfig it comes only from requireDimCount. Code that bypasses FacetsConfig (TaxonomyWriter.addCategory(new FacetLabel("Author")) plus its own SortedNumericDocValuesField) can still index one on purpose.

Strongest case for "works as designed": the javadoc says previously indexed, and the ancestors are indexed; the demo user guide says "Multi value support requires storing all parent category ordinals in the category list field for accurate counting"; the reader takes no FacetsConfig; ordinals carry no record of who added them; the class is @lucene.experimental.

Why it fails for the documentation: it defeats a code-bug framing, which is why this is classed as a documentation defect, but not the gap itself. The guide sentence is about counting, lives in the demo module, is not linked from TaxonomyFacetLabels and says nothing about the dimension ordinal; a user who picks the class over stored fields cannot learn that the labels differ from what they added. Cross-model reviews (OpenAI GPT-5.6) split the same way: not a code bug (refute lens), a real documentation defect (works-as-designed lens).

No general reader-side filter can remove only the implicit ancestors: after dedup they are indistinguishable from ancestors the user added, and the reader receives no FacetsConfig. An application that never adds ancestors explicitly can filter them itself, or keep stored fields for those dims.

Add a NOTE paragraph to the class javadoc, with the Date/2010/10 example, and change the getFacetLabelReader @param from BinaryDocValues to SortedNumericDocValues (stale since 9.0.0).

 * Utility class to easily retrieve previously indexed facet labels, allowing you to skip also
 * adding stored fields for these values, reducing your index size.
 *
+ * <p><b>NOTE</b>: a label is returned for every taxonomy ordinal indexed for the document, which is
+ * not always exactly the labels that were added. For a dimension that is {@link
+ * org.apache.lucene.facet.FacetsConfig#setMultiValued(String, boolean) multi-valued} and also
+ * {@link org.apache.lucene.facet.FacetsConfig#setHierarchical(String, boolean) hierarchical} or
+ * {@link org.apache.lucene.facet.FacetsConfig#setRequireDimCount(String, boolean) requires the
+ * dimension count}, {@link org.apache.lucene.facet.FacetsConfig} also indexes the ancestors of
+ * every added path, so that counts are correct, and these are returned too: a document given only
+ * {@code Date/2010/10} returns {@code Date/2010} and {@code Date/2010/10}. With {@code
+ * requireDimCount}, the dimension itself ({@code Date}) is indexed and returned as well, except by
+ * {@link FacetLabelReader#nextFacetLabel(int, String)}. Such ancestors cannot be told apart from
+ * ancestors that were added explicitly; keep stored fields for these dimensions if you need exactly
+ * the added labels.
+ *
 * @lucene.experimental
 */
...
-   * @param readerContext LeafReaderContext used to access the {@code BinaryDocValues} facet field
+   * @param readerContext LeafReaderContext used to access the {@code SortedNumericDocValues} facet
+   *     field

Trade-offs

• No behaviour change, no risk.
• Optionally add one line to the demo guide's MultiValued bullet and to FacetsConfig#setMultiValued pointing at the label consequence.

What the checks validated

Scratch worktree at 645d4fe14a20: checkGoogleJavaFormat, ecjLintMain and renderJavadoc for :lucene:facet pass (the independent fact check found the note's five links resolving in the rendered page); TestTaxonomyFacetLabels passes; the minimal test still fails, as it must; the same method with the documented expectation passes with assertions on and off.

In a field built by FacetsConfig, a length-1 label can only come from requireDimCount (FacetField needs at least one path component), and the dim-filtered overload already excludes it. null passes through as before.

       assert currentPos <= currentDocOrdinalCount;
-      if (currentPos == currentDocOrdinalCount) {
-        return null;
+      while (currentPos < currentDocOrdinalCount) {
+        int ord = (int) ordinalValues.nextValue();
+        currentPos++;
+        FacetLabel label = taxoReader.getPath(ord);
+        // Skip the bare dimension label: FacetsConfig indexes it only for requireDimCount
+        // counting, and a FacetField always has at least one path component
+        if (label == null || label.length > 1) {
+          return label;
+        }
       }
-
-      int ord = (int) ordinalValues.nextValue();
-      currentPos++;
-
-      return taxoReader.getPath(ord);
+      return null;

Trade-offs

• A behaviour change in an experimental API that someone might rely on (for example to list a document's dims); it fixes only the requireDimCount half.
• It changes the generic meaning of a public reader and would also hide a bare dimension that code bypassing FacetsConfig indexed on purpose; passing the facet suite does not show it is compatible for such callers.
• With B, reword A's requireDimCount sentence.

What the checks validated

On top of A: format, ECJ lint and javadoc pass; the whole :lucene:facet:test suite passes (306 tests, 7 skipped); the independent reproducer's requireDimCount test ([Color, Color/blue, Color/red] on stock main) passes, while its three ancestor tests and the minimal test still fail, as expected.

A leaf-only filter would silently drop ancestors the user added (indistinguishable after dedup). Recording which ordinals were added explicitly needs an index-format change and extra storage, out of proportion for an experimental utility. Workaround, not a Lucene fix: an application that never adds ancestors explicitly can apply the leaf-only filter itself.

• Method. A random model-based harness indexed random documents under random dimension configurations and checked the public facet API against a reference model written from the javadoc, independent of the implementation. Relation F7: the labels returned for a document equal the paths added to it. Run on main: 400 iterations, seed 2, 327,122 checks, 3,010 F7 mismatches (extra ancestor and dimension labels).
• Verification. The finder's public-API JUnit, an independent reproducer (7 tests), cross-model reviews (OpenAI GPT-5.6, reading only), a devil's advocate with probes, an adjudication re-run and a follow-up with a release matrix; the minimal repro then cut it to one method.