SortedSetDocValues facets lose a label when a flat multiValued + requireDimCount dimension name contains U+001F or U+001E

SortedSetDocValues (SSDV) facets store each facet path as one term: the dimension and the label joined by U+001F, with U+001F and U+001E inside a name escaped by U+001E (pathToString), and the reader decodes every term with the inverse, stringToPath. For a flat dimension configured setMultiValued + setRequireDimCount, FacetsConfig also indexes a bare dimension term so the dimension count is exact, and that one term is written raw, new BytesRef(facetField.dim) (FacetsConfig.java:494). A dimension name containing U+001F or U+001E, which the API accepts, is then read back as a different path. With 3 documents {x, y}, {x}, {z} under color\u001Fcode, getTopChildren returns value=2 childCount=2 y=1 z=1 instead of value=3 childCount=3 x=2 y=1 z=1, and getAllDims lists a phantom dimension color with label code. A name with two U+001F whose first part is a flat dimension (the default) makes the DefaultSortedSetDocValuesReaderState constructor throw IllegalArgumentException for the whole facet field, and with assertions on a name ending in U+001E throws AssertionError. In a single-dimension index, taxonomy facets, hierarchical SSDV dimensions and flat dimensions without requireDimCount handle the same names as a plain name. It came with LUCENE-9952 (PR #611, 2022), whose first revision escaped the term; a review suggestion replaced it with the raw name as an assumed-equivalent simplification. Every release from 9.1.0 to 10.6.0 (10.6.0 by reading) and main is affected. Escaping the term fixes it and leaves every other index byte-identical; segments written before the fix keep the raw term until they are reindexed.

• SSDV facets: SortedSetDocValuesFacetField(dim, label) indexes each facet value as a term in one SortedSetDocValues field ($facets by default). DefaultSortedSetDocValuesReaderState walks the field's terms in byte order and groups them into dimensions; SortedSetDocValuesFacetCounts counts them.
• Path encoding: FacetsConfig.pathToString joins the parts with U+001F (DELIM_CHAR) and puts U+001E (ESCAPE_CHAR) before any U+001F or U+001E inside a part (FacetsConfig.java:543-553, :569-593). stringToPath is the inverse: "Turns an encoded string (from a previous call to pathToString) back into the original String[]" (:595-626).
• multiValued + requireDimCount: with several labels per document, adding up label counts over-counts the dimension, so for this opt-in configuration FacetsConfig indexes one more term per document, the bare dimension, and the readers read the dimension count from it (LUCENE-9952).

1. The write (FacetsConfig.java:491-497): the bare term is new BytesRef(facetField.dim), raw; the label term is pathToString(facetLabel.components, facetLabel.length), escaped. The hierarchical branch writes its bare term escaped, pathToString(facetLabel.components, i + 1) (:477-480).
2. The terms in byte order for color\u001Fcode (executed term dump of this name on main, from the diagram probe; an independent check's dump of a similar name has the same shape):

ord 0   color \u001E \u001F code \u001F x    escaped label term  -> [color\u001Fcode, x]
ord 1   color \u001E \u001F code \u001F y                         -> [color\u001Fcode, y]
ord 2   color \u001E \u001F code \u001F z                         -> [color\u001Fcode, z]
ord 3   color \u001F code                    raw dimension term  -> [color, code]

After color, U+001E (escaped terms) is smaller than U+001F (raw term), so the raw term sorts last. For a plain name the bare term color is a prefix of every label term and sorts first, which the readers rely on.

1. The read (DefaultSortedSetDocValuesReaderState.java:101-111, :209-221): every term is decoded with stringToPath, and a new dimension starts when the first part changes. Ords 0-2 become dimension color\u001Fcode; ord 3 becomes a second flat dimension color with label code.
2. The count (AbstractSortedSetDocValueFacetCounts.java:274-281, :341-350; getTopDims :172-178): for multiValued + requireDimCount the first ordinal of the dimension is taken as the bare term; the dimension count is read from it and it is skipped when listing labels. Here it is label x: value = count(x) = 2, and x is not listed.

A reader cannot repair this: the raw term D\u001FL has exactly the bytes of the escaped term of dimension D, label L.

• Who is exposed. SSDV faceting with a flat dimension set to setMultiValued(dim, true) and setRequireDimCount(dim, true) (both off by default; the javadoc calls requireDimCount "unusual", FacetsConfig.java:121-122), directly or through a getDefaultDimConfig() override ("Useful when the dimension is not known beforehand", :143-144), and a dimension name with U+001F or U+001E. FacetField.verifyLabel rejects only null or empty names (FacetField.java:64-73), and SortedSetDocValuesFacetField applies it to the dimension (SortedSetDocValuesFacetField.java:53). Realistically: dimension names built from data that uses these characters as separators.
• No scale needed. 3 documents, an uncustomized IndexWriterConfig, assertions on or off; the JUnit's randomized test codec and a main() probe with Lucene's plain defaults give the same counts.
• Silent. With one U+001F or U+001E and assertions off nothing fails: a label disappears, its count is shown as the dimension total, and a made-up dimension appears in getAllDims.
• Whole-field failure. A name with two U+001F whose first part is a flat dimension (the default) makes the reader state constructor throw, so SSDV faceting fails for every dimension in that index field, also for queries that do not match the document, and also after the document is deleted, until a merge removes it (executed on main and 10.5.2, with controls).
• Lasting. A fix changes only the writer. A fixed build reading segments written before the fix still lists the phantom dimension and leaves the old documents out of the dimension total: 3 old + 3 new documents give value=3 x=4 y=2 z=2 instead of 6, and forceMerge(1) keeps it (executed). Affected documents must be reindexed.
• Not affected by this raw term. In a single-dimension index: taxonomy facets, hierarchical SSDV dimensions and flat SSDV dimensions without requireDimCount (executed; two-dimension indexes can still hit the sibling ordering bug). Drill-down, which uses its own escaped terms (DrillDownQuery.java:51, FacetsConfig.java:419-460; executed with drill-down oracles). Search hits (by reading the code: the raw term is only a doc-values value, never an indexed term).
• How likely. Rare: dimension names are usually constants, the bounded prior-report searches found no report in four years, and a public code search found no project that reaches the trigger.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[[x (1), ]y (1)]> but was:<[[]y (1)]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.facet.sortedset.TestSortedSetDocValuesDimCountReservedChar.testDimNameWithDelimChar(TestSortedSetDocValuesDimCountReservedChar.java:48)

• No restriction. No javadoc, package doc, CHANGES or MIGRATE entry limits the characters of a dimension name; FacetField.verifyLabel checks only null or empty (FacetField.java:64-73).
• Escaping is the design. LUCENE-5339 (2013; its commit "remove delim char") replaced the TODO "nuke delimChar? can we escape instead?" with pathToString/stringToPath, which escape every part, the dimension included. stringToPath takes "an encoded string (from a previous call to pathToString)" (FacetsConfig.java:595-598).
• Everything else escapes. The hierarchical bare term (:479-480), the flat label term (:496-497), drill-down terms (:419-460, DrillDownQuery.java:51) and getSpecificValue (AbstractSortedSetDocValueFacetCounts.java:103).
• The same names work elsewhere. In a single-dimension index, taxonomy facets and hierarchical SSDV return value=3 x=2 y=1 z=1 for them, and flat SSDV without requireDimCount returns the same labels with value=-1 (no dimension count for a multiValued dimension, by design, AbstractSortedSetDocValueFacetCounts.java:350-353), exactly as for the plain name (executed). Upstream TestTaxonomyFacetCounts#testLabelWithDelimiter asserts reserved characters in labels are supported.
• History. PR #611's first revision wrote String dimString = pathToString(facetLabel.components, 1);. Review comment: "I believe you can just use facetField.dim here instead of reconstructing the dimension from the label." Reply: "Doh, yep. Thanks!" The "pr feedback" commit switched to the raw name. The two are equal only for names without U+001F/U+001E, and neither the PR nor its issue discusses them.

The strongest "intended" case (the devil's-advocate verifier and the prior-report search): dimension names are application constants, so a control character in one is garbage input; a 2013 review comment on the first SSDV facet code (LUCENE-4795) says "I want to assert that the label is allowed to use the delimiter; only the dimension is not allowed to"; FacetsConfig and Facets are @lucene.experimental and DELIM_CHAR is @lucene.internal; no upstream test puts a reserved character in a dimension name; and the AssertionError is stringToPath's documented precondition.

Why it fails: rarity bounds the impact, not the validity, and getDefaultDimConfig's javadoc and the "sparse" faceting of Facets#getAllDims allow for dimensions not known beforehand (which makes data-derived names plausible; it does not say they come from untrusted data). The 2013 remark predates LUCENE-5339, which replaced that code with escaping of every part, dimensions included; today's API never rejects such a name and handles it in every other path. "Experimental" allows API changes, not wrong counts. An untested input is not an unsupported one. And it is FacetsConfig itself that feeds stringToPath a string pathToString did not produce. Verdict: a defect (adjudication: confirmed, high).

Only the writer can be fixed: the raw term D\u001FL has the same bytes as the escaped term of dimension D, label L, so a reader cannot tell them apart, and a reader-side fix for the sibling ordering bug in the same area would not repair this. Segments written before the fix keep the raw term; executed: a fixed build reading 3 old and 3 new documents gives value=3 x=4 y=2 z=2 (expected 6) and keeps the phantom dimension, also after forceMerge(1). Add the new test with the fix (worth adding: a name equal to another dimension + U+001F + one of its labels), and a CHANGES note that documents indexed with such dimension names must be reindexed. Until a fix ships, applications can reject or normalize U+0000..U+001F in dimension names, or use taxonomy facets. Two more options change behaviour: configuring such dimensions as hierarchical (their bare term is escaped, but hierarchical dimensions can hit the sibling ordering bug when one name is another plus a trailing control character), or dropping requireDimCount on data-derived flat dimensions (the dimension value is then -1, no count).

In FacetsConfig.processSSDVFacetFields, write the bare term of a flat multiValued + requireDimCount dimension through pathToString, as the hierarchical branch and every label term already do. This restores the encoding of PR #611's first revision. Equivalent spellings with the same bytes: pathToString(new String[] {facetField.dim}, 1) (the finder's) or pathToString(facetField.dim). +3/-1 in FacetsConfig.java.

           if (dimConfig.multiValued && dimConfig.requireDimCount) {
             // If non-hierarchical but multi-valued and requiring dim count, make sure to
             // explicitly index the dimension so we can get accurate counts for it:
-            doc.add(new SortedSetDocValuesField(indexFieldName, new BytesRef(facetField.dim)));
+            // Escape the dimension like every other facet term, so stringToPath decodes it back:
+            String dimString = pathToString(facetLabel.components, 1);
+            doc.add(new SortedSetDocValuesField(indexFieldName, new BytesRef(dimString)));
           }

Trade-offs

• No change for correct data: pathToString of one part without U+001F/U+001E is the name itself, so every index that works today gets byte-identical terms. For a reserved name the bare term becomes color\u001E\u001Fcode, a prefix of its label terms, so it sorts first and decodes back to [color\u001Fcode].
• Old segments keep the raw term until reindexed (executed: the phantom dimension and a dimension total of the new documents only stay, also after forceMerge(1)).

What the checks validated

• Gradle, scratch worktree of a9779ef3c193, JDK 25: stock + the new test fails 4 of 5 with assertions on and off; with the fix the new test and TestSortedSetDocValuesFacets (2 suites, 24 tests: 5 + 19, the one nightly-only test skipped) pass with assertions on and off; TestSortedSetDocValuesFacets with nightly tests, 19 of 19; the whole lucene/facet module, 40 suites, 311 tests, 0 failures with assertions on (7 skipped) and off (8 skipped); :lucene:facet:check -x test (ecjLint, forbiddenApis, checkGoogleJavaFormat) passes.
• Standalone runner: the new test 5 of 5 with -ea and -da; a fresh index written by the fixed build gives value=3 x=2 y=1 z=1 and one dimension.
• Earlier checks with the finder's spelling: the finder's, the independent reproducer's and the devil's advocate's tests and the 7-test whole-field probe pass; TestSortedSetDocValuesFacets 18 of 18 run on the standalone runner; in the finder's bounded-exhaustive harness the failing scenarios drop from 274 to 36 of 1,380, all 36 from the sibling ordering bug.

For example in FacetField.verifyLabel for the dimension, or in FacetsConfig.build only for flat multiValued + requireDimCount dimensions.

Trade-offs

Breaks names that work today in taxonomy facets and hierarchical SSDV, and reverses LUCENE-5339's choice of escaping over forbidding; the narrow variant is an odd rule for one configuration. Neither repairs existing indexes. Reasonable only as an application-side workaround.

Recognize or skip the raw bare term in DefaultSortedSetDocValuesReaderState.

Trade-offs

Impossible in general: the raw term D\u001FL is byte-identical to the escaped term of dimension D, label L. A reader-side fix of the sibling ordering bug would not repair this one either.

Method: bounded-exhaustive differential check on the stock main jars (the study's "semi-formal" tier). A hunt unit, "in-band separators and reserved symbols", 2026-10-26.

• A static inventory of the uses of DELIM_CHAR and ESCAPE_CHAR showed FacetsConfig.java:494 as the only SSDV facet term not built by pathToString.
• The model ran SSDV facet counts against taxonomy facets and brute-force drill-down counts for every dimension name of length 1 to 2 over {a, U+001E, U+001F}, labels {a, U+001E, U+001F, U+1F600}, 5 dimension configurations and one- and two-dimension indexes: 1,380 scenarios, 274 failing with assertions on and with them off. With the one-line fix shadowed in, 36 remain, all two-dimension indexes in which one name is the other followed by U+001E or U+001F (for example a and a\u001E), 12 in each of three configurations: the sibling ordering bug of the same area.
• The model ran on a separate test host with the previous main ea57360912ed, whose lucene/facet is identical; all later mainline confirmations used a9779ef3c193, and the release checks used their release jars.
• An AI finder then reduced it to a public-API JUnit, and independent AI tools reproduced, reviewed and judged it (see the checks).