SortedSetDocValues faceting throws NullPointerException for the root of a hierarchical dimension that no document has, instead of returning null

Facets#getTopChildren and Facets#getAllChildren return null "if this dimension was never seen", and SortedSetDocValues (SSDV) faceting has done so for flat dimensions since LUCENE-9971 (9.0.0). For a dimension that FacetsConfig declares hierarchical, the same request throws when no document in the reader has the dimension: NullPointerException: Cannot read field "dimStartOrd" because "dimTree" is null, from SortedSetDocValuesFacetCounts and ConcurrentSortedSetDocValuesFacetCounts, with assertions on and off. A shortcut added in 9.3.0 by #915 (LUCENE-10585, a cleanup) reads state.getDimTree(dim).dimStartOrd for a request without a path, and getDimTree returns null for a dimension with no indexed term. The flat branch next to it, a path below the same dimension, getSpecificValue and taxonomy faceting all treat that state as "never seen". In 9.1.0 and 9.2.0 getTopChildren returns null (getAllChildren did not exist yet: LUCENE-10550 added it in 9.3.0, already affected); 9.3.0 to 10.5.2 and main throw. A 4-line null check that mirrors the flat branch restores the documented result.

• SSDV faceting: facet counting over SortedSetDocValuesFacetField values, without a taxonomy index. DefaultSortedSetDocValuesReaderState reads the facet terms of a reader once; SortedSetDocValuesFacetCounts and ConcurrentSortedSetDocValuesFacetCounts count them for a query and share the base class AbstractSortedSetDocValueFacetCounts.
• Hierarchical dimension: FacetsConfig#setHierarchical(dim, true) lets a dimension have paths such as category/books/fiction. For each such dimension found in the reader, the state builds a DimTree (its first ordinal dimStartOrd plus child and sibling links); a flat dimension gets an OrdRange instead.
• Root request: getTopChildren(n, dim) or getAllChildren(dim) with no path asks for the dimension's top-level children.

1. getTopChildren (AbstractSortedSetDocValueFacetCounts.java:56-63 on main a9779ef3c193) returns null when hasCounts() is false (:58), else calls getTopChildrenForPath, which calls prepareChildIteration (:294). getAllChildren (:66-75) calls prepareChildIteration first (:68) and checks hasCounts() only afterwards (:73).
2. prepareChildIteration (:247-285), hierarchical branch: DimTree dimTree = state.getDimTree(dim); (:252). With a path it runs dv.lookupTerm(...) (:254) and returns null for pathOrd < 0 ("path was never indexed", :259-262). Without a path it takes the shortcut "If there's no path, this is a little more efficient to just look up the dim:" pathOrd = dimTree.dimStartOrd; (:256-257), with no null check.
3. DefaultSortedSetDocValuesReaderState#getDimTree returns prefixToDimTree.get(dim) (:381-387). The map gets an entry only while the state walks a dimension's indexed terms (:185), so a dimension with no indexed term has none, getDimTree returns null and :257 throws.
4. The flat branch handles the same case: if (ordRange == null) { // means dimension was never indexed return null; } (:269-273).
5. So getAllChildren(dim) throws whenever it is called; getTopChildren(n, dim) throws when hasCounts() is true: in practice when the query had hits (strictly, once a segment with hits and the facet field was counted, SortedSetDocValuesFacetCounts.java:79-83, 173-187), always with the count-all constructor, and always for ConcurrentSortedSetDocValuesFacetCounts, whose hasCounts() returns true (ConcurrentSortedSetDocValuesFacetCounts.java:80-84).
6. Not affected: requests with a path, getSpecificValue (term lookup, -1), and getAllDims/getTopDims of the plain counts, which iterate the indexed dimensions only.

One document with only author=bob, category configured hierarchical, default codec and searcher. getTopChildren(10, "category") with hits, getAllChildren("category"), the count-all constructor and the concurrent class: 4 of 4 throw the NPE at AbstractSortedSetDocValueFacetCounts.java:257. Controls on the same state: flat publisher null from both calls; getTopChildren(10, "category", "books") and getAllChildren("category", "books") null; getSpecificValue("category") -1. The independent reproducer adds: getAllChildren after a zero-hit query throws, and getTopChildren returns a result while one document has the dimension, then throws after that document is deleted and forceMerge(1) runs.

• Who. Applications that use SortedSetDocValues facets with a dimension declared hierarchical (FacetsConfig#setHierarchical, not the default) and ask for its root, getTopChildren(n, dim) or getAllChildren(dim), on a reader in which no document has that dimension: the dimension is configured before its first document arrives; one shard, tenant, time partition or filtered index of an application-wide FacetsConfig lacks it; or every document with it was deleted and merged away (unmerged deletes still keep it "seen"). The dimension must be absent from the whole reader, because the state is built from the reader's terms: a filtered index means one built from a subset of the documents, and a query or filter that excludes every document with the dimension leaves it "seen" and does not trigger the bug (executed, probe S2).
• Through Lucene's own code. DrillSideways builds one SortedSetDocValuesFacetCounts per drill-sideways dimension and returns them in a MultiFacets, whose getAllDims calls getTopChildren(topN, dim) on each (MultiFacets.java:81-88). A drill-down on a configured hierarchical dimension that no document has returns its hits (0), then getAllDims(10) on the result throws (executed on ea57360912ed and 10.5.2).
• Default dimension config. An application whose getDefaultDimConfig (FacetsConfig.java:142-151, for dimensions "not known beforehand") makes hierarchical the default gets the exception for any dimension name that is not indexed (executed).
• What happens. The facet call throws instead of returning null; a caller without per-facet error handling fails the whole request. After the exception the same Facets object and reader state answer other dimensions correctly (executed). No wrong results, no index change.
• Default vs opt-in. The only non-default setting is the hierarchical flag of an SSDV dimension: the diagram probe, a plain main() with ByteBuffersDirectory, new IndexWriterConfig() (default codec) and a plain IndexSearcher, throws the same NPE with -ea and -da; the JUnit repro, under LuceneTestCase's randomized directory, codec and IndexWriterConfig, failed with every seed run (executed).
• Scale. Deterministic, one document is enough, assertions on or off.
• Workaround. Check state.getDimTree(dim) != null before asking for the root of a hierarchical dimension (the method throws UnsupportedOperationException for flat ones).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.NullPointerException: Cannot read field "dimStartOrd" because "dimTree" is null
       org.apache.lucene.facet.sortedset.AbstractSortedSetDocValueFacetCounts.prepareChildIteration(AbstractSortedSetDocValueFacetCounts.java:257)
       org.apache.lucene.facet.sortedset.AbstractSortedSetDocValueFacetCounts.getAllChildren(AbstractSortedSetDocValueFacetCounts.java:68)
       org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts.getAllChildren(SortedSetDocValuesFacetCounts.java:57)
       org.apache.lucene.facet.sortedset.TestSsdvHierarchicalDimNotIndexed.testGetAllChildrenOfUnseenHierarchicalDim(TestSsdvHierarchicalDimNotIndexed.java:43)

• The contract. Facets.java:33-37 (getAllChildren) and :39-44 (getTopChildren): "Returns null if the specified path doesn't exist or if this dimension was never seen." Nothing restricts this to flat dimensions.
• The precedent. LUCENE-9971 (#149, 9.0.0), CHANGES.txt:4974-4976: "Requesting facet counts for unseen dimensions in SortedSetDocValueFacetCounts and ConcurrentSortedSetDocValueFacetCounts now returns null / -1 instead of throwing IllegalArgumentException as per Javadoc spec in Facets." In that PR's review (2021-05-25) @gsmiller wrote "I think it's reasonable to return null (it's much like a Java Map)", and in its discussion (2021-05-26) "+1 for making the change you proposed, particularly since it's consistent with the Javadoc".
• Every sibling returns null for the same state (executed): the flat branch (AbstractSortedSetDocValueFacetCounts.java:269-273), the same hierarchical dimension with a path (lookupTerm < 0), getSpecificValue (-1; Facets.java:46-50 "Returns -1 if this path doesn't exist") and taxonomy faceting (FastTaxonomyFacetCounts) with the same FacetsConfig.
• History. Up to 9.2.0 the hierarchical code looked the dimension up with dv.lookupTerm(...) first and returned null for pathOrd < 0; it read getDimTree only after that. #915 (LUCENE-10585) replaced the lookup with the dimStartOrd shortcut; its author wrote that the change "is purely meant to remove some code duplication and make some very minor optimizations, and doesn't modify the API", the PR says "No new tests were written", and CHANGES calls it a "Facet module code cleanup" (CHANGES.txt:4067-4068). When hierarchical SSDV facets were added (#509, 9.1.0), a dimTree == null check was removed as unreachable ("If we reach this point, it means the path was indexed. Shouldn't there always be a dim tree then?"), which was right while the lookup ran first.
• Why tests missed it. TestSortedSetDocValuesFacets#testHierarchicalNonExistentDimension configures fizz as hierarchical but asks for "non-existent dimension", which is flat by default, and asserts null (comment: "make sure the result is null (and no exception was thrown)") (:1779-1815).
• The trigger is legal. Public API only, and every document is built with the same FacetsConfig the search uses, as FacetsConfig's javadoc asks (FacetsConfig.java:53-55).

The strongest case for "intended or misuse", built by the intended-behaviour verifier, a review by a model from another family (OpenAI GPT-5.6) and the adjudication: the facet classes are @lucene.experimental, SortedSetDocValuesReaderState#getDimTree is not documented as non-null, a dimension that is configured but has no documents could be called a configuration/index mismatch, and the upstream never-seen tests only use an unconfigured name.

A narrower precedent a committer may raise: in the same #149 review @gsmiller argued for keeping an IllegalArgumentException when a dimension configured in FacetsConfig is looked up in the wrong index field ("hey, we actually know about this dimension but you're looking in the wrong place!"), and taxonomy faceting does that today (TaxonomyFacets#verifyDim, TaxonomyFacets.java:213-235). It does not cover this case: the repro reads the dimension's own (default) index field, the dimension simply has no document, and that review argued for a deliberate, descriptive exception for a misconfiguration, not for an NPE on a legal state.

Why it fails: experimental means the API may change, not that it may throw an NPE against its own javadoc; the javadoc covers exactly the "never seen" case; every document was built with the same config, as required; the flat branch, the path branch, getSpecificValue and taxonomy faceting all treat the state as never seen; 9.2.0 returned null; and #915 was declared API-neutral. Every reviewer rated the intended argument weak or none. Verdict: not intended, a regression.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF), together with a new upstream-style test in TestSortedSetDocValuesFacets; option B was run on 9.3.0 by the version-history runs; option C was not run.

Four lines in AbstractSortedSetDocValueFacetCounts#prepareChildIteration, right after getDimTree. The shortcut stays and the term lookup that #915 removed does not come back (the cost of the added null check was not benchmarked); the root of an unindexed hierarchical dimension returns null, as in 9.2.0, and a path request for it returns the null it already returned, one step earlier. A second diff adds testConfiguredHierarchicalDimensionNeverIndexed to TestSortedSetDocValuesFacets, next to testHierarchicalNonExistentPath: root and path requests return null and getSpecificValue returns -1, through getAllFacets (sequential or concurrent at random).

     if (dimConfig.hierarchical) {
       DimTree dimTree = state.getDimTree(dim);
+      if (dimTree == null) {
+        // means dimension was never indexed
+        return null;
+      }
       if (path.length > 0) {

Trade-offs

• None for results: it returns what the javadoc, the flat branch and 9.2.0 return.
• It also turns any other route to a missing tree into null. LFV-324 (the SSDV tree is built from ord order and can drop a dimension that is present) may reach the same line for a dropped dimension (inferred, not executed); this check would turn that crash into null but does not fix LFV-324, which belongs in the tree construction.
• Placement: the check sits before the path/no-path split. With the default reader state getDimTree is null only when the dimension has no indexed term, where a path request already returned null; but a custom SortedSetDocValuesReaderState that returns no tree for a dimension with indexed terms, or a tree dropped by LFV-324, would now get null for a path request too instead of an NPE at dimTree.iterator(pathOrd) (:263). To keep that loud, put the check only in the empty-path branch, or use option B. Neither variant was run.

What the checks validated

• Stock main a9779ef3c193: the repro fails 4 of 5 and the new upstream-style test fails (6 tests, 5 failed), with assertions on and off.
• With the fix (gradle, scratch worktree at a9779ef3c193, seed DEADBEEF): the repro and the new test pass, 6 of 6, with -Ptests.asserts=true and =false.
• With the fix, the whole lucene/facet test suite, including both new tests: 312 tests in 40 classes, 0 failures, 8 skipped; checkGoogleJavaFormat and ecjLint of the facet module pass.
• Classpath runner, the fix shadowed onto the stock main jars: the repro passes 5 of 5 and the independent reproducer's test 8 of 8. In the hunt, the finder's fix (the same check) turned the finder harness's 15 NullPointerException entries into 0, and with the hunt unit's fix sketches shadowed together 75 upstream facet tests passed (TestSortedSetDocValuesFacets, TestDrillSideways, TestParallelDrillSideways, TestMultipleIndexFields and three taxonomy suites, seeds DEADBEEF and 1234ABCD).
• Not run: a benchmark (the change adds one null check per root request).

Drop the shortcut: resolve pathOrd with dv.lookupTerm(pathToString(dim, path)) for every request, return null for pathOrd < 0, then read getDimTree, as 9.2.0 did. The version-history runs's one-hunk revert on 9.3.0 is this option, and both repros pass with it.

Trade-offs

One extra term lookup per root request, the cost #915 wanted to save; not run on main.

Return an empty DimTree for a hierarchical dimension with no indexed term.

Trade-offs

Changes a public abstract method of SortedSetDocValuesReaderState that custom implementations override, and an empty tree needs care so that the root is not reported as an existing path; more change than the bug needs.

Method: randomized model-based metamorphic testing (non-formal tier), in a hunt unit on the facet API.

• A written specification of the five Facets methods, taken from their javadoc (for example: getTopChildren and getAllChildren return null for a dimension that was never seen), served as an executable reference.
• The harness built random facet schemas (0-3 count dimensions, flat or hierarchical of depth 1-3, single- or multi-valued, plus association dimensions) over indexes of at most 40 documents in 1-4 segments with random deletions, and compared taxonomy, SSDV and string-value faceting, MultiFacets and TaxonomyFacetLabels, sequential and concurrent, against the reference.
• In 500 iterations with seed 11 its exception buckets for SSDV faceting held 15 NullPointerException entries, all with the dimTree message and all in schemas where a hierarchical SSDV dimension had no document; with the one null check shadowed onto the jar they went to 0. The case was then reduced to a public-API JUnit.
• Found 2026-10-26 on main ea57360912ed; then checked by an independent reproducer, an intended-behaviour devil's advocate, read-only reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a history, prior-report and security follow-up, and a credibility audit.