SortedSetDocValues facets size the child queue from topN, so getTopChildren(Integer.MAX_VALUE, dim) throws where taxonomy facets return every label

SortedSetDocValues (SSDV) facets choose the top children of a dim with a priority queue that they size to the caller's topN, with no cap at the number of labels (AbstractSortedSetDocValueFacetCounts.java:325). Lucene's PriorityQueue allocates topN + 1 slots up front and refuses 2,147,483,631 or more. So getTopChildren(Integer.MAX_VALUE, dim), which Lucene's own taxonomy tests use to ask for every child, throws IllegalArgumentException on a dim with 3 labels as soon as one of its children has a count, and getAllDims(Integer.MAX_VALUE) and getTopDims(.., Integer.MAX_VALUE) fail the same way, in the plain and the concurrent counts. Facets.validateTopN accepts every topN > 0, and taxonomy facets cap the same queue with Math.min(taxoReader.getSize(), topN) and return every label. Below the limit the call returns the right labels but reserves the slots: 67,111,104 bytes for topN = 2^24 on 3 labels; close to the limit it can run out of heap instead. Default configuration, any index size, assertions on and off; 4.7.0 to 10.6.0 and main are affected (18 version points run, the releases in between by reading). Up to 4.6.1 the old API documented Integer.MAX_VALUE as "all children" and SSDV honoured it. The fix is one line: cap the queue at the field's label count, as taxonomy facets do.

1. getTopChildren checks topN with validateTopN (AbstractSortedSetDocValueFacetCounts.java:57), which rejects only topN <= 0 (Facets.java:80-84), then calls getTopChildrenForPath (:61) and computeTopChildren (:300-301). getAllDims (:112-140) does the same for every dim; getTopDims passes topNChildren the same way (:189-191, :226).
2. computeTopChildren (:309-339) walks the path's child ordinals and, at the first child with a count, creates the queue: q = new TopOrdAndIntQueue(topN); (:325). Nothing caps topN at the number of children or labels. With no counted child the queue is never built and the call returns normally.
3. TopOrdAndIntQueue and TopOrdAndNumberQueue pass topN straight to PriorityQueue (TopOrdAndIntQueue.java:24, TopOrdAndNumberQueue.java:42). The PriorityQueue constructor throws IllegalArgumentException for maxSize >= ArrayUtil.MAX_ARRAY_LENGTH (PriorityQueue.java:121-124; Integer.MAX_VALUE - NUM_BYTES_ARRAY_HEADER, 2,147,483,631 here) and otherwise allocates new Object[maxSize + 1] (:129, :134). Its javadoc says it pre-allocates that array (:34); the class is @lucene.internal (:39).
4. SortedSetDocValuesFacetCounts and ConcurrentSortedSetDocValuesFacetCounts both extend AbstractSortedSetDocValueFacetCounts, so both fail; DrillSideways (executed) and MultiFacets (by reading) hand them out for SSDV dims.
5. Taxonomy facets build the same queue type as new TopOrdAndIntQueue(Math.min(taxoReader.getSize(), topN)) (TaxonomyFacets.java:250-251), and the other implementations cap too. The bound SSDV needs is at hand: SortedSetDocValuesReaderState.getSize(), "Number of unique labels", is 3 in the repro index.

Allocation is the minimum over 5 calls after 3 warm-up calls (thread allocation counter). The planning probe on another index measured 381 MiB per dim at topN = 10^8.

• Who is hit. Callers of getTopChildren, getAllDims or getTopDims(.., topNChildren) on SortedSetDocValuesFacetCounts or ConcurrentSortedSetDocValuesFacetCounts (also through DrillSideways, executed, or MultiFacets, by reading, for SSDV dims) with Integer.MAX_VALUE, or any topN from 2,147,483,631 up. The call throws only when the dim or path has at least one child with a count; with none, the queue is never built (AbstractSortedSetDocValueFacetCounts.java:319-325). Code that moves from taxonomy to SSDV facets with that idiom, or shares it between both, meets it on the first such call.
• Large but legal topN. Values such as 2^24 or 10^8 used as "all" work and return correct results, but reserve topN + 1 queue slots per dim with hits, per call: 67,111,104 bytes at 2^24 on 3 labels, 381 MiB per dim at 10^8. Values close to 2,147,483,631 can fail with OutOfMemoryError instead, depending on the heap: with a 1 GiB heap the hunt's harness got OutOfMemoryError: Java heap space at topN = 2,147,483,630.
• Default vs opt-in. Default FacetsConfig, codec and IndexSearcher; no setting is needed. SSDV faceting is an opt-in API marked @lucene.experimental; the trigger is the topN value.
• Who is not. The child queues of taxonomy facets and the other Facets implementations, which cap topN (taxonomy's getTopDims dimension queue is uncapped: the separate LFV-983). SSDV calls below 2,147,483,631 return the right labels as long as the heap holds topN + 1 slots.
• Severity. Loud: an exception for that call; no wrong results, nothing written to the index.
• How likely. Rare: no public SSDV caller with the Integer.MAX_VALUE idiom was found (downstream code uses it on taxonomy and range facets, which cap), and the searches found no public report since 4.7.0 (2014).

The test indexes 3 documents with one SortedSetDocValuesFacetField("dim", label) each (a, b, c), counts MatchAllDocsQuery with SortedSetDocValuesFacetCounts and calls getTopChildren(Integer.MAX_VALUE, "dim"), expecting childCount 3. Single thread; every seed run gave the same failure (on main DEADBEEF and C0DECAFE with assertions on and off, 5EED1234 and 0DD5EED7 with them off, C0FFEE01 with them on; on the 10.6.0 jars DEADBEEF, C0DECAFE and 5EED1234 with both), although the test framework randomizes settings such as the Directory implementation. The IllegalArgumentException is caught once and rethrown as an AssertionError so that the message states expected against actual; the exception is kept as the cause. Set -Ptests.asserts explicitly: with seed DEADBEEF and no value, Lucene's build picks assertions off. Before it goes into a PR, ./gradlew tidy wraps one comment line that is longer than 100 characters; also use MatchAllDocsQuery.INSTANCE there, since new MatchAllDocsQuery() is deprecated on main and in 10.6.0 (the runs used the constructor; it does not affect the failure). The test needs the org.apache.lucene.tests framework and FacetsCollectorManager (recent releases); older releases were checked with release-portable main() programs.

TestSortedSetFacetsTopNMaxValue > testGetTopChildrenWithMaxValueTopN FAILED
    java.lang.AssertionError: expected all 3 labels of dim, got java.lang.IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:3B208D4DB1BA11E3]:0)
        at org.apache.lucene.facet.sortedset.TestSortedSetFacetsTopNMaxValue.testGetTopChildrenWithMaxValueTopN(TestSortedSetFacetsTopNMaxValue.java:56)

        Caused by:
        java.lang.IllegalArgumentException: maxSize must be >= 0 and < 2147483631; got: 2147483647
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:123)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:80)
            at org.apache.lucene.facet.TopOrdAndNumberQueue.<init>(TopOrdAndNumberQueue.java:42)
            at org.apache.lucene.facet.TopOrdAndIntQueue.<init>(TopOrdAndIntQueue.java:24)
            at org.apache.lucene.facet.sortedset.AbstractSortedSetDocValueFacetCounts.computeTopChildren(AbstractSortedSetDocValueFacetCounts.java:325)
            at org.apache.lucene.facet.sortedset.AbstractSortedSetDocValueFacetCounts.getTopChildrenForPath(AbstractSortedSetDocValueFacetCounts.java:300)
            at org.apache.lucene.facet.sortedset.AbstractSortedSetDocValueFacetCounts.getTopChildren(AbstractSortedSetDocValueFacetCounts.java:61)
            at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts.getTopChildren(SortedSetDocValuesFacetCounts.java:57)
            at org.apache.lucene.facet.sortedset.TestSortedSetFacetsTopNMaxValue.testGetTopChildrenWithMaxValueTopN(TestSortedSetFacetsTopNMaxValue.java:53)
1 test completed, 1 failed

# the same with -Ptests.asserts=false; the 10.6.0 release jars fail the same way with -ea and -da

• The API accepts the value. validateTopN "checks if topN is valid for getTopChildren and getAllDims" and rejects only topN <= 0 (Facets.java:74-84); LUCENE-10467 (#751, 9.2) made that range consistent across implementations. getTopChildren "Returns the topN child labels under the specified path" (Facets.java:40-41).
• The sibling implementation caps on purpose. Math.min(taxoReader.getSize(), topN) came with the LUCENE-5339 rewrite (189feaacb6f, Michael McCandless), in the same commit that ported the old Integer.MAX_VALUE test to taxonomy getTopChildren(Integer.MAX_VALUE, "dim"); it is still on main (TestTaxonomyFacetCounts.java:556, TestTaxonomyFacetCounts2.java:338, :343). FloatTaxonomyFacets, StringValueFacetCounts, LongValueFacetCounts, RangeFacetCounts, RangeOnRangeFacetCounts and MatchingFacetSetsCounts cap too.
• A large top-n is the normal way to ask for every child. On #843 Yuting Gan (Yuti-G) wrote that callers keep getting all children "as long as the user specifies a large top-n"; on #974 Greg Miller (gsmiller) wrote that "getAllChildren is not identical behavior as getTopChildren with a huge top-n value", treating the latter as an ordinary call.
• History. Up to 4.6.1 the FacetRequest javadoc (Shai Erera, LUCENE-5136) said: "If set to Integer.MAX_VALUE, all immediate child categories will be returned", and Michael McCandless's SSDV accumulator (LUCENE-4795) answered any request for at least the dim's label count, so Integer.MAX_VALUE too, as "all available results" without a queue. The LUCENE-5339 rewrite (4.7.0) dropped that for SSDV only; none of its comments (90 in the JIRA record, 89 on the migrated #6403), no CHANGES or MIGRATE entry and no test discusses an uncapped SSDV queue. Executed: 4.6.1 returns all labels, 4.7.0 throws.
• Core precedent. LUCENE-4938 (#6002) fixed an IndexSearcher queue that was not capped at maxDoc (reported and fixed by Robert Muir; Michael McCandless and Uwe Schindler traced it), and on LUCENE-8542 (#9588) Michael McCandless called pre-allocating the full topN "really unexpected behavior".

The strongest case for "works as designed": topN is a size the caller chooses; PriorityQueue documents that it pre-allocates and throws on purpose "to prevent confusing OOME"; Facets is @lucene.experimental; getAllChildren exists for "all children"; the sandbox TopnOrdinalIterator also sizes its queue from topN, with a TODO by its author Egor Potemkin (#13568) doubting the taxonomy getSize() cap for very large taxonomies; and the 4.7.0 redesign never repeated the "Integer.MAX_VALUE = all" promise.

Why it fails: validateTopN accepts the value; PriorityQueue is @lucene.internal and its message names an internal maxSize, not topN; getAllChildren is unordered ("Users should make no assumptions about ordering of the children") and has no getAllDims or getTopDims form, and both of those fail too; the same call works on taxonomy facets through a cap added in the same rewrite; and the TODO questions how loose that cap is, which argues for a tighter or lazily growing queue, not for failing. The redesign point is fair, so the history is best presented as a behaviour change in 4.7.0 rather than a broken promise. Verdict: not intended, a defect (low severity).

Option A was applied in two separate scratch worktrees of 645d4fe14a20 and run with gradle (both removed after the runs); the hunt's checks validated the same diff. Options B and C were not built.

One line in computeTopChildren (AbstractSortedSetDocValueFacetCounts.java:325), mirroring TaxonomyFacets.makeTopOrdAndNumberQueue (TaxonomyFacets.java:250-251).

if (q == null) {
  // Lazy init, so we don't create this for the
  // sparse case unnecessarily
  // the path cannot have more children than the field has labels:
  q = new TopOrdAndIntQueue(Math.min(topN, state.getSize()));
}

Trade-offs

• A path's children are distinct ordinals of the field, so the queue never needs more than state.getSize() slots; results are unchanged for every topN that works today.
• Loose when the field holds many labels and the dim few: by default all dims share one field, so a 3-label dim in a field with millions of labels still gets a queue of that size for Integer.MAX_VALUE (the same looseness as the taxonomy cap). The cross-model reviewer (OpenAI GPT-5.6) pointed this out.

What the checks validated

• Gradle, stock 645d4fe14a20 (seed DEADBEEF), in two separate worktrees: the repro fails with -Ptests.asserts=true and false.
• Gradle with the fix, the whole facet module (:lucene:facet:test, seed C0DE0994, assertions on): 40 classes (39 upstream and the repro), 307 tests, 0 failures, 7 skipped (tests a default run skips, e.g. the nightly group); the repro also passes with assertions off. :lucene:facet:checkGoogleJavaFormat passes on the fix.
• First run, with the fix: the repro and TestSortedSetDocValuesFacets, TestDrillSideways, TestParallelDrillSideways, TestFacetQuery and TestFacetLabel, assertions on: 62 tests, 0 failures; the repro and TestSortedSetDocValuesFacets with assertions off and nightly tests on: 20 tests, 0 failures, 0 skipped.
• The executed probe with the fix: Integer.MAX_VALUE returns the 3 labels, and 1 << 24 allocates 2,248 bytes instead of 67,111,104.
• The hunt's checks: the finder's 6 tests and an independent reproducer's 10 tests pass; 7 upstream facet test classes (82 tests) pass with this fix and the sibling LFV-983 fix; the differential harness's saturation and allocation failures drop to 0.

For a flat dim, ordRange.end() - ordRange.start() + 1; a hierarchical DimTree exposes only child iterators, so a child count would have to be computed or stored. Prompted by the cross-model reviewer's remark that A is loose.

Trade-offs

Removes the looseness of A; more code, and the hierarchical case needs new bookkeeping.

Start small and grow only as children arrive, instead of allocating topN + 1 slots up front.

Trade-offs

Would help every top-N facet path, but core PriorityQueue has a fixed capacity, so it needs a different structure; larger than the bug needs.

Add an upper bound to Facets.validateTopN.

Trade-offs

Turns the working taxonomy call getTopChildren(Integer.MAX_VALUE, ...), and Lucene's own tests of it, into errors.

Method: non-formal first signal, then semi-formal checking (an executed probe, then model-based differential testing on the stock jars).

• A planning AI tool for numeric-limit hunts read the facet code and ran a probe (FacetTopNProbe) on main a9779ef3c193, 10.5.2, 9.12.3 and the 10.6 release candidate: SSDV getTopChildren, getAllDims and getTopDims with Integer.MAX_VALUE threw, and topN = 10^8 allocated 381 MiB per dim.
• The hunting AI tool wrote a reference model of top-N facet results (saturation: for topN at least the number of children, the result equals the result at topN = 1000; allocation bounded by the labels) and a differential harness over 40 seeded random indexes, 18 Facets constructions and 5 call shapes: 93,485 checks, of which 9,924 saturation and 3,060 allocation checks failed on stock main (including the sibling dim-queue sites, LFV-983) and 0 with both fixes. An SMT solver confirmed that exactly topN from 2,147,483,631 to 2,147,483,647 throws.
• It reduced the case to a public-API JUnit with a taxonomy control, and later to the one-test repro.