FacetSetsField.create sizes its buffer from the first facet set, so a later, larger custom set can throw ArrayIndexOutOfBoundsException

FacetSetsField.create(name, sets...) packs the facet sets of one field into one binary doc-values value. It allocates Integer.BYTES + facetSets[0].sizePackedBytes() * facetSets.length bytes (FacetSetsField.java:55 on main 645d4fe14a20) and then advances its write offset by what each set's packValues really writes (:74). FacetSet.sizePackedBytes() is per instance ("the size of the packed values in this facet set"; an upper limit is allowed when the size is unknown in advance, FacetSet.java:52-58), and FacetSets.adoc:137-139 invites custom encodings "e.g. by compressing them". So a custom compressed set that packs larger than the first set can overrun the buffer: create("f", set(7), set(2^40)) with sets of 2 and 7 bytes allocates 8 bytes for 13 and throws ArrayIndexOutOfBoundsException: Index 8 out of bounds for length 8, while the same two sets with the larger first pack, index and count correctly. Packing fails exactly when the sets' mean packed size exceeds the first set's (726 of 1,554 lists of 1-4 sets of 1-6 bytes, checked exhaustively on the real jar). The built-in Int/Long/Float/DoubleFacetSet are fixed-size and cannot trigger it. The exception is thrown while the field is built, so nothing wrong is indexed. Present since facet sets were added in 9.3.0 (#841); a fix that sums every set's size (a four-line loop) passes the repro and the whole facet module.

• FacetSet (FacetSet.java, @lucene.experimental): a set of dims values per document. packValues(byte[] buf, int start) "Packs the dimension values into the given byte[] and returns the number of packed-values bytes ... you can override to implement your own scheme" (:39-43). sizePackedBytes(): "Returns the size of the packed values in this facet set. If the value is unknown in advance (e.g. if the values are compressed), this method can return an upper limit." (:52-58).
• FacetSetsField (FacetSetsField.java): a BinaryDocValuesField whose value is the number of dimensions (an int, 4 bytes) "followed by each set's values" (:23-27). create requires that all sets "must have the same number of dimensions" and throws IllegalArgumentException "if the field name is null or the given facet sets are invalid" (:32-39); the code also requires the same class (:66-73).
• FacetSetDecoder / MatchingFacetSetsCounts: at search time a decoder reads one set and "returns the number of bytes read" (FacetSetDecoder.java:54-58); the counter advances by that value (MatchingFacetSetsCounts.java:124-125). So counting already handles sets of different sizes, given a matching decoder; the stock decodeLongs and decodeInts are fixed-width, like the built-in sets (FacetSetDecoder.java:37-52).

1. FacetSetsField.java:55: byte[] buf = new byte[Integer.BYTES + facetSets[0].sizePackedBytes() * facetSets.length]; sizes the buffer from the first set only.
2. :74: offset += facetSet.packValues(buf, offset); advances by what each set really wrote.
3. The writes need 4 + sum(sizes) bytes; the buffer has 4 + n * sizes[0]. It overflows exactly when sum(sizes) > n * sizes[0], that is when the sets' mean size exceeds the first set's; for two sets, when the second is larger.
4. The buffer is a fresh Java array, so the JVM's bounds check throws ArrayIndexOutOfBoundsException from inside the set's packValues, out of create, before any document reaches IndexWriter.

Bytes printed by an executed probe on main: the dims int is 80 00 00 01, set(7) packs 01 07, set(2^40) packs 06 01 00 00 00 00 00. Smaller first: buffer 8 bytes, holds 80 00 00 01 01 07 06 01, and the next write, at index 8, throws. Larger first: buffer 18 bytes, the field keeps 13 (80 00 00 01 06 01 00 00 00 00 00 01 07), and MatchingFacetSetsCounts counts 1 document for each value. With three sets of 2, 2 and 7 bytes the buffer is 10 bytes and the write fails at index 10. A bounded-exhaustive check (1 to 4 sets, sizes 1 to 6, 3,108 checks) found 726 of 1,554 lists failing, exactly those the formula predicts, and 0 failures when all sizes are equal or every set returns one class-wide bound.

• Who is exposed. Applications that write their own FacetSet subclass whose packed size differs from one instance to the next (compressed, varint or length-prefixed) and return each set's exact size from sizePackedBytes(), as the javadoc describes. The four built-in sets (IntFacetSet, FloatFacetSet: dims * 4; LongFacetSet, DoubleFacetSet: the default dims * 8) and the demo's TemperatureReadingFacetSet (a constant size, CustomFacetSetExample.java:353-355) cannot trigger it.
• Opt-in API. Facet sets are an @lucene.experimental part of the facet module. Within it the failure does not depend on IndexWriterConfig, the codec or assertions: it happens while the field is built.
• Effect. The field cannot be built with its sets in that order, so the document cannot be indexed with it. The field "can be added to a document only once, so you will need to construct all the facet sets in advance" (FacetSets.adoc:98-99), so all the sets for that field go through this one call (an application may still add other FacetSetsFields under other names). The exception comes from a fresh, bounds-checked Java array while the field is built, before anything reaches IndexWriter: no corrupt data, no wrong counts.
• Scale. Any create call with two or more sets whose mean packed size exceeds the first set's: in the bounded check, 726 of 1,554 lists of 1 to 4 sets.
• Workarounds today. Pass the largest-packing set first (then n * sizes[0] >= sum(sizes)), or return one bound valid for every instance from sizePackedBytes().
• How likely: rare. A public code search for sizePackedBytes "extends FacetSet" found 39 files, all copies of Lucene's own sets; no outside variable-size FacetSet was found, and nobody has reported it since 9.3.0.

TestFacetSetsFieldVariableSizeSets (64 lines with the license header) sits in package org.apache.lucene.facet.facetset next to the upstream facet-set tests but uses only public API. Both sets come from one factory method, so they are instances of one anonymous class and create's same-class check passes. The input is fixed and the test is single-threaded and needs no index; LuceneTestCase still randomizes its environment, and every seed tried fails (DEADBEEF and 5EED1009 for this test, 5EED4413 and C0FFEE01 for the finder's equivalent one). The file passes the facet module's google-java-format, ecjLint and forbiddenApis checks.

The test only builds the field. Indexing and counting are not needed: the exception is thrown inside create. The larger-first control (index, commit, count 1 and 1 with MatchingFacetSetsCounts) was run by the finder on every version and is not part of the minimal repro.

cp $REPRO/TestFacetSetsFieldVariableSizeSets.java $LUCENE/lucene/facet/src/test/org/apache/lucene/facet/facetset/
cd $LUCENE && ./gradlew :lucene:facet:test --tests org.apache.lucene.facet.facetset.TestFacetSetsFieldVariableSizeSets -Ptests.seed=DEADBEEF -Ptests.asserts=true

# gradle, clean main 645d4fe14a20 + the test, -Ptests.seed=DEADBEEF -Ptests.asserts=true (the same with =false)
TestFacetSetsFieldVariableSizeSets > testSmallerSetFirst FAILED
    java.lang.AssertionError: expected a field for sets of 2 and 7 packed bytes, got java.lang.ArrayIndexOutOfBoundsException: Index 8 out of bounds for length 8
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:86C9314D31A6E643]:0)
        at org.apache.lucene.facet.facetset.TestFacetSetsFieldVariableSizeSets.testSmallerSetFirst(TestFacetSetsFieldVariableSizeSets.java:58)

        Caused by:
        java.lang.ArrayIndexOutOfBoundsException: Index 8 out of bounds for length 8
            at org.apache.lucene.facet.facetset.TestFacetSetsFieldVariableSizeSets$1.packValues(TestFacetSetsFieldVariableSizeSets.java:35)
            at org.apache.lucene.facet.facetset.FacetSetsField.toPackedValues(FacetSetsField.java:74)
            at org.apache.lucene.facet.facetset.FacetSetsField.create(FacetSetsField.java:45)
            at org.apache.lucene.facet.facetset.TestFacetSetsFieldVariableSizeSets.testSmallerSetFirst(TestFacetSetsFieldVariableSizeSets.java:52)
:lucene:facet:test (FAILURE): 1 test, 1  failure

# standalone JUnit runner on the built jars, seed DEADBEEF
main 645d4fe14a20  -ea   tests_run=1 failures=1
main 645d4fe14a20  -da   tests_run=1 failures=1   (also seed 5EED1009)
10.6.0             -ea   tests_run=1 failures=1
10.5.2             -da   tests_run=1 failures=1
9.3.0              -ea   tests_run=1 failures=1
  => the same AssertionError and cause frames (FacetSetsField.java:74 <- :45)

• Contract. FacetSet.java:52-58: "Returns the size of the packed values in this facet set. If the value is unknown in advance (e.g. if the values are compressed), this method can return an upper limit." An instance method about "this facet set". It permits an upper limit when the size is unknown in advance; it does not say that a set whose exact size is known must return a bound shared by its whole class. Its own example of an unknown size is compressed values, which a maintainer may read as pointing to such a bound (see the strongest case below). packValues "returns the number of packed-values bytes ... you can override to implement your own scheme" (:39-43).
• The docs invite variable sizes. FacetSets.adoc:137-139: "If you implement your own FacetSet and choose to encode the values differently (e.g. by compressing them), you should provide a corresponding implementation of a FacetSetDecoder." The class javadoc of FacetSetsField describes the value as the dimensions "followed by each set's values" (:23-27), with no uniform per-set size.
• The generic path handles variable sizes. The field keeps only the bytes written (new BytesRef(buf, 0, offset), :76); FacetSetDecoder.decode returns "the number of bytes read" and MatchingFacetSetsCounts advances by it (:124-125). The stock decoders are fixed-width, like the built-in sets, so a variable-size set brings its own decoder, as FacetSets.adoc:137-139 says. Inside create, only :55 assumes that every set has the first set's size.
• No documented precondition. create documents the same-dimensions rule and an IllegalArgumentException for invalid sets (:32-39). A size rule would be checked there, not left to an order-dependent array overrun inside the user's packValues.
• No test pins it. The facet-set tests use only the fixed-size built-in sets; no CHANGES.txt entry or MIGRATE.md note mentions sizes.
• History. The line and the javadoc arrived together in f6bb9d218c0e (#841, 2022-06-25). Inside #841, PR commit 40e2992e25 (shaie, 2022-06-15) summed every set's size, new byte[4 + Arrays.stream(facetSets).mapToInt(FacetSet::sizePackedBytes).sum()]; PR commit ab0d4e1624 ("Finish FacetSets.adoc", shaie, 2022-06-19) replaced it with today's line, in the same hunk that added the same-class check. So :55 has been in tension with a per-instance reading of the javadoc from the start. The only review remark on the line (gsmiller, 2022-06-15: "Maybe use Integer.BYTES instead of 4 here?") predates the switch; nobody discussed it (25 commits, 169 review comments, 62 issue comments read).

Strongest case for "works as designed". The switch from the per-set sum to first-set-times-count was a deliberate edit, made together with the same-class check, so the authors pictured "one class, one size". Read that way, the "upper limit" clause is the hook for compressed encodings: a compressed set should return a class-wide worst case. Every shipped set is fixed-size, and the API is experimental.

Why we still read it as a bug. None of that is written down. The javadoc speaks of "this facet set" and offers an upper limit "if the value is unknown in advance"; it does not require a class-wide bound, so a set that knows its exact size and returns it is a natural reading. create documents no size rule and checks none, the same two sets work in one order and crash in the other with an undocumented ArrayIndexOutOfBoundsException, the generic counting loop advances per set, and the PR's own first version summed per-set sizes.

How strong the other reading is: moderate. The compressed-values example in the same javadoc sentence does support a class-wide-bound reading, and the switch in #841 was deliberate.

One fix was built and validated with gradle; two alternatives were not built. A maintainer may prefer option C, which documents a class-wide bound instead.

Replace line 55 of FacetSetsField.java with a loop that adds up every set's sizePackedBytes(). This restores the per-set-sum strategy of #841 PR commit 40e2992e25 by Shai Erera (shaie; it used Arrays.stream(...).sum()), with overflow-checked accumulation: Math.addExact turns an int overflow of the sum into an ArithmeticException. A cross-model review suggested checked accumulation; Math.addExact is the finder's way of doing it.

int sizePackedBytes = 0;
for (FacetSet facetSet : facetSets) {
  sizePackedBytes = Math.addExact(sizePackedBytes, facetSet.sizePackedBytes());
}
byte[] buf = new byte[Integer.BYTES + sizePackedBytes];

Trade-offs

• One extra pass of n sizePackedBytes() calls per document, negligible next to packing; not measured.
• For the fixed-size built-in sets the allocation is byte-for-byte the same as today (n equal sizes sum to n * size).
• Sets that return an upper limit still work: the field keeps only the written bytes.
• The final Integer.BYTES + sizePackedBytes addition is not checked, so a sum above Integer.MAX_VALUE - 4 still wraps to a negative array size. Today's unchecked sizes[0] * n can wrap too: at the same point when all sizes are equal, and never later for a list that works today (those have n * sizes[0] >= sum).
• The new loop calls every set's sizePackedBytes() before the existing same-dimensions and same-class checks (:59-73). Valid lists behave as described; for an invalid mixed list, an unusual sizePackedBytes() of a set of another class (negative, or large enough to overflow) could now throw before the documented IllegalArgumentException. Running the checks in the summing loop would keep today's order (not built).
• It still trusts each set: a negative sizePackedBytes(), or a packValues that writes more bytes than its sizePackedBytes() reports, still fails, as it does today.
• A "Bug Fixes" CHANGES.txt line would be usual; none is drafted.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle: the repro fails on the clean tree (assertions on and off) and passes with the fix.
• :lucene:facet:test --tests "org.apache.lucene.facet.facetset.*" -Ptests.seed=DEADBEEF (assertions off by the seed): 4 suites, 15 tests, 0 failed, 1 skipped (it needs assertions).
• :lucene:facet:test -Ptests.seed=DEADBEEF -Ptests.asserts=true: the whole facet module, 307 tests, 0 failed, 7 skipped (the gradle summary line). The suite count, 40 (39 upstream plus the new one), and the skip reasons (5 nightly-only, 1 monster, 1 without a message) were transcribed from the JUnit XML, which was not kept; the independent fact check re-ran the module in its own worktree and its JUnit XML gave the same 40 suites, 307 tests and 7 skips.
• :lucene:facet:check -x test (google-java-format, ecjLint, forbiddenApis): passes. The separate EditorConfig linter (eclint) is not configured on this machine and did not run.
• Standalone runner with the fix shadowed in: 1 of 1 passes (-ea, -da); an executed probe then packs the smaller-first case as the 13 bytes 80 00 00 01 01 07 06 01 00 00 00 00 00 and counts 1 and 1.
• Earlier, by the hunt: the finder's 2-test JUnit and the upstream facet-set suites (3 + 6 + 5 tests) pass with the same fix; an independent reproducer's own sum fix passes its own test.
• Not run: nightly tests, a benchmark.

Start from 4 + sizes[0] and call buf = ArrayUtil.grow(buf, offset + facetSet.sizePackedBytes()) before :74.

Trade-offs

One pass and also correct, but more code and possibly several array copies for many sets; no advantage over the sum for the small set counts facet sets are meant for.

Change the sizePackedBytes() javadoc to require a bound that holds for every instance of the class, and in toPackedValues throw IllegalArgumentException when a set's sizePackedBytes() exceeds the first set's.

Trade-offs

Keeps today's allocation and turns the order-dependent crash into the documented exception, but narrows a documented extension point and still rejects lists the sum accepts; compressed encodings would have to return their worst case.

Method: semi-formal (the hunt's hunt unit for binary doc-values multi-dimension fields that trust the first element's shape, 2026-10-29, main 645d4fe14a20).

• Lead. While modelling the unit's "trusts facetSets[0]" pattern, the finder read that line 55 sizes the buffer from the first set while line 74 advances by each set's real size.
• Model. Writes are cumulative and every size is positive, so packing overflows iff sum(sizes) > n * sizes[0].
• Bounded-exhaustive translation validation on the stock main jar: n = 1 to 4 sets, packed sizes 1 to 6, two size policies (exact per-instance size; one class-wide upper limit), 3,108 checks. With exact sizes, 726 of 1,554 javadoc-legal lists throw, and the observed failure set equals the model's (0 disagreements). Vacuity: 0 failures with the class-wide limit and 0 when all sizes are equal; assertions on and off identical.
• Reduction. A public-API JUnit with a failing test and a larger-first control (index, commit, count), then minimized to the single failing test.