MatchingFacetSetsCounts ignores BytesRef.offset, so a facet-set doc value returned as a slice is counted from the wrong bytes

MatchingFacetSetsCounts.count() reads each document's facet-set value from BinaryDocValues#binaryValue(), but it decodes the dimension count from bytesRef.bytes[0] and runs the set decoder from position 4 to bytesRef.length, both counted from the start of the backing array instead of from bytesRef.offset. A BytesRef is documented as "a slice (offset + length) into an existing byte[]" and binaryValue() promises nothing about the offset, so whenever a reader returns the value as a slice, count() takes the dimension count, and with the stock decoders the sets too, from the bytes that start the array: silently wrong counts, a count of nothing, or a NegativeArraySizeException / ArrayIndexOutOfBoundsException from the constructor. Readers that return slices include a MemoryIndex given a sliced value, a custom or filtering reader that serves values from a shared buffer, and the backward-codecs Lucene80DocValuesFormat in BEST_COMPRESSION mode. FacetSetsField.create with IndexWriter and the default codec always gives offset 0 and is not affected. The code dates from the introduction of facet sets (LUCENE-10274, #841, first released in 9.3.0). The fix is two statements in count().

• Facet sets: a document holds one or more tuples of numbers, such as (3,4), in a FacetSetsField, a binary doc value. MatchingFacetSetsCounts counts, per FacetSetMatcher (an exact tuple or a range per dimension), the matching sets of the hits. The classes are @lucene.experimental.
• Encoding: FacetSetsField packs "a packed byte[] where the first value denotes the number of dimensions in all the sets, followed by each set's values": for the long set (3,4) that is 4 bytes for the count 2, then two 8-byte values, 20 bytes in all.
• Slice: a BytesRef is (bytes, offset, length). A reader may return a value as a window on a larger array (offset > 0) instead of a fresh array (offset 0).

1. count() takes the value, BytesRef bytesRef = binaryDocValues.binaryValue(); byte[] packedValue = bytesRef.bytes;, and decodes the dimension count with IntPoint.decodeDimension(packedValue, 0): byte 0 of the array, not bytesRef.offset (MatchingFacetSetsCounts.java:105-107).
2. It then calls the decoder for start from Integer.BYTES while start < bytesRef.length (:124-125): positions counted from 0.
3. Every stock decoder treats start as an index into bytesRef.bytes: FacetSetDecoder.decodeLongs calls LongPoint.unpack(bytesRef, start, dest), which reads bytesRef.bytes from start (FacetSetDecoder.java:37-40, LongPoint.java:126-134); decodeInts does the same (FacetSetDecoder.java:47-52), and so does the demo's custom decoder. The two sides agree only when bytesRef.offset == 0.
4. For a slice, the document is therefore decoded from the first length bytes of the backing array: another set's bytes (wrong counts), bytes that match nothing, or a nonsense dimension count. Four zero bytes decode to Integer.MIN_VALUE dimensions, and new long[numDims] (:110) throws NegativeArraySizeException in the constructor (:84).

• MemoryIndex keeps the caller's value as a shallow clone, offset included (MemoryIndex.java:954), and returns it from getBinaryDocValues (:1684-1694). The value is a slice when the application builds a BinaryDocValuesField over part of its own buffer, or resets a FacetSetsField with the expert Field#setBytesValue, which does not copy (Field.java:329-346).
• The backward-codecs Lucene80DocValuesProducer in BEST_COMPRESSION mode decompresses 32 documents into one array and returns each value as a window on it (Lucene80DocValuesProducer.java:902-912).
• Any custom or filtering reader that serves values from a shared buffer.
• Not the default codec: FacetSetsField.create builds a fresh array (FacetSetsField.java:52-77), IndexWriter copies values, and the default format returns them at offset 0.

Executed on main a9779ef3c193, assertions on and off (identical):

An independent reproducer swept offsets 1 to 24 of one slice: offsets 1-3 throw ArrayIndexOutOfBoundsException, 4-19 count nothing, 20-24 count the neighbouring set; 24 of 24 differ from the compact copy. These ranges are for that reproducer's layout: other bytes in front of the slice can give other wrong sets, a count that is right by coincidence, or a nonsense dimension count.

• Who. Applications that count facet sets with MatchingFacetSetsCounts (with the stock or demo decoders; a custom decoder may behave differently) over a reader that returns the values as slices: a MemoryIndex given a sliced value (a BinaryDocValuesField over a pooled or shared buffer, or a reused FacetSetsField reset with Field#setBytesValue); a custom or filtering LeafReader that serves binary doc values from a shared buffer; a field explicitly configured with the backward-codecs Lucene80DocValuesFormat(BEST_COMPRESSION). All three were executed.
• The Lucene80 format is not a common path. On main no codec uses it; in 10.x and 9.x only the 8.x codecs do, to read 8.x segments, which cannot hold FacetSetsField values unless hand-encoded. For new data it takes explicit per-field configuration of a format whose own comment says it "is only used on older versions". It shows that a stock Lucene reader returns non-zero offsets. (Read, not run: on 9.x a facet-set doc-values update on an 8.7+ segment is written with that format, so it might be read back as a slice.)
• Not affected. FacetSetsField.create + IndexWriter + the default codec: the controls pass on every executed version (9.3.0, 9.4.0, 9.12.3, 10.0.0, 10.5.2, main) and in every run. The test framework's randomized doc-values formats also return offset 0, which is why upstream's randomized tests cannot catch it.
• Effect. Silent wrong counts in both directions (the document's own sets go uncounted, sets it does not hold are counted); FacetResult.value, the number of documents with a match, can be wrong too (0 instead of 1 in an independent int-set repro). With other leading bytes the constructor throws instead. Search hits and the index are not affected.
• Scale. Deterministic and per document: with the stock decoders, every document whose value has offset != 0 is decoded from the wrong bytes (the count is right only by coincidence, when those bytes hold the same set); one document is enough. With the Lucene80 format, 31 of the 32 values in each full block have a non-zero offset (k-1 of k in a partial last block); when the values have equal length, each is counted as its block's first value.
• How likely. Rare: facet sets are @lucene.experimental, no use was found in Elasticsearch, OpenSearch or Solr, and nothing has been reported since 9.3.0 (2022).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[(3,4) (1), (1,2) (0)]> but was:<[(3,4) (0), (1,2) (1)]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.facet.facetset.TestMatchingFacetSetsCountsSlicedValue.testBinaryValueWithOffset(TestMatchingFacetSetsCountsSlicedValue.java:50)

• The contract allows slices. BytesRef: "Represents byte[], as a slice (offset + length) into an existing byte[]" (BytesRef.java:22). BinaryDocValues#binaryValue(): "Returns the binary value for the current document ID" (BinaryDocValues.java:31-37). No javadoc of MatchingFacetSetsCounts, FacetSetsField, FacetSetDecoder or the facet-sets guide asks for offset 0.
• Lucene's own readers return slices. MemoryIndex keeps the caller's value, offset included (MemoryIndex.java:954; Field#setBytesValue: "the provided BytesRef is not copied"), and the Lucene80 compressed reader returns windows on a decompressed block.
• The same module honours the offset. TaxonomyFacetIntAssociations and TaxonomyFacetFloatAssociations loop from bytesRef.offset to bytesRef.offset + bytesRef.length (TaxonomyFacetIntAssociations.java:76-80, TaxonomyFacetFloatAssociations.java:184-188).
• Lucene hunts this bug class. newBytesRef in the test framework "sometimes uses a non-zero offset ... to tickle latent bugs that fail to look at BytesRef.offset" (LuceneTestCaseParent.java:1165-1213). Earlier fixes of the same kind: LUCENE-10052 ("FSTTermsReader.IntersectEnum was illegally ignoring BytesRef.offset"), LUCENE-5339 (taxonomy facet counts ignored IntsRef.offset), LUCENE-4423, and even a latent one in 2010, "fix nasty bug (doesnt affect any codecs today): respect bytesref offset".
• History. The lines are unchanged since #841 introduced facet sets; its review never discusses offsets.
• The fix keeps offset-0 behaviour. For offset 0 the fixed code computes the same positions.

The intended-behaviour checker built the strongest case for "works as designed": facet sets are documented to be indexed with FacetSetsField, whose factory always gives offset 0; the default codec, SimpleText and every randomized test format return offset 0; the classes are @lucene.experimental; Field#setBytesValue is "Expert"; the Lucene80 format "is only used on older versions"; and FacetSetDecoder.decode never says whether start is absolute or relative to the offset.

Why it fails: nothing makes offset 0 a precondition, and the triggers use only public API and legal inputs: a stock reader (MemoryIndex) given the documented encoding through a plain BinaryDocValuesField, and a stock, registered, writable doc-values format. The decoder question only decides where the fix goes: every stock and demo decoder treats start as absolute, and with either reading today's count() takes the dimension count from the wrong place. (It does add a compatibility caveat to the recommended fix for custom decoders that read start as relative; see the fix trade-offs.) These points lower reach and severity; they do not make the behaviour intended. Verdict: not intended, a defect.

Option A was validated through gradle in a scratch worktree of main a9779ef3c193; option B with the classpath runner only. The others are untested alternatives.

Two statements in MatchingFacetSetsCounts.count(); FacetSetDecoder, LongPoint.unpack and custom decoders stay as they are. Add one javadoc line on FacetSetDecoder#decode (start is an index into bytesRef.bytes) and a CHANGES.txt note, since a custom decoder that read start as relative breaks (trade-offs).

-        int numDims = IntPoint.decodeDimension(packedValue, 0);
+        int numDims = IntPoint.decodeDimension(packedValue, bytesRef.offset);
 ...
-        for (int start = Integer.BYTES; start < bytesRef.length; ) {
+        for (int start = bytesRef.offset + Integer.BYTES, end = bytesRef.offset + bytesRef.length;
+            start < end; ) {
           start += facetSetDecoder.decode(bytesRef, start, dimValues);

Trade-offs

• Matches every stock and demo decoder (absolute start); the same positions as today for offset-0 values; no copy and no allocation.
• FacetSetDecoder#decode does not say whether start is absolute or relative to bytesRef.offset. An outside decoder that reads it as relative can count a slice correctly today, whenever the backing array starts with the same 4-byte dimension count (for example every value but the first in an equal-dimension Lucene80 block, or the MemoryIndex buffer of the repro), and breaks with this fix, which makes it add the offset twice. Executed with such a decoder on the repro's offset-20 slice: stock main counts (3,4)=1 (1,2)=0 (correct); with the fix it throws ArrayIndexOutOfBoundsException: Index 44 out of bounds for length 33. No stock or demo decoder does this. With this fix, document start in FacetSetDecoder#decode and note the change in CHANGES.txt; option B avoids the issue.
• Backportable: the lines are identical at 9.3.0, 9.12.3, 10.5.2 and main. Independent of the open PR #13948.

What the checks validated

• Gradle, scratch worktree of main a9779ef3c193 with both repro files and the lucene/memory test dependency: both files pass with assertions on and off (4 of 4 tests each time), and the SlicedValues test over 10 iterations (30 of 30).
• The whole lucene/facet module (facet sets, taxonomy, sorted-set, range and drill-sideways facets, both repro files included): 41 classes, 310 tests, 0 failed, 8 skipped; stock (SlicedValues test added): 309 tests, 2 failed (its two slice tests).
• :lucene:facet:ecjLint, checkGoogleJavaFormat and renderJavadoc: BUILD SUCCESSFUL. The fix plus the build.gradle line and both tests applies cleanly to a clean a9779ef3c193 tree.
• The hunt's tests (finder, independent reproducer, the cross-model stage's test, devil's advocate) and the counts-outside-the-result-set probe all pass with the same fix.

If bytesRef.offset != 0, copy the value into a reused BytesRefBuilder and decode the copy (+9 lines).

Trade-offs

Works whatever a decoder thinks start is relative to; no change for offset-0 values; but one copy per sliced value for slice-returning readers (31 of the 32 values in each full Lucene80 block).

What the checks validated

Classpath runner only: this doc's SlicedValues test, the finder's and the devil's advocate's tests pass with it (assertions off). Not run through gradle or the module suite.

Read the dimension count at bytesRef.offset and have decodeLongs, decodeInts and LongPoint.unpack add bytesRef.offset.

Trade-offs

Changes a public core method (LongPoint.unpack) and the meaning of start for every custom decoder. Not recommended; not run.

The open, stale PR "Allow reading binary doc values as a RandomAccessInput" (2024, approved, never merged) rewrites count() and would read at the right offset as a side effect.

Trade-offs

A large API change across many files; the two-statement fix is independent of it and backportable.

With assertions off, a value whose first 4 bytes decode to 0 dimensions makes count() loop forever (each decode returns 0 bytes). With assertions on, the stock ExactFacetSetMatcher throws AssertionError: Encoded dimensions (dims=0) is incompatible with FacetSet dimensions (dims=2) instead (ExactFacetSetMatcher.java:39-45); a matcher without that assert would still spin.

Trade-offs

Not this bug: the same loop happens for a hand-built value at offset 0. A clear exception would close both.

Method: semi-formal, a static def-use scan plus a metamorphic harness on the real jars, in a hunt unit on BytesRef offset and length handling (2026-10-26, main ea57360912ed).

• Scan. An ASM-based def-use pass over the bytecode of every non-sandbox module flagged array reads of a BytesRef's bytes whose receiver never has its offset read. A mutation probe checked the scanner: all 7 planted defective methods were flagged; of 4 correct ones, 3 got no flag and 1 only a length-bound flag. It flagged MatchingFacetSetsCounts.count at line 106 (and LongPoint.unpack and FacetSetDecoder.decodeInts on their parameter, whose contract is ambiguous, so they are not reported as separate bugs).
• Harness. For each flagged consumer, a bounded harness compared f(slice) with f(compact copy) on the stock jars: MatchingFacetSetsCounts over a MemoryIndex mismatched in 30 of 30 cases, while planted vacuity mutants reported mismatches as expected. Bounded scope only; no model checker or prover was involved.
• Checks. An independent reproducer, a reading-only review by a model from another family (OpenAI GPT-5.6), an intended-behaviour devil's advocate and an adjudication followed; a refute-lens review by the same outside model returned no verdict (tooling failure).