A shape doc value built from a BytesRef slice relates as the bytes at the start of the array

LatLonShape.createLatLonShapeDocValues(BytesRef), XYShape.createXYShapeDocValues(BytesRef) and createDocValueField(String, BytesRef) accept any BytesRef, which Lucene defines as a slice (offset + length) of a larger array. ShapeDocValues reads the slice's header at the offset, so the bounding box and centroid are right, but its reader then calls ByteArrayDataInput.rewind(), which sets the position to 0 of the whole array; every relate() decodes from there. With square B in bytes 0-75 and square A in bytes 76-151, A's doc value answers OUTSIDE, OUTSIDE, CROSSES for boxes that cross A, contain A and touch only B, instead of CROSSES, INSIDE, OUTSIDE: B's answers; on a MemoryIndex, an INTERSECTS newSlowDocValuesBoxQuery misses A and DISJOINT matches it. With a valid shape before the slice, as here, there is no error, assertions on or off; other prefix bytes give garbage or an exception. Reached through the public factories (since 9.5.0), MemoryIndex (since 9.4.0) and, since 10.6.0 (released 2026-10-29), the bulk BinaryDocValues#binaryValues API, which returns slices; ordinary IndexWriter + IndexSearcher on Lucene's codecs reads values at offset 0 and is not affected. The classes are @lucene.experimental. The defect came with the feature (LUCENE-10654, 2022); a small fix in the inner class ShapeDocValues.Reader that rewinds to the slice's start passes the repros and the shape and MemoryIndex suites.

• Shape doc values: LatLonShape and XYShape can store a shape as one binary doc value: a header (version, number of triangles, bounding box, centroid) followed by a tree of the shape's triangles. LatLonShapeDocValues / XYShapeDocValues read it back; relate(Component2D) answers whether a query shape is outside, inside or crossing it. The slow doc-values shape queries call it per document.
• BytesRef slice: a BytesRef is bytes[offset .. offset + length), "a slice (offset + length) into an existing byte[]" (BytesRef.java:22). Several values can share one array at different offsets.
• ByteArrayDataInput: a reader over a byte array. reset(bytes, offset, len) starts it at the offset; rewind() sets the position to 0, under the comment "NOTE: sets pos to 0, which is not right if you had called reset w/ non-zero offset!!" (ByteArrayDataInput.java:51-55).

1. ShapeDocValues.Reader opens new ByteArrayDataInput(binaryValue.bytes, binaryValue.offset, binaryValue.length) (ShapeDocValues.java:581-582), so it starts at the slice.
2. The comparator reads the header there (:685-694) and then rewinds (:695); Reader.rewind() is this.data.rewind() (:594-596), which goes to position 0 of the array.
3. relate() reads the bounding box from that position, returns at once if the query does not cross it, otherwise walks the tree, and rewinds to 0 again in finally (:743-800). readByte() does not check the slice's end (ByteArrayDataInput.java:131-133), so nothing notices.
4. getBoundingBox() and getCentroid() are computed once from the header (:91-92), and numberOfTerms() and getHighestDimension() return values read there too (:101-102, :139-140), so they stay correct and hide the problem; relate() is the public method that reads the bytes again.

Squares B (40-50) and A (0-10) encode to 76 bytes each; B fills bytes 0-75 and A bytes 76-151. For boxes 5-15 (crosses A), -20 to 20 (contains A) and 45-55 (touches only B), A's own bytes give CROSSES, INSIDE, OUTSIDE; the slice at offset 76 gives OUTSIDE, OUTSIDE, CROSSES, B's answers, because a complete valid shape (B) starts the array. The same for XYShape. In a MemoryIndex holding the slice: INTERSECTS for box 5-15 does not match, DISJOINT matches. With bytes before the slice that are not a valid encoding (the finder's 3-byte zero prefix) the answer is garbage: OUTSIDE for a crossing box, or, for XYShape with assertions on, an AssertionError in XYEncodingUtils.decode. From the code, other invalid prefixes can also throw an ArithmeticException or ArrayIndexOutOfBoundsException (:599-605; ByteArrayDataInput.java:80-82, :131-133 do not check the limit; not executed).

• Who is exposed. Code that uses the experimental shape doc-values API directly with a BytesRef that does not start at index 0: application or library code that calls createLatLonShapeDocValues / createXYShapeDocValues(BytesRef) on a slice (the factories were made public in #11753 so that clients can relate shape doc values in their own collectors); applications that add a createDocValueField(String, BytesRef) field built from a slice to a MemoryIndex and query it with newSlowDocValuesBoxQuery (MemoryIndex keeps the offset, MemoryIndex.java:954); and, on 10.6.0 (released 2026-10-29), main and branch_10x (GITHUB#16286, listed under 10.6.0), code that passes values from the bulk BinaryDocValues#binaryValues straight to the factories: for a dense field the default codec returns every value after the first of a batch as a slice of one array (for values of varying length only in a batch of consecutive doc ids; Lucene90DocValuesProducer.java:1341-1357, :1383-1415), so each relates as the first value. Sparse fields get offset-0 copies (BinaryDocValues.java:99-108).
• Not affected. Ordinary IndexWriter + IndexSearcher with the slow doc-values shape queries on Lucene's codecs: the writer stores the slice's bytes, and per-document binaryValue() returns a buffer at offset 0 (Lucene90DocValuesProducer.java:1331-1338, :1372-1380; the executed control, one segment with the default codec, passes; the other shipped codecs and paths were checked by reading the code only). No Lucene class creates such a slice itself: a slice reaches Lucene's shape queries only when the application supplies one (for example through MemoryIndex), and nothing in Lucene passes bulk values to the shape classes. A custom codec or LeafReader that returns slices would trigger it.
• Opt-in. The classes are @lucene.experimental; otherwise default configuration. One value suffices for a factory call or a MemoryIndex field; the bulk path needs at least two values in one batch.
• What you see. CROSSES or INSIDE become OUTSIDE and OUTSIDE becomes CROSSES, so a box query misses shapes it overlaps and DISJOINT matches them. No exception when a valid shape precedes the slice (other prefix bytes give garbage or an exception); nothing is written to the index.
• How likely. Rare (registry). A downstream remark, from code reading only: one downstream search engine builds its shape doc values from per-document values at offset 0, so it is not affected today.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<CELL_CROSSES_QUERY> but was:<CELL_OUTSIDE_QUERY>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.document.TestShapeDocValuesBytesRefOffset.testRelateWithNonZeroOffset(TestShapeDocValuesBytesRefOffset.java:42)

• The contract. BytesRef.java:22: "Represents byte[], as a slice (offset + length) into an existing byte[]." The factories and field constructors (LatLonShape.java:210-214, :344-352; XYShape.java:193-196, :274-282) take any BytesRef and say nothing about the offset.
• Inconsistent with itself. The reader starts at the offset and the header is read there (ShapeDocValues.java:582, :685-694); relate() alone rewinds to 0.
• The helper's own warning. ByteArrayDataInput.java:51-52, "sets pos to 0, which is not right if you had called reset w/ non-zero offset!!", dates from 2011, before ShapeDocValues existed.
• Lucene produces slices. MemoryIndex.java:954 keeps the caller's offset; and those values reach the shape queries (BaseShapeDocValuesQuery.java:78); the bulk binaryValues (new in 10.6.0) returns slices of one batch array (Lucene90DocValuesProducer.java:1356, :1401), each "a deep copy owned by the caller" (BinaryDocValues.java:75-76) but not at offset 0.
• Intent. The code came with the feature (#1017, 2022); none of its review comments mentions offsets. #11753 made the BytesRef factories public for "custom collectors that work with shapes", which is where values from doc-values APIs, and so slices, arrive. No test calls the BytesRef factories or uses a non-zero offset.
• Precedent. Lucene fixed the same class of bug before: "respect bytesref offset" (1b671cfba3a, 2010) and LUCENE-8766 "Pass BytesRef offset/length when decoding from input stream" (2019).

The strongest case for "intended": the classes and ByteArrayDataInput are @lucene.experimental; the format is opaque; the constructor javadoc speaks of "an already retrieved binary format" (LatLonShapeDocValues.java:41-42), which could mean the field's own value or a per-document binaryValue(), both at offset 0 on Lucene's codecs; and rewind() is documented to go to 0. So offset 0 might be an unwritten precondition, and callers should deep-copy first.

Why it fails: no javadoc states that precondition, and BytesRef defines the value as the slice; the class honours the offset for the header, so offset 0 is not even its own assumption; MemoryIndex and the bulk API, both Lucene's, produce slices; @lucene.experimental relaxes API stability, not the meaning of the bytes passed in; the rewind NOTE warns callers of the helper, and ShapeDocValues is the caller that ignored it; the fix leaves offset-0 behaviour unchanged (not benchmarked). The devil's advocate, a reviewer from another model family (OpenAI GPT-5.6, intended lens) and the adjudication all rated the argument weak, as did the cross-model fact check. For the bulk path alone, a maintainer could instead make binaryValues return offset-0 copies; that would not help the factory and MemoryIndex paths, which are in releases. Verdict: a defect.

One fix was built and validated through gradle (the same change as the finder's); three alternatives were not built. Workaround for users today: BytesRef.deepCopyOf(value) before building a shape doc value from a slice.

Store binaryValue.offset in the Reader and have rewind() set the position back to it.

/** start of the doc value in the backing array */
private final int startOffset;

Reader(BytesRef binaryValue) {
  this.data = new ByteArrayDataInput(binaryValue.bytes, binaryValue.offset, binaryValue.length);
  this.startOffset = binaryValue.offset;
  ...
}

protected void rewind() {
  // ByteArrayDataInput#rewind() goes to position 0, not to the start of the slice
  this.data.setPosition(startOffset);
}

Trade-offs

One int per reader; setPosition is a field store like rewind(). Identical for offset 0, which is every value Lucene's on-disk codecs hand to the shape queries. Reader.clone() rebuilds from the same BytesRef (ShapeDocValues.java:589-591), and the tree walk's position arithmetic is relative (:813, :823), so nothing else depends on absolute positions. No benchmark was run.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: stock + the two new tests and the finder's bulk test: 4 of 4 fail with assertions on and off; with the fix all 4 pass.
• With the fix, :lucene:core:test --tests 'org.apache.lucene.document.*Shape*' and the whole :lucene:memory:test: 33 suites, 251 tests, 0 failures, 19 skipped (18 by the nightly group), with assertions on and off. Besides the new tests: 27 existing core suites (196 tests, among them TestShapeDocValues and the six *ShapeDVQueries) and the 3 existing memory suites (51 tests). The independent fact check got the same counts on a clean extraction.
• :lucene:core:check -x test :lucene:memory:check -x test (format, ecjLint, forbiddenApis) with the fix and the tests: BUILD SUCCESSFUL.
• Standalone runner with the fix: this doc's tests (assertions on and off), and earlier the finder's, the independent reproducer's and the intended-lens stage's tests.
• Not run: the full Lucene test suite, and any benchmark.

rewind() calls data.reset(bytes, offset, length) with the BytesRef's fields.

Trade-offs

Same effect as long as nobody changes the BytesRef after construction: ShapeDocValues keeps the caller's mutable BytesRef (ShapeDocValues.java:84-85), and this variant re-reads its fields on every rewind while the recommended one captures the offset once. Also restores the limit, which nothing changes today.

In ShapeDocValues(BytesRef), use BytesRef.deepCopyOf(binaryValue) when the offset is not 0.

Trade-offs

Copies every sliced value (for example every bulk value after the first) and leaves the reader's absolute rewind() for the next user of Reader.

Remember the reset offset and rewind to it, retiring the 2011 NOTE.

Trade-offs

Changes the meaning of a method that other code calls: the blocktreeords and idversion terms enum frames call rewind() on a ByteArrayDataInput (OrdsSegmentTermsEnumFrame.java:244, IDVersionSegmentTermsEnumFrame.java:215); the default lucene103 frame no longer does (its old calls at SegmentTermsEnumFrame.java:300-307 are inside a comment). Needs its own discussion.

Method: code reading during semi-formal work (the hunt's semi-formal tier).

• The hunt unit built a bounded-exhaustive translation-validation harness for the shape doc-values format: every sequence of up to 4 elements over small grids plus random trees, each encoded with the stock jar, decoded by an independent reference decoder and related against brute force. It checked 63,276,054 trees and 499,269,156 relate() calls with 0 mismatches, assertions on and off, and killed 8 of 8 mutants of ShapeDocValues.
• All of those values sat at offset 0, so the harness could not see this bug. The signal came from writing the reference decoder: Reader opens at binaryValue.offset, rewind() goes to 0, and ByteArrayDataInput.rewind()'s comment warns about exactly that.
• An executed probe and then public-API JUnits (the factories, MemoryIndex, the bulk API) confirmed it.