StoredFieldDataInput(ByteArrayDataInput) takes the input's end position as its length, so a slice or a partly read input stores extra bytes or closes the IndexWriter

StoredFieldDataInput (#14213, 10.2.0, @lucene.experimental) stores a binary field straight from a DataInput and a byte count. Its convenience constructor StoredFieldDataInput(ByteArrayDataInput) takes that count from ByteArrayDataInput.length(), which returns the input's end position (offset + len), not the bytes it holds. Both copy implementations (the default codec's writer and the base method other formats inherit) then copy that many bytes from the input's current position, so an input that is not at position 0 (a slice at a non-zero offset, or an input partly read first) stores as many extra bytes from the caller's array as its position, silently: the value survives merges and CheckIndex reports the index clean. When too few bytes follow, the copy throws ArrayIndexOutOfBoundsException inside the stored-fields write, which is an aborting exception, and the IndexWriter closes and rolls back to its last commit. Every release from 10.2.0 to 10.6.0 (10.6.0 by reading) and main is affected; a one-line fix (length() - getPosition()) passes the repro and the closest test suites.

• StoredFieldDataInput: a record (DataInput in, int length), "A fixed size DataInput which includes the length of the input", with @param length the length of the data input (StoredFieldDataInput.java:22-29, @lucene.experimental). new StoredField(name, value) stores it as a binary field (StoredField.java:116-122). It was added in 10.2.0 by #14213.
• ByteArrayDataInput: a DataInput over a byte[] with a read position pos and an end limit. new ByteArrayDataInput(bytes, offset, len) and reset(bytes, offset, len) set pos = offset and limit = offset + len (ByteArrayDataInput.java:39-41, :65-69). length() returns limit (:71-73): an end position, deliberately so since 2011, and Lucene's own readers use it that way (while (in.getPosition() < in.length())).

1. The convenience constructor is this(byteArrayDataInput, byteArrayDataInput.length()) (StoredFieldDataInput.java:32-34): the end position becomes the byte count.
2. The default codec's writer writes getLength() as the stored length and copies that many bytes from the input's current position (Lucene90CompressingStoredFieldsWriter.java:312-320). Formats that keep the base method do the same with readBytes (StoredFieldsWriter.java:76-82; SimpleText executed). These are the only two copy implementations in Lucene; the test framework's wrapper writers delegate to them, and index sorting first writes through a temporary default-codec writer. ByteArrayDataInput.readBytes is a plain System.arraycopy that ignores limit (ByteArrayDataInput.java:135-140).
3. So Lucene copies [pos, limit + pos) instead of [pos, limit): pos bytes too many.
4. If those bytes exist in the caller's array, they are stored with no error and read back as part of the value (the reader trusts the stored length, Lucene90CompressingStoredFieldsReader.java:293).
5. If not, arraycopy throws ArrayIndexOutOfBoundsException inside the stored-fields write. IndexingChain treats any exception there as aborting (IndexingChain.java:1504-1509), which becomes a tragic event: the IndexWriter rolls back to its last commit and closes (IndexWriter.java:6098-6106).

For a fresh slice (bytes, offset, len) the value is silently offset bytes too long when 2*offset + len <= bytes.length; otherwise the writer closes. Executed on main a9779ef3c193:

The 12-byte value of the first row survives forceMerge(1) of 2 segments into 1 and CheckIndex reports the index clean, with plain new IndexWriterConfig() defaults; an index written that way and opened with the fix still returns the 12 bytes for that document and 8 for a new one. In a probe of a shared 128-byte buffer, a 16-byte record at offset 16 stored 32 bytes, the record plus the next record of the buffer, and that value survived forceMerge(1) from 2 segments to 1; CheckIndex reported the index clean.

• Who is exposed. Callers of new StoredField(name, new StoredFieldDataInput(in)) where in is a ByteArrayDataInput built with new ByteArrayDataInput(bytes, offset, len) or reset(bytes, offset, len) with offset > 0 (for example from a BytesRef with a non-zero offset, or a slice of a pooled buffer), or an input that was partly read before wrapping (a header). Every such document is hit.
• Settings. Plain defaults are enough: a probe with new IndexWriterConfig() (default codec, default merge policy) stores the 12 bytes. The JUnit pins the default codec and takes the rest of its configuration from the test framework's randomized newIndexWriterConfig(). Other stored-fields formats are hit through the base writer method: SimpleText fails the same way (executed with the hardened upstream test).
• Silent and lasting. In the common case nothing fails: the value is pos bytes too long, made of whatever follows in the caller's array (the next record of a shared buffer, or a previous request's leftovers in a reused one, in a probe). It survives commit and merges, CheckIndex reports the index clean, and a fixed Lucene still returns the extra bytes for segments already written (executed: an index written by unfixed main, opened with the fix, returns 12 bytes for the old document and 8 for a new one).
• Loud variant. When fewer than pos bytes follow, addDocument throws ArrayIndexOutOfBoundsException and the IndexWriter rolls back to its last commit and closes. A rollback discards every change since the last commit (IndexWriter.rollback() javadoc); the probe shows one buffered, uncommitted document lost besides the bad one (the committed index keeps 1 of the 2 documents added before it), and the next addDocument throws AlreadyClosedException.
• Bounded. At most pos extra bytes per value, never outside the caller's own array; no query-side effect.
• How likely. Plausible but narrow. Slicing a buffer is a natural way to build the input (the introducing PR's own test builds it from value.offset), but no non-test Lucene code calls this constructor, and public code search found no downstream caller in the repositories searched (the only hit outside Lucene, a codec wrapper, only delegates writeField; private code is not visible).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[4 5 6 7 8 9 a b]> but was:<[4 5 6 7 8 9 a b c d e f]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.index.TestStoredFieldDataInputSlice.testStoredValueOfSliceAtNonZeroOffset(TestStoredFieldDataInputSlice.java:44)

• Contract. StoredFieldDataInput: "A fixed size DataInput which includes the length of the input", @param length the length of the data input, and getLength() "Returns the length of the data input" (StoredFieldDataInput.java:22-29, :41-44). StoredFieldVisitor.binaryField(FieldInfo, StoredFieldDataInput): "Implementors of this method must read StoredFieldDataInput#length bytes" (StoredFieldVisitor.java:43-45). The reader builds the record with a byte count (Lucene90CompressingStoredFieldsReader.java:293).
• The author's test. #14213 added to BaseStoredFieldsFormatTestCase.testMergeFilterReader a field b2 = new StoredFieldDataInput(new ByteArrayDataInput(value.bytes, value.offset, value.length)) and asserts it equals value (:687-693, :724-726). value is a fresh BytesRef with offset 0. The same pattern at offset 3 fails: "b2 must equal b expected:<[61 62 63 64]> but was:<[61 62 63 64 7 7 7]>"; with a random offset of 1-5 it fails in the Lucene90 and SimpleText format tests.
• Same PR, same intent. testNullStoredDataInputField, "test a null data input value doesn't abort the entire segment", asserts no tragic exception and that the earlier document survives (TestIndexWriterExceptions.java:1757-1783). (There the null is rejected by the StoredField constructor before indexing, so it shows intent, not this path.)
• No precondition anywhere. The constructor's javadoc is "Creates a StoredFieldDataInput from a ByteArrayDataInput" (StoredFieldDataInput.java:31). The #14213 discussion never mentions offsets, slices or length(); CHANGES has only "GITHUB#14213: Allowing indexing stored-only StoredField directly from DataInput" under 10.2.0 (lucene/CHANGES.txt:1920).
• Effect. Silently wrong stored bytes, or a closed IndexWriter with lost documents, for a valid input built with public constructors.
• Precedent. LUCENE-4423 (2012), an offset/length bug on the read side of binary stored fields, was fixed as a bug.

The strongest case for "works as designed": ByteArrayDataInput.length() returning the end position is deliberate and old (LUCENE-3224, 2011, changed limit = len to limit = offset + len; length() returns limit since LUCENE-3030), and Lucene code uses it as an end bound (Lucene104PostingsReader.java:1451). Both classes are @lucene.experimental; ByteArrayDataInput "omits all low-level checks" and even warns that rewind() "sets pos to 0, which is not right if you had called reset w/ non-zero offset!!" (ByteArrayDataInput.java:23-24, :51-55). The canonical constructor StoredFieldDataInput(DataInput, int) handles slices. So the convenience constructor just copies length(), and slice users should pass the length.

Why it fails: the record's length is documented and consumed as a byte count, nothing tells callers the convenience constructor needs position 0, and the introducing PR's own test builds the input from an offset and expects exactly len bytes back. The quirks of ByteArrayDataInput lower the severity; they do not make silently wrong stored data or a closed writer intended. Verdict: a defect in the convenience constructor, not in ByteArrayDataInput. The independent reproduction and the cross-model reviews rated the intended-behaviour argument weak; the intended-behaviour check and the adjudication rejected it without a strength rating.

The partly read input is the weaker half. The cross-model fact check of the brief (2026-10-27, reading only) rates the argument weak for a fresh slice but moderate for an input read partly before it is wrapped: for an input that starts at 0, length() as its total extent is more defensible, and no upstream test wraps a partly consumed input (the #14213 test covers only the slice form). There a documentation-only answer or removing the convenience constructor is a more plausible maintainer response. Still against it: the record's length is consumed as a byte count from the current position, and no javadoc says the input must be unread.

Option A was built and validated; option B (a test change) was validated with it. C and D are alternatives that were not built. Until a fix ships, pass the length explicitly, new StoredFieldDataInput(in, len), or wrap a fresh ByteArrayDataInput at position 0 over an array that holds exactly the value. A fix does not rewrite segments already written: users who used the pattern need to check and reindex.

StoredFieldDataInput(ByteArrayDataInput) passes length() - getPosition() and its javadoc says it holds the bytes from the current position to the end. 1 file, +6/-2.

-  /** Creates a StoredFieldDataInput from a ByteArrayDataInput */
+  /**
+   * Creates a StoredFieldDataInput from a ByteArrayDataInput, holding the bytes from its current
+   * position to its end
+   */
   public StoredFieldDataInput(ByteArrayDataInput byteArrayDataInput) {
-    this(byteArrayDataInput, byteArrayDataInput.length());
+    // ByteArrayDataInput#length() is the end position of the input, not the number of bytes left
+    this(byteArrayDataInput, byteArrayDataInput.length() - byteArrayDataInput.getPosition());
   }

Trade-offs

• Unchanged for every input at position 0, which covers all of Lucene's own callers (TestBinaryDocument, TestDirectoryReader, BaseStoredFieldsFormatTestCase).
• Does not repair values already written.
• Expected cost: one accessor call and one subtraction when the record is built; not benchmarked. The same line is on branch_10x and branch_10_6 (read), so the diff should apply there; the fix ran against the 10.2.0 jars, not the branch heads.
• Unchanged by the fix: the record wraps the live input and takes its length once, at construction. Moving the input after wrapping makes the count stale, and adding the same Document twice re-reads a consumed input. The diff's javadoc "holding the bytes" is loose on this; "whose length is the number of bytes from its current position to its end when it is created" is closer.
• Edge cases (read, not executed): pos == limit gives an empty value; pos > limit (possible via the unchecked setPosition and skipBytes) gives a negative length, which the base writer rejects with NegativeArraySizeException and the default codec's writer, with assertions off, writes as a negative vInt with no bytes. The record does not validate an explicit negative length either. Throwing IllegalArgumentException in the constructor when getPosition() > length() would close this before addDocument, without aborting the writer.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: stock + the new test, 4 of 5 fail with assertions on and off; + the fix, the new test passes 5 of 5, and the closest suites pass with assertions on (seed DEADBEEF) and off (seed 5EED0175): TestStoredFieldDataInputPosition, TestBinaryDocument, TestDirectoryReader, the three TestLucene90StoredFieldsFormat* classes, TestCompressingStoredFieldsFormat and TestIndexSorting in lucene/core (188 tests), TestSimpleTextStoredFieldsFormat (19) and TestAssertingStoredFieldsFormat (19): 226 tests, 13 skipped, 0 failures, with option B applied and a line-wrapping draft of the new test; the final test with the fix passes in gradle (5 + 20 + 19 tests, assertions on; and 121 tests with A + B and assertions off in the fact check). On the final files google-java-format, ecjLintTest and forbiddenApisTest pass; an earlier check run on the draft passed ecjLint and forbiddenApis but failed google-java-format on the draft test's line wrapping.
• Standalone runner: 5 of 5 on main with assertions on and off, and on the 10.2.0 jars with the class shadowed in. Earlier checks with the same line: the finder's test 3 of 3, the independent reproducer's 5 of 5, the cross-model reviewer's 4 of 4, the security probe 4 of 4. The persistence probe passes with the fix (8 bytes after forceMerge(1), CheckIndex clean).
• Not run: the full lucene/core test suite.

In BaseStoredFieldsFormatTestCase.testMergeFilterReader, build b2 from a copy of value at offset 1-5 of a randomly padded array (+6/-2), so every stored-fields format test covers the case.

Trade-offs

A test change only, to ship with option A. One small padded array per test document: expected negligible, not measured.

What the checks validated

On stock main it fails in TestLucene90StoredFieldsFormat and, through the base StoredFieldsWriter.writeField, in TestSimpleTextStoredFieldsFormat (ArrayIndexOutOfBoundsException: arraycopy: last source index 38 out of bounds for byte[33] for seed DEADBEEF), with assertions on and off; with option A all stored-fields format suites above pass. Why it fails on stock for any seed: with offset o (1-5) and padding p (0-5), stock reads up to index 2o + len of an array of o + len + p, so p < o throws and p >= o stores o extra bytes, which the read-back check catches; the DEADBEEF message fits o = 5, p = 0 and a 28-byte value.

Say in the constructor javadoc that the input must be fresh and at position 0, and point slice users to StoredFieldDataInput(DataInput, int). Not built.

Trade-offs

Leaves silently wrong stored data for anyone who misses the sentence, and contradicts the introducing PR's own test pattern.

The API is experimental, so it may change. Not built.

Trade-offs

API churn for the whole-array case, which works today; callers must compute the length themselves.

Throw when a read passes limit.

Trade-offs

The class deliberately "omits all low-level checks" on a hot path, and an exception inside the stored-fields write would still abort it and close the writer.

Method: code reading, then a JUnit on the stock jars (non-formal).

• The hunt unit on the compressing stored-fields format built a Z3 model of the writer's chunk arithmetic. That model does not mention StoredFieldDataInput.
• While reading the write paths for the model, the finder noticed that StoredFieldDataInput(ByteArrayDataInput) copies length(), which ByteArrayDataInput defines as its limit, and wrote a 3-test public-API JUnit that failed on main, 10.5.2 and 10.2.0. It was found on 2026-10-26 in about 0.7 hours.
• Before the candidate was filed, an independent reviewer from another model family (OpenAI GPT-5.6) tried to refute it and instead reproduced it with its own test. An independent reproducer, a devil's advocate, a second cross-model review and an adjudication followed, then history, prior-report and security follow-ups.