ByteBuffersDataInput.readLongs and readFloats can fail when blocks are smaller than 8 bytes

ByteBuffersDataInput reads a stream stored in equal, power-of-two-sized ByteBuffer blocks. Its bulk readLongs and readFloats read through cached LongBuffer/FloatBuffer views of a block, one per alignment, and compute that alignment from the absolute stream position ((int) pos & 0x7, (int) pos & 0x3) instead of the offset inside the block. For blocks of 8 bytes or more the two agree. For the 1-, 2- and 4-byte blocks Lucene allows (LIMIT_MIN_BITS_PER_BLOCK = 1, or the List<ByteBuffer> constructor), a bulk read at certain misaligned positions positions a block past its end and throws IllegalArgumentException, before the code can take its existing slow path for values that span blocks; the catch clause does not handle this type. Defaults (1 KiB blocks and up, or a single buffer) are not affected; this defect always throws, it never returns a wrong value. In every release since 9.0.0 and on main.

// store/ByteBuffersDataInput.java
public void readLongs(long[] arr, int off, int len) throws EOFException {   // :328
  try {
    while (len > 0) {
      LongBuffer longBuffer = getLongBuffer(pos);                           // :331  throws here
      longBuffer.position(blockOffset(pos) >> 3);                           // :332  in-block offset
      int chunk = Math.min(len, longBuffer.remaining());
      if (chunk == 0) {
        // read a single long spanning the boundary between two buffers
        arr[off] = readLong(pos - offset);                                  // :336  slow path
        ...
  } catch (BufferUnderflowException | IndexOutOfBoundsException e) {        // :350  IAE not caught

private LongBuffer getLongBuffer(long pos) {                                // :372
  int bufferIndex = blockIndex(pos);
  int alignment = (int) pos & 0x7;                                          // :375  absolute position
  int longBufferIndex = bufferIndex * Long.BYTES + alignment;
  if (longBuffers[longBufferIndex] == null) {
    ByteBuffer dup = blocks[bufferIndex].duplicate();
    dup.position(alignment);                                                // :379  > limit if block < 8 B
    longBuffers[longBufferIndex] = dup.order(ByteOrder.LITTLE_ENDIAN).asLongBuffer();
  }
  return longBuffers[longBufferIndex];
}

getFloatBuffer (:359-369) is the same with (int) pos & 0x3 (:362) and dup.position(alignment) (:366); readFloats (:296-325) has the same catch clause (:318).

1. The 11 bytes (3 zeros, then the long 42) sit in 2-byte blocks. Position 3 is in block 3 >> 1 = 1, at in-block offset 3 & 1 = 1.
2. getLongBuffer(3) computes alignment = 3 & 7 = 3 and calls position(3) on a duplicate of block 1, whose limit is 2: IllegalArgumentException: newPosition > limit: (3 > 2).
3. The catch clause at :350 handles only BufferUnderflowException and IndexOutOfBoundsException, so the exception reaches the caller.
4. With the in-block offset (blockOffset(3) & 7 = 1) the view would have 1 byte left, room for 0 longs, so chunk == 0 and the slow path readLong(3) would return 42.

• Exact condition, for full blocks of B bytes (every block but the last): readLongs fails iff B < 8 and (pos & 7) > B; readFloats iff B < 4 and (pos & 3) > B. It follows from :375/:379 and :362/:366, and every cell of the tested matrices agrees (the study's, output blocks of 2 to 32 bytes after 0-7 prefix bytes, re-run on 645d4fe14a20; the fact check's, output blocks of 2 to 16 bytes and List pages of 1 to 16 bytes). With 2-byte blocks readLongs fails at alignments 3-7, with 4-byte blocks at 5-7; readFloats with 2-byte blocks at alignment 3.
• This defect never returns a wrong value. For B >= 8, pos & 7 equals blockOffset(pos) & 7; single-buffer inputs have blockMask = ~0 (:65-67). For B < 8 a long view has room for 0 longs, so each read takes the slow path or throws. Every matrix cell is either correct or the exception. (Separate, and not part of this bug: the study noted, without classifying it, that the List constructor does not check the last buffer's position().)
• Legal inputs. ByteBuffersDataOutput accepts minBitsPerBlock down to LIMIT_MIN_BITS_PER_BLOCK = 1, 2-byte blocks (ByteBuffersDataOutput.java:87-88, :154). new ByteBuffersDataInput(List<ByteBuffer>) checks for a non-empty list and, with more than one buffer, a first buffer whose position() + remaining() is a power of two and intermediate buffers of that size at position() == 0 (ByteBuffersDataInput.java:50-54, :448-479); nothing rejects 1-, 2- or 4-byte pages.

• How you get tiny blocks (opt-in only): new ByteBuffersDataOutput(minBitsPerBlock, maxBitsPerBlock, ...), the "Expert" constructor, with minBitsPerBlock 1 or 2, then toDataInput(); a ByteBuffersDirectory whose output supplier makes such outputs, as Lucene's own TestMultiByteBuffersDirectory does (Math.max(LIMIT_MIN_BITS_PER_BLOCK, ...)); or new ByteBuffersDataInput(List<ByteBuffer>) with 1-, 2- or 4-byte pages.
• Defaults are safe. ByteBuffersDirectory() uses ByteBuffersDataOutput::new (ByteBuffersDirectory.java:111), whose blocks start at DEFAULT_MIN_BITS_PER_BLOCK = 10, 1 KiB; ByteBuffersDataOutput(long expectedSize) clamps to at least 10 bits (computeBlockSizeBitsFor, ByteBuffersDataOutput.java:554-565). The input's last buffer can still be shorter than 1 KiB (toBufferList flips each block, ByteBuffersDataOutput.java:236-248), and a short output gives a single-buffer input. Neither matters: with blocks of 1 KiB or more the alignment already equals the in-block offset modulo 8, and a single-buffer input has blockMask = ~0.
• With 2-byte blocks, ordinary index reads can hit it. Live docs are read with readLongs (Lucene90LiveDocsFormat.java:133, :139), and ByteBuffersIndexInput.readLongs delegates to ByteBuffersDataInput (ByteBuffersIndexInput.java:202-206). In the study class, 200 documents with one deleted in a 2-byte-block ByteBuffersDirectory: DirectoryReader.open throws IllegalArgumentException: newPosition > limit: (3 > 2) (re-run on 645d4fe14a20). Whether an index fails depends on where its live-docs longs start: here at pos & 7 = 3, so the same index with 4- and 8-byte blocks opens (numDocs 199, fact-check probe on 645d4fe14a20).
• Who in practice: the only known users are tests that use tiny blocks to exercise boundary handling, which is what the small limit was lowered for; downstream use was not surveyed beyond two code searches. Upstream's bulk-read tests never reach it: the aligned and unaligned readLongs/readFloats tests in BaseDirectoryTestCase (:208-258, :320-376) run on 1 KiB to 1 MiB chunks in the chunked directories (BaseChunkedDirectoryTestCase.java:47-48), and the two cross-boundary tests use 16- and 8-byte chunks (:364-381, :383-400).
• Effect: a loud unchecked exception; no corruption, no wrong values from this defect, nothing written. Likelihood rare; priority low.

• Public API only: the Expert ByteBuffersDataOutput constructor, writeBytes, writeLong, toDataInput, skipBytes, readLongs. No index, no randomness in the input, single thread.
• Covers readLongs through ByteBuffersDataOutput only, not readFloats or the List<ByteBuffer> route. readFloats has the same defect in getFloatBuffer and is fixed by the same diff; it is shown by the study's class and by the cross-model reviewer's own test, not by this repro. A maintainer may prefer a second, float test or one small parameterized test over both methods.
• As the regression test: this same method, for example added to TestByteBuffersDataInput. It passes with the fix.

cp $REPRO/TestByteBuffersDataInputSmallBlockReadLongs.java $LUCENE/lucene/core/src/test/org/apache/lucene/store/
cd $LUCENE
./gradlew :lucene:core:test --tests org.apache.lucene.store.TestByteBuffersDataInputSmallBlockReadLongs -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on (the same with -Ptests.asserts=false)
TestByteBuffersDataInputSmallBlockReadLongs > testReadLongsWithTwoByteBlocks FAILED
    java.lang.AssertionError: expected readLongs to return {42}, got java.lang.IllegalArgumentException: newPosition > limit: (3 > 2)
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:74129CDD307F4B10]:0)
        at org.apache.lucene.store.TestByteBuffersDataInputSmallBlockReadLongs.testReadLongsWithTwoByteBlocks(TestByteBuffersDataInputSmallBlockReadLongs.java:36)

        Caused by:
        java.lang.IllegalArgumentException: newPosition > limit: (3 > 2)
            at java.base/java.nio.Buffer.createPositionException(Buffer.java:351)
            at java.base/java.nio.Buffer.position(Buffer.java:326)
            at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1561)
            at org.apache.lucene.store.ByteBuffersDataInput.getLongBuffer(ByteBuffersDataInput.java:379)
            at org.apache.lucene.store.ByteBuffersDataInput.readLongs(ByteBuffersDataInput.java:331)
            at org.apache.lucene.store.TestByteBuffersDataInputSmallBlockReadLongs.testReadLongsWithTwoByteBlocks(TestByteBuffersDataInputSmallBlockReadLongs.java:33)

# 10.5.2 and 10.6.0, assertions on and off: the same message, at getLongBuffer:379 <- readLongs:331
# with the fix: :lucene:core:test (SUCCESS): 1 test, assertions on and off

• The size is documented and accepted. LIMIT_MIN_BITS_PER_BLOCK = 1 is javadoc'd "Smallest {@code minBitsPerBlock} allowed" (ByteBuffersDataOutput.java:87-88); the Expert constructor rejects only values below it (:154). The List constructor's javadoc asks only for "an identical remaining number of bytes in the buffer (that is a power of two)" (ByteBuffersDataInput.java:50-54), and its check (:448-479) rejects no page size under 8 bytes.
• The contract has no precondition. DataInput.readLongs: "Read a specified number of longs"; readFloats: "Reads a specified number of floats into an array at the specified offset". Their base implementations loop readLong()/readInt() (DataInput.java:150-160, :176-188). No IllegalArgumentException is documented, and the per-value reads work on the same input (study control).
• Intent when the limit was lowered. In the review of PR #360 (LUCENE-10155, 6c6a3bd5bd5, minimum lowered from 10 bits to 1), in a thread on the ByteBuffersDataOutput constructor's limits (dweiss asked why small blocks were restricted), rmuir wrote on 2021-10-07: "the defaults should be fast, but the permissible values should be ones that work correctly", and "It is also not efficient to use MMapDirectory with 4-byte memory mappings: but it works correctly and we take advantage of this in testing". Later that day, in the same thread, he wrote that the min/max constructor should be labelled "expert" and that Lucene should "reserve the right to change it in the future (for example, if we aligned our index files to at least 8 bytes, we could then make 1 << 3 the minimum size ...". So it supports "allowed values should work", not a promise that the minimum stays at 2 bytes.
• Coverage was assumed. PR #363 (LUCENE-10150, 61c15c8c10f, which added readLongs): "The method is tested well by BaseDirectoryTestCase subclass already (aligned, unaligned, etc)". Those aligned and unaligned tests (BaseDirectoryTestCase.java:208-258) run on 1 KiB to 1 MiB chunks in the chunked directories (BaseChunkedDirectoryTestCase.java:47-48), and the two cross-boundary tests on 16- and 8-byte chunks (:364-400). At every block size of 8 bytes or more absolute and in-block alignment agree, so none of them can hit it.
• The original guarded it. LUCENE-9027 (7755cdf03fc, 2019) built ByteBufferIndexInput's long views with if (i < curBuf.limit()) ... else EMPTY_LONGBUFFER and took the alignment from the position inside the current buffer.
• Same calls, other directory: MMapDirectory with 2-byte chunks returns the right values (study control, re-run on 645d4fe14a20).
• No javadoc, test, CHANGES entry or discussion accepts the exception.

Strongest case for "not a bug": the trigger is an "Expert", non-default configuration (ByteBuffersDirectory and DataInput.readLongs are even @lucene.experimental); the only known users are tests; the failure is loud, not silent corruption; no report was found in about five years; and the reviewer who lowered the limit also said Lucene may raise the minimum to 8 bytes one day. "Do not use blocks that small", or a higher LIMIT_MIN_BITS_PER_BLOCK, would be an acceptable answer.

Why it does not settle it: "Expert" and @lucene.experimental warn about tuning and API stability, not about exceptions for documented values; the limit was lowered so tests could use tiny blocks, with the stated intent that permissible values work correctly; and the code already has a slow path for a value that crosses blocks, it just never gets there. The refute-lens cross-model reviewer rated this argument weak; the second cross-model reviewer, in the fact check, rated it moderate. What it does establish: the priority is low, and "won't fix" or a higher minimum would be defensible maintainer answers.

Three options; option A was applied in a scratch worktree of main 645d4fe14a20 and run against the minimal test, every test class in org.apache.lucene.store and core's static checks.

Two lines in ByteBuffersDataInput.java (the study verifier's fix). blockOffset(pos) is what readLongs/readFloats already use to position the view (:332, :300).

--- a/lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataInput.java
+++ b/lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataInput.java
@@ -359,7 +359,7 @@
   private FloatBuffer getFloatBuffer(long pos) {
     // This creates a separate FloatBuffer for each observed combination of ByteBuffer/alignment
     int bufferIndex = blockIndex(pos);
-    int alignment = (int) pos & 0x3;
+    int alignment = blockOffset(pos) & 0x3;
     int floatBufferIndex = bufferIndex * Float.BYTES + alignment;
     if (floatBuffers[floatBufferIndex] == null) {
       ByteBuffer dup = blocks[bufferIndex].duplicate();
@@ -372,7 +372,7 @@
   private LongBuffer getLongBuffer(long pos) {
     // This creates a separate LongBuffer for each observed combination of ByteBuffer/alignment
     int bufferIndex = blockIndex(pos);
-    int alignment = (int) pos & 0x7;
+    int alignment = blockOffset(pos) & 0x7;
     int longBufferIndex = bufferIndex * Long.BYTES + alignment;
     if (longBuffers[longBufferIndex] == null) {
       ByteBuffer dup = blocks[bufferIndex].duplicate();

Trade-offs

• For blocks of 8 bytes or more, and single-buffer inputs (blockMask = ~0), the value is unchanged, so the fast path, the cached views and the results stay the same; the cost is one extra AND.
• For smaller blocks the alignment is the real in-block offset, never past the block's limit; the view then has room for 0 values and every value takes the slow path. That is slow but correct, which is what tiny blocks are for.

What the checks validated

• Scratch worktree of 645d4fe14a20, ./gradlew, OpenJDK 25.0.3: the minimal test fails on stock and passes with the fix, with assertions on and off. (The fact check re-ran the stock command in its own clone of 645d4fe14a20 with the same result.)
• Every test class in org.apache.lucene.store (seed DEADBEEF): the 28 existing classes plus the new test, 777 tests, 17 skipped, 0 failures, including TestByteBuffersDataInput, TestByteBuffersDataOutput, TestByteBuffersDirectory, TestMultiByteBuffersDirectory, TestMultiMMap and TestMMapDirectory.
• :lucene:core:check -x test passes (google-java-format, ecj lint, forbidden APIs, license checks). Not the full static-analysis suite: error-prone was skipped and eclint is not configured in that run.
• Earlier, with the fix on the main jars: the study's 7-test class passes 7 of 7, including the readFloats matrix, the List constructor, the IndexInput case and the live-docs index; the cross-model reviewer's own 5 tests and the four TestByteBuffers*/TestMultiByteBuffersDirectory suites pass (256 run, 0 failed).

In getLongBuffer/getFloatBuffer, if alignment is past the block's limit, use an empty view, the LUCENE-9027 pattern in ByteBufferIndexInput.

Trade-offs

Also sends the read to the slow path, but keeps an alignment that means nothing for small blocks and adds a branch. Not validated.

Catch IllegalArgumentException in readLongs/readFloats and fall back; or raise LIMIT_MIN_BITS_PER_BLOCK to 3 and reject List pages under 8 bytes.

Trade-offs

Catching hides other errors. Forbidding changes a documented public limit and removes the tiny-chunk coverage TestMultiByteBuffersDirectory relies on; rmuir floated an 8-byte minimum in the #360 thread, but conditional on index files being 8-byte aligned, which they are not today (the live-docs longs of the 2-byte-block index start at pos & 7 = 3). Not validated.

• Finder: the prior study's non-formal baseline, the Jazzer target DataIOFuzzer: a typed write/read round trip on ByteBuffersDataOutput/Input, ByteBuffersDirectory, MMapDirectory (16- and 64-byte chunks and the default) and NIOFSDirectory, with slices, clones, skipBytes, random access and checksums; 1.5 + 20 minutes, 278 k executions.
• Verification: an independent AI verifier wrote a from-scratch 7-test class (block-size x alignment matrices for both methods, the List constructor, a ByteBuffersIndexInput, an end-to-end live-docs index, per-value and MMapDirectory controls), derived the exact condition from the code and validated the fix.
• Minimization: cut to the one-method test (2026-10-29). Re-verified on main 645d4fe14a20 and 10.5.2 (2026-11-01), with a cross-model review.