Bulk BinaryDocValues#binaryValues sizes its buffer in int

On the default codec, the bulk read of binary doc values that is new in 10.6 puts the requested values of a dense field into one shared array whose size is computed as a Java int. One call that asks for 2 GiB or more fails; at 4 GiB or more the size can wrap and, with assertions off, the call can return other documents' bytes. CheckIndex at level 2 or 3 makes such calls itself, so it can report a healthy index as broken.

Binary doc values store one byte string per document, column-wise (BinaryDocValuesField). A field is dense when every document has a value, and fixed length when all values have the same length. The writer accepts values of up to ArrayUtil.MAX_ARRAY_LENGTH bytes each (2,147,483,631 on this JVM), and segments may hold far more than 2 GiB in total.

#16286 added a bulk read, BinaryDocValues#binaryValues(size, docs, values), new in 10.6.0. Its javadoc says the result "behaves as if implemented as below, which is the default implementation": a loop of advanceExact(doc) plus BytesRef.deepCopyOf(binaryValue()). It states no size limit.

The default codec stores doc values with Lucene90DocValuesFormat, and for dense fields it overrides the bulk method. It reads all requested values into one shared byte[], and every returned BytesRef is a window into it. The array size is a Java int (at most 2^31 - 1 = 2,147,483,647), and larger values wrap around by 2^32:

• fixed length: byte[] bulk = new byte[size * length]; (main Lucene90DocValuesProducer.java:1347), computed before the contiguity check, so non-contiguous requests use it too;
• variable length, consecutive doc ids: int totalBytes = (int) (lastEnd - firstStart); then new byte[totalBytes] (main :1393-1394);
• the window offsets use the same int arithmetic (:1352, :1356, :1399).

32 documents of 138,547,333 bytes, one call for docs 0 to 31: T = 4,433,514,656 wraps to 138,547,360, so the array holds doc 0 plus the first 27 bytes of doc 1. With assertions off the call returns normally and 31 of the 32 values differ from the per-value API. Doc 0's window is correct. Doc 31's offset, 31 x 138,547,333 = 4,294,967,323, wraps to 27: its window is in bounds and passes isValid(), but it holds doc 0's bytes from position 27 on. The other 30 windows are out of bounds. With -ea the call throws IllegalStateException: offset+length out of bounds from the BytesRef constructor's assertion.

#16286 also made CheckIndex call the bulk API (checkBulkFetchBinaryDocValues, main CheckIndex.java:4193-4234): checkIndex() runs it at level 2 or higher, and the public static CheckIndex.testDocValues runs it with no level gate. It asks in batches of up to 16 documents with gaps (0, 2, 5, 9, ...). A batch of two or more documents is never contiguous, so only fixed-length fields are affected. When one batch totals at least 2^31 - 2 bytes, CheckIndex prints test: docvalues...........ERROR [...] and "1 broken segments", and exits 1. Following its advice, -exorcise dropped the segment and every document in it.

• Who is exposed. Callers of the new bulk API on the default codec that hold the codec's own iterator (a plain DirectoryReader or SegmentReader). Wrapping readers skip the override, because FilterBinaryDocValues does not forward binaryValues (for example ExitableDirectoryReader), and no query, collector or sort in Lucene calls it. The one in-tree production caller is CheckIndex at -level 2 or -level 3; the default CLI level 1 is not affected.
• Symptoms. Usually loud: an OutOfMemoryError or NegativeArraySizeException. Silent wrong bytes need at least 4 GiB in one contiguous call with assertions off; the wrong bytes always belong to other documents in the same call, field and segment.
• CheckIndex reports a valid index as broken. It is a read-side bug: indexes written by the released 10.5.2 and 9.12.3 jars are flagged as soon as the 10.6 CheckIndex reads them at level 2. Smallest triggering shapes: a 3-to-5-document segment of equal values of at least 2^30 - 1 bytes (about 3 GiB), or 40 or more documents of at least 128 MiB (5 GiB). -exorcise then removes the segment; opening an IndexWriter afterwards deletes its files for good.
• Scale. Rare: one call must total at least 2^31 - 2 bytes, for example 2 x 1 GiB or 32,768 x 64 KiB. Typical binary values are small.
• Test suites. TestUtil.checkIndex and the BaseDirectoryWrapper check on close (both at level 3), and TestUtil.checkReader (through the ungated CheckIndex.testDocValues), reach it only when the doc values are not wrapped by the Asserting codec, whose AssertingBinaryDocValues runs the default loop. Downstream test suites with such data can fail with some random codecs; Lucene's randomized tests with the Asserting codec cannot catch it.
• Released in 10.6.0. The API first shipped in 10.6.0 (released 2026-10-29); releases 9.0.0 to 10.5.2 do not have it. The existing bulk tests use 512 documents of 8 bytes, or of 1 to 32 bytes. The large tests written for this bug (local branches, not upstream) are @Monster, which is off by default, and no in-repo workflow enables it.

A short @Monster test, a 22-line method plus one import, added to the existing TestLucene90DocValuesFormat. It writes three 1 GiB values with the default codec, merges them into one segment, and fetches docs {0, 2} (2 x 1 GiB) in one bulk call: the same documents CheckIndex asks for on a 3-document segment. It applies to main and to branch_10_6 with git apply. Use an 8g heap; a run needs about 6 GB of free disk while it merges, and a failing run leaves about 3.1 GB under lucene/core/build/tests-tmp (./gradlew :lucene:core:clean removes it).

The test assumes forceMerge(1) keeps the document order: TieredMergePolicy breaks size ties by segment name, the three 1-document segments should be the same size (not measured), and every passing run with the fix kept the order. To make it order-proof, compare each bulk value with the per-value API instead.

No test framework: write the index with the released 10.5.2 jar, then run the real CheckIndex CLI and a direct bulk call with a lucene-core jar built from branch_10_6 or main. java -Xmx8g -cp <classes>:<lucene-core-10.5.2.jar> MakeIndex <dir> 3 1073741824, then java -Xmx8g -cp <lucene-core jar> org.apache.lucene.index.CheckIndex <dir> -level 2.

import java.nio.file.Paths;
import java.util.Arrays;
import org.apache.lucene.document.BinaryDocValuesField;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.LeafReader;
import org.apache.lucene.index.SegmentReader;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.Version;

/**
 * Writes one segment with a dense, fixed-length BinaryDocValuesField "f": numDocs docs, each value
 * `length` bytes. Every byte of doc d is ('A' + d), except the last byte, which is ('a' + d).
 * Public API only, default codec, default merge policy and scheduler. Compiles against 10.5.2.
 *
 * <p>Usage: java -cp lucene-core.jar:. MakeIndex DIR NUMDOCS LENGTH
 */
public class MakeIndex {
  public static void main(String[] args) throws Exception {
    String path = args[0];
    int numDocs = Integer.parseInt(args[1]);
    int length = Integer.parseInt(args[2]);
    System.out.println("lucene=" + Version.LATEST + " numDocs=" + numDocs + " length=" + length);
    byte[] bytes = new byte[length];
    try (Directory dir = FSDirectory.open(Paths.get(path));
        IndexWriter w = new IndexWriter(dir, new IndexWriterConfig())) {
      for (int d = 0; d < numDocs; d++) {
        Arrays.fill(bytes, (byte) ('A' + d));
        bytes[length - 1] = (byte) ('a' + d);
        Document doc = new Document();
        doc.add(new BinaryDocValuesField("f", new BytesRef(bytes)));
        w.addDocument(doc);
        w.flush();
      }
      w.forceMerge(1);
      w.commit();
    }
    try (Directory dir = FSDirectory.open(Paths.get(path));
        DirectoryReader r = DirectoryReader.open(dir)) {
      LeafReader leaf = r.leaves().get(0).reader();
      System.out.println(
          "leaves="
              + r.leaves().size()
              + " numDocs="
              + r.numDocs()
              + " codec="
              + ((SegmentReader) leaf).getSegmentInfo().info.getCodec().getName()
              + " dvFormat="
              + leaf.getFieldInfos().fieldInfo("f").getAttribute("PerFieldDocValuesFormat.format"));
    }
  }
}

java -Xmx10g -cp <classes>:<lucene-core jar> BulkCall <dir> 0,2 0,1; for the controls, add -ea, or put default as the first argument, before <dir>.

import java.nio.file.Paths;
import org.apache.lucene.index.BinaryDocValues;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.FilterBinaryDocValues;
import org.apache.lucene.index.LeafReader;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.Version;

/**
 * Opens the index written by MakeIndex and makes ONE bulk BinaryDocValues#binaryValues call per
 * doc-id list given on the command line (e.g. "0,2" "0,1,2"). Each returned value is compared with
 * the per-value API (advanceExact + binaryValue) on a second iterator. With the first argument
 * "default", the iterator is wrapped in a FilterBinaryDocValues, which does not override
 * binaryValues, so the documented default implementation runs instead (control).
 *
 * <p>Usage: java -cp lucene-core.jar:. BulkCall [default] DIR DOCS...
 */
public class BulkCall {
  public static void main(String[] args) throws Exception {
    int a = 0;
    boolean useDefault = args[0].equals("default");
    if (useDefault) a++;
    String path = args[a++];
    boolean ea = false;
    assert ea = true;
    System.out.println(
        "lucene=" + Version.LATEST + " assertions=" + ea + " mode=" + (useDefault ? "default-impl" : "codec-bulk"));
    try (Directory dir = FSDirectory.open(Paths.get(path));
        DirectoryReader r = DirectoryReader.open(dir)) {
      LeafReader leaf = r.leaves().get(0).reader();
      for (; a < args.length; a++) {
        String[] parts = args[a].split(",");
        int[] docs = new int[parts.length];
        for (int i = 0; i < docs.length; i++) docs[i] = Integer.parseInt(parts[i]);
        BinaryDocValues bulk = leaf.getBinaryDocValues("f");
        if (useDefault) bulk = new FilterBinaryDocValues(bulk) {};
        BytesRef[] values = new BytesRef[docs.length];
        long requested = 0;
        BinaryDocValues single = leaf.getBinaryDocValues("f");
        for (int d : docs) {
          single.advanceExact(d);
          requested += single.binaryValue().length;
        }
        String head = "binaryValues(" + docs.length + ", {" + args[a] + "}) requested=" + requested + " B";
        try {
          bulk.binaryValues(docs.length, docs, values);
        } catch (Throwable t) {
          StackTraceElement top = t.getStackTrace().length > 0 ? t.getStackTrace()[0] : null;
          System.out.println(head + " -> THREW " + t + (top == null ? "" : "  at " + top));
          continue;
        }
        System.out.println(head + " -> returned normally");
        BinaryDocValues check = leaf.getBinaryDocValues("f");
        int wrong = 0;
        for (int i = 0; i < docs.length; i++) {
          BytesRef v = values[i];
          check.advanceExact(docs[i]);
          BytesRef expected = check.binaryValue();
          String valid;
          try {
            v.isValid();
            valid = "true";
          } catch (Throwable t) {
            valid = "false (" + t.getClass().getSimpleName() + ")";
          }
          String same;
          try {
            same = String.valueOf(expected.bytesEquals(v));
          } catch (Throwable t) {
            same = "n/a (" + t.getClass().getSimpleName() + ")";
          }
          String last;
          try {
            last = String.valueOf((char) v.bytes[v.offset + v.length - 1]);
          } catch (Throwable t) {
            last = t.getClass().getSimpleName();
          }
          if (!"true".equals(same)) wrong++;
          if (docs.length <= 8 || !"true".equals(same) && (i < 3 || i >= docs.length - 3)) {
            System.out.println(
                "  doc "
                    + docs[i]
                    + ": offset="
                    + v.offset
                    + " length="
                    + v.length
                    + " bytes.length="
                    + v.bytes.length
                    + " isValid="
                    + valid
                    + " equalsPerValue="
                    + same
                    + " lastByte="
                    + last
                    + " (expected "
                    + (char) expected.bytes[expected.offset + expected.length - 1]
                    + ")");
          }
        }
        System.out.println("  values differing from the per-value API: " + wrong + " of " + docs.length);
      }
    }
  }
}

# branch_10_6 d04ee73177, seed DEADBEEF
TestLucene90DocValuesFormat > testBinaryValuesBulkFetchOver2GB FAILED
    java.lang.NegativeArraySizeException: -2147483648
        at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$16.binaryValues(Lucene90DocValuesProducer.java:1350)
        at org.apache.lucene.index.BinaryDocValues.binaryValues(BinaryDocValues.java:84)
        at org.apache.lucene.codecs.lucene90.TestLucene90DocValuesFormat.testBinaryValuesBulkFetchOver2GB(TestLucene90DocValuesFormat.java:394)
# main b83101cdd2: the same exception at Lucene90DocValuesProducer.java:1347

# CheckIndex CLI on 3 x 1 GiB written by 10.5.2, branch_10_6 jar, -level 2 (exit status 1); excerpt
    test: docvalues...........ERROR [-2147483648]
WARNING: 1 broken segments (containing 3 documents) detected
# 10.5.2's CheckIndex on the same index: docvalues OK, no problems detected

• The contract is the per-value loop. The javadoc (BinaryDocValues.java:39-82) defines the result as the default loop and states only that doc ids are ascending with no duplicates. That loop, run through FilterBinaryDocValues, returns correct values in every failing case, and so does the per-value API on the same documents.
• No documented limit. The writer accepts each value up to MAX_ARRAY_LENGTH, and Lucene90 documents no read-side limit. The expected objection is DocValuesType.BINARY's "different codecs may enforce their own limits"; Lucene90 enforces its limit at write time, on single values, and the same values read back correctly one at a time.
• History shows an accident, not a design. The PR's first codec commit (5d5ad9b85d, 2026-06-23) allocated one array per value and could not overflow. After a reviewer asked on 2026-06-26 for something more compact than BytesRef[], commit d26fd36beb "Improve SIMD implementation by reducing allocations" added new byte[size * length] and the (int) cast. No comment or review in the PR mentions a size limit.
• The numeric sibling has a guard. NumericDocValues#longValues (#16129, same author, shipped in 10.5.0) computes the byte count as a long and falls back to the default when it exceeds Integer.MAX_VALUE (main Lucene90DocValuesProducer.java:676-679), and #16286's description opens with "Mirrors NumericDocValues#longValues for binary fields".
• Lucene's own caller cannot avoid it. CheckIndex chooses the batch size, and it reports as broken an index that 10.5.2 reports clean.

The intended-behaviour checker argued the other side. The best case for "intended or acceptable": Java arrays are int-sized; this is a performance API where the caller picks size (the PR benchmarked 8-128 byte values in batches of 1024); DocValuesType.BINARY allows codec limits; SECURITY.md makes limits on unbounded input the caller's job; and the API was still unreleased then (10.6.0 has since shipped it).

Why that case fails: the javadoc contract is the per-value loop with no size precondition; Lucene90 documents no read limit and accepts the values at write time; CheckIndex picks the batch, so a -level 2 user cannot avoid it; calls of 4 GiB or more silently return wrong bytes, which no reading of the contract allows; and the int buffer arrived with a review-time allocation optimization. Verdict: not intended and not documented; a real bug that is hard to reach. Not a security issue: the size is chosen by the caller and the JVM's bounds checks hold, so a public issue is the right channel.

The checks validated the guard with a per-value fallback; the alternatives come from the brief's list for reviewers.

Local branches r106-bulk-binaryvalues-2gib-fix (f0f3745a2c, branch_10_6) and -main-fix (b9e0701027, main): 10 lines plus one import. super is the documented default loop; it leaves doc on the last requested document, as the bulk path and AssertingLeafReader expect. After the guards, the window offsets cannot wrap.

if ((long) size * length > ArrayUtil.MAX_ARRAY_LENGTH) {          // fixed length, before new byte[]
  // too many bytes for a single array
  super.binaryValues(size, docs, docsOffset, values, valuesOffset);
  return;
}
...
if (lastEnd - firstStart > ArrayUtil.MAX_ARRAY_LENGTH) {          // variable length, before the cast
  // too many bytes for a single array
  super.binaryValues(size, docs, docsOffset, values, valuesOffset);
  return;
}

Trade-offs

• Behaviour changes only when one call totals more than MAX_ARRAY_LENGTH bytes. Between 2,147,483,632 and 2,147,483,645 bytes the old code worked; the fix now takes the slower, correct per-value path there.
• Cost: one long multiply and compare per call (SECURITY.md asks that robustness fixes not slow hot paths). The fallback allocates the same total bytes, one array per value.
• The CHANGES entry has GITHUB#TBD placeholders and says "2GB": write "2 GiB" or name the real cap. Open PR #16832 edits the same variable-length loop and keeps the int cast; the two patches apply in either order. #16896 fixes a different overflow in the same file.

What the checks validated

• branch_10_6 plus fix: the minimal test, the fuller tests 2/2, and org.apache.lucene.codecs.lucene90.* with TestCheckIndex, TestBinaryDocValuesUpdates and TestDocValues: 844 tests, 0 failures (re-run by the code-history checker).
• main plus fix: the minimal test with -ea and -da; the three monster methods; 861 tests, 0 failures; :lucene:core:check -x test passes.
• Both: CheckIndex levels 1-3 clean; direct bulk calls correct; the wrap case has 0 of 32 wrong with and without -ea; the variable-length cases are correct.

Fall back earlier, to avoid single arrays of nearly 2 GiB.

Trade-offs

More calls take the per-value path; the cut-off is a judgement call.

Keep the bulk read but split it across arrays that each stay under the limit.

Trade-offs

More code in a hot path; the returned windows then point into several arrays.

Non-contiguous fixed-length requests would read value by value without the shared array.

Trade-offs

A complement to the guard, not a fix on its own: a contiguous request over 2 GiB still overflows.

Mirror the numeric sibling's guard exactly.

Trade-offs

Leaves the 2-byte band (2^31 - 2 and 2^31 - 1) where the JVM refuses the array with an OutOfMemoryError, so MAX_ARRAY_LENGTH is safer.

Method: a CodeQL inventory of integer arithmetic, then bit-vector SMT checks of the transcribed method (the study's "formal" tier: the solver's counterexample came first). Run 3 of the study, experiment "overflow-at-scale" (2026-10-22, main 7cecf79523):

1. An AI tool built a CodeQL 2.27.3 inventory of non-constant int multiplications and left shifts in a list of packages that includes codecs/lucene90: 602 rows. The query also includes x * C and drops products that CodeQL's range analysis can bound.
2. It ranked the rows by git blame age. The row Lucene90DocValuesProducer.java:1347 binaryValues array-size dated from 2026-06-30, about four months old.
3. It transcribed the method into Z3 4.12.2 bit-vector obligations: B1 "the int product is exact" gave a counterexample; B1w, the witness size = 2, length = 2^30, gives a negative size; B2, the variable-length narrowing cast (not a CodeQL row; the AI tool added it while transcribing), gave a counterexample. cvc5 1.4.2 later returned sat on all three.
4. It wrote a JUnit repro on main (2 x 1 GiB). Order by file times (UTC): CodeQL 05:35, blame 05:43, first Z3 run 05:57, JUnit repro 06:14, final Z3 run and cvc5 06:59.

That AI tool believed, wrongly, that CheckIndex reaches only the per-document branch. A separate AI verifier, given the candidate, reproduced it with its own tests, overturned that claim and found the CheckIndex false positive, the -exorcise data loss and the 40 x 128 MiB threshold. The run-3 task text named the area ("integer arithmetic that only breaks at scale") and listed the doc-values writer, not the bulk API or the reader. Z3 checked the transcribed arithmetic; the test shows the bug itself, so it is not "formally proven".