ByteBuffersDataOutput.copyBytes narrows its length to int: a copy of 2 GiB or more silently copies nothing or only part

ByteBuffersDataOutput.copyBytes(DataInput input, long numBytes) begins with int length = (int) numBytes; (ByteBuffersDataOutput.java:314 on main 645d4fe14a20). Both the loop that reads straight into the block arrays and the fallback to the long-safe base method run only while that int is positive, and the only check, assert numBytes >= 0, tests the long. So the method copies max(0, (int) numBytes) bytes and returns normally: every length up to Integer.MAX_VALUE is correct, every length from 2^31 to 2^32-1 copies nothing, and 2^32+5 copies 5 bytes. The rest of the input is left unread.

ByteBuffersIndexOutput.copyBytes delegates to it, and ByteBuffersDirectory.copyFrom (from BaseDirectory) copies a file with os.copyBytes(is, is.length()). A file of 2 GiB or more copied into a ByteBuffersDirectory therefore comes out empty or short, with no error. Through IndexWriter.addIndexes(Directory...) into a ByteBuffersDirectory, the copy and commit() succeed and the next DirectoryReader.open throws CorruptIndexException. The override came with LUCENE-10657 (#1034) in 9.3.0 as an optimisation; 9.2.0 copied correctly. Every release from 9.3.0 to 10.6.0 and main is affected; the default FSDirectory outputs are not. The fix is two lines.

1. The narrowing. store/ByteBuffersDataOutput.java:311-332: int length = (int) numBytes; (:314), then while (length > 0) { ... } (:315) copies chunk by chunk straight into the current block's array, and if (length > 0) super.copyBytes(input, length); (:329-331) handles blocks without an array. When (int) numBytes <= 0 both are skipped. The only check, assert numBytes >= 0 (:313), tests the long.
2. What it overrides. DataOutput.copyBytes counts down a long left in 16 KiB chunks and is correct for any length (store/DataOutput.java:280-293). ByteBuffersDataOutput inherited it until 9.3.0.
3. How users reach it. ByteBuffersIndexOutput.copyBytes delegates (store/ByteBuffersIndexOutput.java:139-143). ByteBuffersDirectory extends BaseDirectory (store/ByteBuffersDirectory.java:50), whose copyFrom calls copyThroughCreateOutput (store/BaseDirectory.java:54-58), which does os.copyBytes(is, is.length()) (store/Directory.java:189-197, line 193). IndexWriter.addIndexes(Directory...) (index/IndexWriter.java:3103) copies each segment file with directory.copyFrom in copySegmentAsIs (:3648, :3692); that directory is the writer's LockValidatingDirectoryWrapper (:958), whose copyFrom checks the lock and delegates (store/LockValidatingDirectoryWrapper.java:47-51). Wrapped destinations reach the same method too (code reading, not run): FilterDirectory.copyFrom goes through its own createOutput (store/FilterDirectory.java:126-128), FileSwitchDirectory.copyFrom delegates to the directory chosen for the file (store/FileSwitchDirectory.java:241-244), and NRTCachingDirectory, a FilterDirectory, writes into an internal ByteBuffersDirectory when the IOContext size estimate fits both maxMergeSizeMB and maxCachedMB (store/NRTCachingDirectory.java:72-84, :155-165, :231-245; for addIndexes the estimate is the whole segment, index/IndexWriter.java:3174-3175), so it needs both limits set above the segment's size.
4. Not affected. Outside test code, the only copyBytes(DataInput, long) overrides in Lucene are this one, the delegating ByteBuffersIndexOutput and EndiannessReverserDataOutput (backward-codecs), and the base method; the FSDirectory family writes through outputs that inherit the long-safe base method. Copying from a ByteBuffersDirectory to disk is fine.

Through addIndexes (executed on main ea57360912ed, default settings): a source index whose _0.fdt is 2,169,703,755 bytes; addIndexes and commit() return normally; the copied _0.fdt has length 0; DirectoryReader.open throws CorruptIndexException (cause EOFException); CheckIndex reports open reader.........FAILED. The source index is not modified.

• Who is hit. A direct copyBytes call of 2^31 bytes or more on a ByteBuffersDataOutput or ByteBuffersIndexOutput; Directory.copyFrom of a file of 2 GiB or more into a ByteBuffersDirectory; IndexWriter.addIndexes(Directory...) with a ByteBuffersDirectory as the writer's directory and a source file of 2 GiB or more. That is the "load an on-disk index into RAM" pattern; LUCENE-8624 (#9670) was reported by a user indexing large data sets with ByteBuffersDirectory.
• Default vs opt-in. Opt-in: the default directory (FSDirectory.open) and copies from a ByteBuffersDirectory to disk are not affected. Nothing else needs to be configured once the destination is heap-based.
• Scale. One copy of 2^31 bytes or more; every smaller copy is correct.
• What goes wrong. Deterministic and silent at copy time, with assertions on and off. A directly copied stream or a copied non-index file is empty or truncated with no error. A copied index commits normally and fails only at the next DirectoryReader.open, away from the copy that caused it. The source is never modified, and on this path search results are never silently wrong.
• Likelihood: contrived. ByteBuffersDirectory is @lucene.experimental; its javadoc says MMapDirectory "is nearly always a better choice" and a heap-based directory can have the advantage for "ephemeral, small, short-lived indexes" (store/ByteBuffersDirectory.java:43-48); lucene/MIGRATE.md says it is "discouraged in favor of the default MMapDirectory". No public report was found in about four years of affected releases.
• Not claimed: a second caller. The default terms-index builder TrieBuilder.append (10.5.0 and later) bulk-copies a sub-trie buffer with this method (codecs/lucene103/blocktree/TrieBuilder.java:153-156). At unit level (package-private API), appending a 2,176,663,776-byte sub-trie kept 3 of 68,003 entries, a 32 MB control kept all, and the fix kept all 68,003. It was not reproduced through IndexWriter, so it is not part of this report; the same two-line fix covers it.

The length is the smallest failing one, 2^31.

As a regression test it has one cost: once fixed, the copy really holds 2 GiB, so the test runs out of memory with Lucene's default 512 MB test heap (executed) and needs a larger heap (-Ptests.heapsize=4g) or the @Monster annotation that Lucene's 2B tests use.

TestByteBuffersDataOutputCopyBytes2GiB > testCopyBytesCopiesAllBytes FAILED
    java.lang.AssertionError: expected:<2147483648> but was:<0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:F37EE34A02796FF1]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.store.TestByteBuffersDataOutputCopyBytes2GiB.testCopyBytesCopiesAllBytes(TestByteBuffersDataOutputCopyBytes2GiB.java:26)

:lucene:core:test (FAILURE): 1 test, 1  failure

# the same failure with -Ptests.asserts=false, and with a standalone JUnit runner on main and 10.6.0 (assertions on and off) and 9.3.0
# 9.2.0: 1 test, 0 failures (a real 2 GiB copy)

• The contract. DataOutput.copyBytes: "Copy numBytes bytes from input to ourself." (DataOutput.java:280), with a long parameter and a base loop on a long (:281-293). Directory.copyFrom: "Copies an existing {@code src} file from directory {@code from} to a non-existent file {@code dest} in this directory." (Directory.java:174-175). Neither states a size limit.
• The class handles more than 2 GiB. ByteBuffersDataOutput.size() returns a long (:349); toArrayCopy() documents and throws a RuntimeException when "the {@link #size()} of the underlying buffers exceeds maximum size of Java array" (:285-298).
• A regression of an optimisation. LUCENE-10657 (#1034, CHANGES 9.3.0 "Optimizations": "CopyBytes now saves one memory copy on ByteBuffersDataOutput") added the override; its issue names IndexWriter#copySegmentAsIs -> copyFrom -> copyBytes, a whole-file copy, as a target. The review covered the fallback for non-heap buffers and arrayOffset, not the length type; its two tests copy about 8 KiB. 9.2.0, which inherits the base method, copies 2^31 bytes correctly (executed).
• The sibling fix. In #1047 (LUCENE-10661, the same author's BytesStore.copyBytes override, a week later), reviewer jpountz wrote "We could make len a long and avoid the unchecked cast, couldn't we?", and BytesStore got a long len with a per-chunk (int) Math.min(...), the fix proposed here. ByteBuffersDataOutput was not revisited.
• Precedent. LUCENE-8624 (#9670, an int overflow in ByteBuffersDataOutput.size()) and LUCENE-8625 (#9671, ByteBuffersDataInput.sliceBufferList) were fixed as bugs in 7.7.0 (CHANGES 7.7.0 "Bug fixes"); LUCENE-8628 (#9674) was closed as a duplicate report of the same overflow.
• No test tolerates it. We found no upstream test that copies 2^31 bytes or more. The two tests added with the override copy about 8 KiB (TestByteBuffersDataOutput.java:201-233). BaseDirectoryTestCase, which also runs for ByteBuffersDirectory (TestByteBuffersDirectory.java:34), copies files of at most 1,777,777 bytes, in chunks of up to 77,777 bytes (testCopyBytes, BaseDirectoryTestCase.java:799-837).

The strongest case: ByteBuffersDirectory is experimental and its javadoc points to "ephemeral, small, short-lived indexes", MIGRATE.md discourages it in favour of MMapDirectory, nobody should hold 2 GiB files in heap buffers, and an index copied this way does fail loudly when it is opened.

Why it fails: that is usage advice, not a documented size precondition, and it says nothing about ByteBuffersDataOutput, a public class that is not marked experimental. The method keeps a long signature and returns normally after copying nothing, where 9.2.0 did the copy. "Fails at open" covers only index files, and only later; direct callers and other files get no error at all. The cross-model reviews rated the "intended" case weak, and the devil's advocate verifier and the adjudication rejected it.

The recommended fix was run in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF), removed afterwards. Options B and C were not run.

Each chunk is at most currentBlock.remaining(), an int, so the per-chunk cast is safe, and the fallback super.copyBytes(input, length) already takes a long. This is the pattern BytesStore got in #1047, and the same two lines as the finder's and the independent reproducer's fixes. The study's Z3 model of the copy count does not encode the loop, so it adds nothing here beyond "a long count does not narrow"; the evidence for the fix is the code and the executed tests.

@@ -314 +314 @@ public void copyBytes(DataInput input, long numBytes)
-    int length = (int) numBytes;
+    long length = numBytes;
@@ -322 +322 @@
-      int chunk = Math.min(currentBlock.remaining(), length);
+      int chunk = (int) Math.min(currentBlock.remaining(), length);

Trade-offs

No change for copies below 2 GiB (same code path); copies of 2 GiB or more now really copy and need the heap for it. The one-copy optimisation is kept. Not fixed by it (code reading, not run): a ByteBuffersDirectory built with the non-default OUTPUT_AS_ONE_BUFFER (also OUTPUT_AS_BYTE_ARRAY) factory still cannot keep a file of 2 GiB or more, because closing the output calls toArrayCopy(), which throws (ByteBuffersDirectory.java:64-81, :264-266; ByteBuffersDataOutput.java:285-298). That failure is loud; the default OUTPUT_AS_MANY_BUFFERS (ByteBuffersDirectory.java:111) has no such limit.

What the checks validated

Executed in a scratch worktree of 645d4fe14a20 with gradle:

• the minimal test: stock 1 of 1 fails with assertions on and off; with the fix it passes with assertions on and off (-Ptests.heapsize=4g, 1.8 s and 1.7 s); with the default 512 MB test heap the fixed copy runs out of memory, because it now really holds 2 GiB;
• org.apache.lucene.store.* and TestTrie: 780 tests, 18 skipped, 0 failures;
• :lucene:core:check -x test: passes (google-java-format, ecjLint, forbiddenApis).

Earlier runs of the same two lines: the independent reproducer's 9 tests pass, including a real 2^31+7-byte copyFrom into a ByteBuffersDirectory with content checks (re-run by the adjudication); the 16 existing TestByteBuffersDataOutput tests pass; the unit-level trie probe keeps all 68,003 entries. These are targeted runs, not the full Lucene test suite.

if (numBytes > Integer.MAX_VALUE) { super.copyBytes(input, numBytes); return; } before the loop.

Trade-offs

Correct by construction (the base method is long-safe) and small, but copies of 2 GiB or more lose the one-copy optimisation, and the int arithmetic stays for the next reader to wonder about.

int length = Math.toIntExact(numBytes); throws ArithmeticException for 2^31 or more.

Trade-offs

Removes the silent part (the adjudication listed it as acceptable), but rejects copies that 9.2.0 performed and that the class otherwise supports, so copyFrom of a large file into a ByteBuffersDirectory would still not work.

Method: non-formal (code reading), with an SMT model used to characterise the inputs, not to find the bug.

• The finder reviewed ByteBuffersDataOutput as part of a hunt unit on DataInput/DataOutput implementations and strings, and noticed that the copyBytes override narrows its long length.
• A Z3 bit-vector model of the copy count (5 queries) encodes the override as copying max(0, (int) numBytes) bytes. In that model it found the minimal failing length 2^31, showed every length up to Integer.MAX_VALUE correct, found lengths in [2^31, 2^32) that copy nothing (every such length casts to a negative int) and larger lengths that copy part of the request. Its fix query only confirms that a long count has no narrowing: the model does not encode the copy loop, so the fix rests on the code (each chunk is at most remaining()) and the executed tests.
• A public-API JUnit on the stock jar (a direct copy, and ByteBuffersDirectory.copyFrom of a sparse 2^31+7-byte file) failed on main with assertions on and off. A cross-model reviewer, an independent reproducer, a second cross-model reviewer, a devil's advocate verifier and an adjudication then confirmed it, and the repro was later cut to the one test above.