Opt-in native SVE dot product crashes the JVM for byte arrays near 2^31 elements: its int32 loop index overflows

Lucene main can compute the byte dot product in C instead of Java: you compile the example lucene/misc/src/c/dotProduct.c into a library and pass -Dlucene.useNativeDotProduct=true. On a CPU with SVE, a library compiled with SVE enabled runs vdot8s_sve, which keeps its loop index i and the vector length in int32_t. Its loop test i + 4 * vec_length <= limit overflows once i + 4*VL passes INT32_MAX (VL = SVE vector length in bytes; at the tested VL = 32 and 16 that is at i = 2^31 - 4*VL): signed overflow, undefined behaviour in C.

In the GCC-built library the 32-bit bound wraps negative, so the test stays true, while the loads use a 64-bit offset that runs past the end of both arrays until an unmapped page. VectorUtil.dotProduct(byte[], byte[]) on an all-zero array of 2^31 - 64 elements, expected 0, kills the JVM with SIGSEGV every time in the GCC 10.5 builds tested, with assertions on and off. At the two vector lengths tested, every length from 2,147,483,520 (256-bit SVE) or 2,147,483,584 (128-bit SVE) up to the JVM maximum crashes; one less returns the exact result.

Only applications that enable the opt-in library on Lucene main (unreleased 11.0) and pass byte arrays of about 2 GiB themselves are hit; indexed vectors (at most 1,024 dimensions) and kNN cannot reach it, and default Lucene and every release are unaffected. Widening the index to int64_t and using svwhilelt_b8_s64 fixes it; the NEON kernel has the same source pattern, compiled correctly by GCC 10.5, and the same fix covers it.

1. Java side (correct). VectorUtil.dotProduct (VectorUtil.java:251-256) checks only that the lengths are equal. With the flag, the provider is NativeVectorizationProvider (Constants.java:100-104, VectorizationProvider.java:168-170), and NativeVectorUtilSupport.dotProduct(byte[], byte[]) (:419-423, :303-306) passes the element count as the C limit (invokeIntMethodHandle, :246-252).
2. Dispatch. resolve_dot8s (dotProduct.c:181-196) returns vdot8s_sve when the CPU reports SVE and the library was compiled with SVE enabled (#if defined(__ARM_FEATURE_SVE), :32, :185); lucene_dotProduct is bound to it through ifunc (:199-201). The resolver exists only on Linux aarch64; elsewhere lucene_dotProduct is the scalar loop (:202-206).
3. The defect. vdot8s_sve(int8_t vec1[], int8_t vec2[], int32_t limit) (:50) declares int32_t i (:52) and int32_t vec_length = svcntb() (:63), and loops for (i = 0; i + 4 * vec_length <= limit; i += 4 * vec_length) (:66) with eight full-vector loads per step (:70-80); the tail uses svwhilelt_b8_s32 (:90, :96).
4. The numbers (VL = 32 bytes, length 2,147,483,584). i runs 0, 128, 256, ...; at i = 2,147,483,520 the test computes i + 128 = 2^31, one more than the largest int32_t. Correct code would leave the main loop here and finish in the tail.
5. The compiled loop (GCC 10.5 -O3). The bound sits in a 32-bit register with a signed compare (cmp w2, w5 / b.ge), the loads use a 64-bit offset (ld1b ... [x0, x9], add x9, x9, x17). The bound wraps to a negative number, which is <= limit. When 4*VL divides 2^31 (any power-of-two VL, including the tested 32 and 16) every later wrapped bound is <= limit too (negative until the offset reaches 2^32 - 4*VL, then 0, 4*VL, ...), so the loop can end only by faulting.
6. At the fault: limit R2 = 0x7fffffc0 (2,147,483,584), load offset R9 = 0xbfbfff80 (1,069,547,456 bytes past the end), the 32-bit bound R5 = 0xbfc00080 (negative as int32_t), SEGV_MAPERR at 0x800000000, the end of the 3 GiB heap reservation.

At the tested vector lengths, every length n >= 2^31 - 4*VL up to the JVM's largest byte[] (2,147,483,645 on JDK 25): 2,147,483,520 to 2,147,483,645 at 256-bit SVE (126 lengths) and 2,147,483,584 to 2,147,483,645 at 128-bit SVE (62 lengths); 2,147,483,519 and 2,147,483,583 return the exact value. Both windows start below Lucene's own ArrayUtil.MAX_ARRAY_LENGTH (2,147,483,631 with compressed oops). The formula holds whenever 4*VL divides 2^31; for any other VL the first overflowing length would be ceil(2^31 / (4*VL)) * 4*VL - 4*VL (arithmetic, not run).

vdot8s_neon has int32_t i (:123) and i + 64 <= limit (:127): the same signed overflow in the source for lengths from 2^31 - 64. GCC 10.5 compiles it to a counted loop, and a NEON-only build returned the exact value at 2,147,483,631, so nothing was observed there. The scalar fallback tests i < limit and cannot overflow.

• Reachable through VectorUtil.dotProduct(byte[], byte[]) and its callers: VectorUtil.dotProductScore, byte VectorSimilarityFunction.DOT_PRODUCT.compare (through dotProductScore) and MAXIMUM_INNER_PRODUCT.compare, and the queries module's ByteVectorSimilarityFunction over two constant vectors. Use dotProduct itself as the oracle: dotProductScore is already wrong at these lengths for a separate reason (the hunt's LFV-143).
• Not reachable with default Lucene (no flag), on any release (none has the native provider), on CPUs without SVE (the NEON build was correct with GCC 10.5), or through indexing and kNN: every stock vectors format caps dimensions at 1,024, about two million times below the trigger. Lucene's own off-heap byte scorers do call the native kernel, but one stored vector per call (getSegment slices one vector), so they do not widen the reach.
• Effect: a deterministic process crash (SIGSEGV), with assertions on and off; not a catchable exception, no wrong result, no write. How far past the arrays the kernel reads before it faults depends on the memory layout (about 1 GiB in this doc's run, about 80 MB in a run with a 6 GB heap). The minimal repro passes one array twice; the independent reproducer's and the devil's advocate's programs pass two distinct arrays and crash the same way.
• Why fix it anyway: it is the example the #15508 discussion tells users to compile, the only implementation of lucene_dotProduct in the repository, and undefined behaviour that another compiler may treat differently; the fix leaves results unchanged below the threshold (350 tests), a quick C microbenchmark showed no measurable speed difference at 64 to 4,096 bytes (not benchmarked through Lucene), and it can land before 11.0.0.

2^31 - 64 is the smallest length that crashes at both 256-bit and 128-bit SVE; one all-zero array passed twice keeps the memory at 2 GiB and the exact result at 0, so no accumulator overflow is involved. Deterministic, single thread, no fill loop. Every run built the library with GCC 10.5.0 and -march=native on an SVE CPU, which enables SVE (GCC 7.3 rejects the file's NEON target attribute; other compilers not checked). As a regression test it is the same program; Lucene's CI does not build the native library, so it fits a manual or native-mode check.

# stock library, main 645d4fe14a20, -da (-ea identical apart from pc and pid)
INFO: Native vectorization enabled; uses preferredBitSize=256; FMA enabled; strict mode=false; native library loaded=true, using NativeVectorizationProvider;
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000ffff261ee920, pid=684, tid=707
#
# Problematic frame:
# C  [libdotProduct.so+0x920]  vdot8s_sve+0x80
#
# hs_err: si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000800000000
#         R2=0x7fffffc0 (limit)  R9=0xbfbfff80 (64-bit load offset)  R5=0xbfc00080 (32-bit bound)
# Java frames: VectorUtil.dotProduct([B[B) <- NativeVectorUtilSupport.dotProduct([B[B) <- invokeIntMethodHandle <- downcall
# exit status 134

# fixed library (int64_t loop index), same program, -ea and -da
INFO: Native vectorization enabled; ... native library loaded=true, using NativeVectorizationProvider;
# exit status 0

• The contract. VectorUtil.dotProduct(byte[], byte[]): "Dot product computed over signed bytes", "@return the value of the dot product of the two vectors"; the only check is equal lengths (VectorUtil.java:244-256). dotProduct.h:18-23: "Each function must return what the VectorUtilSupport method it replaces returns"; n "is the number of elements of its FIRST array argument", with no upper bound. That header text came with #16812 (2026-10-12); #15508 added the header with only the bare declaration, so it is the contract of the current code, not one stated when the feature landed.
• It is the default users are pointed to. In #15508 the PR author, shubhamvishu, proposed that users "optionally generate the binary on their end using gcc -shared -O3 -march=native -funroll-loops ..." and "could either use the default dotProduct.c file" or write their own. #16812 calls it "the example library in lucene/misc/src/c".
• Legal input. Two equal-length arrays below Lucene's own ArrayUtil.MAX_ARRAY_LENGTH, passed to a public method.
• Nothing chooses it. No javadoc, header comment, CHANGES or MIGRATE entry, test or commit message limits the length; the review threads of #15508 and #13572 never mention overflow, 2^31 or a 64-bit index. No Lucene test runs the native path in CI (#15840), and TestVectorUtilSupport stops at 4,098 elements; it passes with the stock library.
• Precedent. #16812 (merged 2026-10-12) fixed a different SIGSEGV in this native path, on the Java side, with no "example code" objection.

The strongest case for "not a bug": dotProduct.c:18 says "EXAMPLE CODE FOR DOT PRODUCT IMPLEMENTATION IN C"; Lucene does not build, ship or CI-test it; the provider is opt-in and @lucene.experimental; the org.apache.lucene.util package says its APIs are "primarily internal"; every in-tree caller passes dimensions capped at 1,024; and the Java byte path itself goes wrong far below this length in other ways (its int accumulator from about 2^17 saturated dimensions, dotProductScore from 65,536).

Why it fails: the header contract is explicit and unbounded; the input is legal and within ArrayUtil.MAX_ARRAY_LENGTH; in the repro the exact result is 0 and the Java providers return it, so no accumulator wrap is involved; Java wraps in a defined, memory-safe way, while the C kernel's undefined behaviour reads past both arrays and kills the JVM; the file is presented as the default users compile; and upstream fixed the same kind of failure here before. The example-code and scale points make it a low-priority hardening fix; they do not change the verdict. The independent reproducer, the devil's advocate itself, the adjudication and the cross-model reviewer all rejected the argument.

The hunt's candidate fix, unchanged, applied with git apply in a scratch worktree of 645d4fe14a20 and built with GCC 10.5.0; the repro ran on the 645d4fe14a20 jars (the fix touches only C) and the closest suites through gradle with the native provider.

In vdot8s_sve: int64_t i (:52), int64_t vec_length (:63), svwhilelt_b8_s64 (:90, :96). In vdot8s_neon: int64_t i (:123). limit stays int32_t, as declared in dotProduct.h:28; the compiled bound compare becomes 64-bit (cmp x2, x6) and the tail uses whilelt p1.b, x3, x2.

-    int32_t i = 0;
+    int64_t i = 0;
-    int32_t vec_length = svcntb();
+    int64_t vec_length = svcntb();
-    svbool_t pred = svwhilelt_b8_s32(i, limit);
+    svbool_t pred = svwhilelt_b8_s64(i, limit);
-        pred = svwhilelt_b8_s32(i, limit);
+        pred = svwhilelt_b8_s64(i, limit);
     // vdot8s_neon
-    int32_t i = 0;
+    int64_t i = 0;

Trade-offs

• No API or ABI change; results unchanged below the threshold (the 350 tests).
• Not benchmarked through Lucene; a quick C microbenchmark of the stock and fixed libraries showed no measurable difference at 64 to 4,096 bytes (fix/stock 0.985 to 1.030). Only 256-bit SVE was run for this doc; 128-bit rests on the earlier run.

What the checks validated

Scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, GCC 10.5.0, 256-bit SVE, seed DEADBEEF:

• Minimal repro: stock library crashes with -ea and -da; fixed library passes with -ea and -da, native provider logged (repro harness and plain java).
• Closest suites through gradle with -Dlucene.useNativeDotProduct=true and the fixed library: TestVectorUtilSupport (native kernel against the scalar default at 22 sizes up to 4,098) and TestVectorUtil: 350 tests, 0 failures, 16 skipped, assertions on and off; both classes log using NativeVectorizationProvider.
• Vacuity: a mutant library returning result + 1 fails 47 of those tests, so they do run the native kernel; the stock library passes all 350, so they cannot see the bug.
• Earlier: an independent reproducer's equivalent patch passes at 2,147,483,631 with 256-bit and 128-bit SVE; CBMC verifies the fixed source.

Removes the main-loop overflow, since limit - 4*VL cannot overflow for limit >= 0.

Trade-offs

The SVE tail's i += vec_length would then overflow for lengths within one vector of 2^31 (2,147,483,617 to 2,147,483,645 at 256-bit SVE, by arithmetic), so the tail needs the same care; more subtle than the type change for no gain.

NativeVectorUtilSupport could use the Panama path above some length.

Trade-offs

Protects against any user-built library with the same flaw, but adds a branch to a hot call and leaves the example's undefined behaviour in place; a complement, not a replacement.

State that n must stay below 2^31 - 4*VL.

Trade-offs

The Java API and the other providers accept every legal length, and the header's contract says each function returns what the Java method returns.

Method: formal (the tool's own counterexample), hunt unit on the native dot product.

• An AI tool wrote CBMC harnesses for the unmodified dotProduct.c, with semantic models of the SVE and NEON intrinsics. Its bounded matrix over ordinary lengths (2,089 runs, lengths up to 384) and a differential test of the real library against Java (0 mismatches; a deliberately wrong library gave 435,918) found nothing.
• To cover lengths near 2^31, it replaced only the loop index's initial value so the main loop could start at any reachable iteration. CBMC's signed-overflow check then failed at dotProduct.c:66 (for example limit = i = 2,147,483,520 at 32-byte vectors) and at the NEON loop (:127); the fixed source verifies. Choosing to probe near 2^31 was the AI tool's decision, so if that counts as a hypothesis the tier is semi-formal.
• The AI tool then ran the crash through the public VectorUtil.dotProduct; an independent reproducer, a devil's advocate and a reviewer from another model family (OpenAI GPT-5.6) each reproduced it with their own programs, and an adjudication re-ran the reproducer's program; all but the cross-model reviewer built their own library (it ran the supplied one, whose source matches main).