Vectorized scalar quantization rounds 0.49999997 up to byte 1, unlike Math.round

ScalarQuantizer.quantize (public API; in Lucene's codecs, the legacy Lucene99 int7/int4 scalar quantization) has had two implementations since #14304 (10.2.0): a scalar loop that rounds with Math.round(scale * dxc), and a vectorized loop, used with the vector incubator module on CPUs with 256-bit or wider vectors, that adds 0.5 and truncates. The two disagree only just below 0.5: when scale * dxc is 0.49999997f, the sum with 0.5 is a tie that rounds to 1.0, so the vector loop writes byte 1 where Math.round writes 0. Example: new ScalarQuantizer(0f, 1f, (byte) 8) and the component 0x3b008080 (0.0019607842) give 0 on the scalar path and 1 on the vectorized path, on main and every release from 10.2.0 to 10.6.0. The accuracy impact is negligible (byte 0 is nearer by only 2^-24 × alpha), but quantized bytes depend on the hardware and JVM flags, contrary to the code comment and to the test that came with the kernel.

1. The scalar code rounds with int roundedDxs = Math.round(scale * dxc); (DefaultVectorUtilSupport.java:446). Math.round is exact: the closest int, ties up.
2. The vector loop, which runs only when VECTOR_BITSIZE >= 256 (PanamaVectorUtilSupport.java:1497), rounds with fma(dxc, dxc.broadcast(scale), dxc.broadcast(0.5f)).convert(VectorOperators.F2I, 0) (:1510-1511), under the comment "Math.round rounds to positive infinity, so do the same by +0.5 then truncating to int" (:1509). fma is a fused multiply-add when the CPU's FMA is fast, else multiply then add (:97-104); F2I truncates.
3. For y = 0.49999997f (exactly 0.5 - 2^-25), y + 0.5 = 1 - 2^-25 lies exactly halfway between the floats 1 - 2^-24 and 1.0. Round-half-to-even gives 1.0, and truncation gives 1. Math.round(y) gives 0. Without FMA the sum is the same tie. In general the vector path writes 1 where Math.round writes 0 when the exact product lies in [0.5 - 2^-25, 0.5 - 2^-26) (with FMA) or the float product is 0.49999997f (without FMA).
4. Only this boundary can differ. For k >= 1 the floats just below k + 0.5 and just below k + 1 share a binade, so adding 0.5 is exact (the study's Z3 query proves this for the unfused form; for the fused form it did not finish, and the argument plus executed scans cover it: every float in [0, 256] through the real vector rounding step, only 0.49999997 mismatches; with FMA on, 0 differences at the boundaries k >= 1 in 13.0M elements).
5. recalculateScalarQuantizationOffset repeats the expression (:1566-1567) but writes no bytes; only its float correction moves.

new ScalarQuantizer(0f, 1f, (byte) 8): scale = 255 (ScalarQuantizer.java:101, 110). x = Float.intBitsToFloat(0x3b008080) = 0.0019607842 = 8421504 / 2^32, so 255 × x = 0.5 - 2^-25 exactly, the float 0x3effffff = 0.49999997. Scalar path: byte 0. Vector path (256-bit SVE, FMA on or off): byte 1. Scanning every float in [0, 1] through the public API finds exactly one such value for 8 bits (this one) and for 4 bits (0x3d088888), none for 7 bits with (min, max) = (0, 1); random legal quantiles have their own witnesses (700 of 20,000 4-bit and 125 of 20,000 7-bit (min, max) pairs near the first boundary).

• Opt-in path. The vector module is opt-in (Lucene's startup warning recommends it, VectorizationProvider.java:150-153), and the loop needs a preferred vector size of at least 256 bits (PanamaVectorUtilSupport.java:1497). 128-bit hosts (NEON, SVE-128, x86 without AVX2) and JVMs without the module use the scalar code. Only 256-bit SVE was executed; x86 follows the same branch by code reading.
• Legacy formats, or direct API use. On main and 10.4+, the kernel runs when a query is quantized against Lucene99 scalar-quantized segments (ScalarQuantizedVectorScorer.quantizeQuery, used by Lucene99ScalarQuantizedVectorScorer:69 and Lucene99MemorySegmentScalarQuantizedVectorScorer:223); the Lucene99 writer is test code there. In 10.2.x and 10.3.x the writer is in lucene-core, so indexing and merging use it too. The current Lucene104 quantized format does not call this kernel. Code that calls the public ScalarQuantizer directly (up to 8 bits, as the repro does) gets it on any version from 10.2.0.
• Rare. About 1.8e-10 per quantized component for inputs drawn like Lucene's own test generator (study simulation, 2 × 20M configurations).
• Almost no accuracy cost. The two bytes have reconstruction errors of -0.49999997 × alpha and +0.50000003 × alpha (byte 0 is nearer by 2^-24 × alpha); the vector path computes its corrective offset from its own byte, so its scores stay self-consistent. No end-to-end recall or ranking effect was measured.
• What is visible. The same query vector quantizes to different bytes on different hosts or JVM flags (vector module on or off, 128 vs 256+ bits; with FMA on vs off the set of diverging inputs also differs), so scores of legacy segments are not bit-reproducible across hosts. And TestVectorUtilSupport#testMinMaxScalarQuantize, which asserts identical bytes, can fail on inputs its own generator produces (48 of 200,000 random (min, max) pairs carry one; about 1 failure in 300,000 full runs, study estimate). The prior-report searches found no report of such a CI failure.

16 elements fill at least one whole vector at 256 and 512 bits, so the first element goes through the vector loop. The assertion is the one Lucene's own TestVectorUtilSupport#testMinMaxScalarQuantize makes: the bytes must equal the scalar implementation's.

Run it with -Ptests.defaultvectorization=true. The test uses the public API, and outside test mode VectorizationProvider.getInstance() falls back to the scalar code when the JVM runs C1 only or when tests.vectorsize is set to a number (VectorizationProvider.java:157-166). A plain local gradle run always hits one of them: the seed picks a vector size among default/128/256/512, and a numeric size triggers the size check, which comes first; with default, the C1 check applies, because local test JVMs get -XX:TieredStopAtLevel=1 (CI builds do not) (TestsAndRandomizationPlugin.java:139-152, 629-642). Without the option the test passes on stock main for the wrong reason.

As the regression test: this same test, for example as a new method in TestScalarQuantizer. It can only fail on hardware with 256-bit or wider vectors and the vector module; elsewhere it passes.

cp TestScalarQuantizerNearestByte.java $LUCENE/lucene/core/src/test/org/apache/lucene/util/quantization/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.util.quantization.TestScalarQuantizerNearestByte -Ptests.seed=DEADBEEF -Ptests.defaultvectorization=true
# assertions off: add -Ptests.asserts=false (same result)

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.defaultvectorization=true, assertions on (the same with -Ptests.asserts=false)
TestScalarQuantizerNearestByte > testJustBelowHalfRoundsDown FAILED
    java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:36974EA7A6649FA]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.util.quantization.TestScalarQuantizerNearestByte.testJustBelowHalfRoundsDown(TestScalarQuantizerNearestByte.java:32)
  2> INFO: Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled

# controls on the same jars (standalone): 128-bit vectors (-XX:UseSVE=0, preferredBitSize=128) pass (-da);
# no vector module (scalar provider) passes (-ea and -da); -Dlucene.useVectorFMA=false still fails with the same message (-ea).
# gradle with the local default test JVM args (-XX:TieredStopAtLevel=1) passes: by the code (VectorizationProvider.java:157-166) the public API then uses the scalar provider.

• The test. TestVectorUtilSupport#testMinMaxScalarQuantize (TestVectorUtilSupport.java:338-363) runs every provider on the same input and asserts assertArrayEquals(outputs.getFirst(), outputs.get(o)) under "// check the outputs are identical" (:360-362). ScalarQuantizer's class javadoc: values are "bucketed into the nearest byte values" (ScalarQuantizer.java:37-41); at every divergence found, both the float product and the exact product are below 0.5.
• A port, not a new algorithm. #14304 ("Create vectorized versions of ScalarQuantizer.quantize and recalculateCorrectiveOffset") moved ScalarQuantizer's (byte) Math.round(...) unchanged into DefaultVectorUtilSupport and added the vector kernel; none of its 13 review comments or 8 conversation comments discusses rounding.
• Later fixes kept the byte assertion. #14402 / PR #14403 and PR #15443 fixed CI failures of the same test by widening only the float-correction tolerance.
• Legal inputs. Public constructor, finite quantiles, legal bit widths (also 4 and 7, the widths the Lucene99 format accepts), an ordinary finite float inside [min, max]; it fails with assertions off, in the output bytes.
• Careful wording. Even the scalar path is not always the nearest byte of the exact real product: when the float product rounds up to exactly k + 0.5, Math.round rounds up (7, 64 and 127 such floats in [0, 1] for 4, 7 and 8 bits). The contract at stake is (a) identical bytes across implementations and (b) Math.round of the float scale * dxc.

Both bytes are almost equally good; Lucene already accepts that the implementations return different float corrections (the widened tolerances); VectorUtil documents that FMA is used where it is fast; and open PR #16790 (vectorizing OptimizedScalarQuantizer loops) uses "+0.5f then truncate" on purpose, with a reviewer proposing t.add(0.5f).convert(F2I) for Math.round(t).

Why it fails: #16790 changes its scalar code to the same "+0.5f then truncate" so both implementations agree ("Adds 0.5 and truncates, as Math.round is costly to match in the Panama version"; "Not fma, so k matches the scalar version"; new javadoc "Every implementation must write the same bytes and return the same sum"). The rounding rule can be chosen, but the implementations must agree, and minMaxScalarQuantize breaks exactly that. The tolerance covers the float correction, not the bytes. Verdict: real but minor; maintainers may still prefer to align the scalar code (option B) or to relax the test (option C).

Option A was validated through gradle in a scratch worktree of 645d4fe14a20; options B and C were not run.

Compute the product unfused, truncate it, and add 1 when the dropped fraction is at least 0.5. For y >= 0 that is exactly Math.round(y): t = (float) (int) y and y - t are exact, so the comparison sees the true fraction. One private helper, used at both rounding sites (+13/-5). The increment is a masked float add, because Lucene's forbidden-APIs check rejects VectorMask#cast ("Potentially slow on some CPUs"):

--- a/lucene/core/src/java25/org/apache/lucene/internal/vectorization/PanamaVectorUtilSupport.java
+++ b/lucene/core/src/java25/org/apache/lucene/internal/vectorization/PanamaVectorUtilSupport.java
@@ -1487,6 +1487,17 @@ final class PanamaVectorUtilSupport implements VectorUtilSupport {
     return subRet0 + (subRet1 << 1) + (subRet2 << 2) + (subRet3 << 3);
   }
 
+  // Math.round(scale * dxc) of DefaultVectorUtilSupport, lane by lane, for dxc >= 0: an unfused
+  // product, truncated, plus one when the dropped fraction is at least 0.5 (t and y - t are exact).
+  // Adding 0.5f before truncating is not the same: it rounds 0.49999997f up to 1.
+  private static Vector<Integer> roundScaled(FloatVector dxc, float scale) {
+    FloatVector y = dxc.mul(scale);
+    Vector<Integer> truncated = y.convert(VectorOperators.F2I, 0);
+    FloatVector t = (FloatVector) truncated.convert(VectorOperators.I2F, 0);
+    VectorMask<Float> roundUp = y.sub(t).compare(VectorOperators.GE, 0.5f);
+    return t.add(1f, roundUp).convert(VectorOperators.F2I, 0);
+  }
+
   @Override
   public float minMaxScalarQuantize(
       float[] vector, byte[] dest, float scale, float alpha, float minQuantile, float maxQuantile) {
@@ -1506,9 +1517,7 @@ final class PanamaVectorUtilSupport implements VectorUtilSupport {
         FloatVector dxc = v.min(maxQuantile).max(minQuantile).sub(minQuantile);
         // Scale the value to the range [0, 127], this is our quantized value
         // scale = 127/(maxQuantile - minQuantile)
-        // Math.round rounds to positive infinity, so do the same by +0.5 then truncating to int
-        Vector<Integer> roundedDxs =
-            fma(dxc, dxc.broadcast(scale), dxc.broadcast(0.5f)).convert(VectorOperators.F2I, 0);
+        Vector<Integer> roundedDxs = roundScaled(dxc, scale);
         // output this to the array
         ((ByteVector) roundedDxs.castShape(BYTE_SPECIES, 0)).intoArray(dest, i);
         // We multiply by `alpha` here to get the quantized value back into the original range
@@ -1563,8 +1572,7 @@ final class PanamaVectorUtilSupport implements VectorUtilSupport {
 
         // same operations as in quantize above
         FloatVector dxc = v.min(maxQuantile).max(minQuantile).sub(minQuantile);
-        Vector<Integer> roundedDxs =
-            fma(dxc, dxc.broadcast(scale), dxc.broadcast(0.5f)).convert(VectorOperators.F2I, 0);
+        Vector<Integer> roundedDxs = roundScaled(dxc, scale);
         FloatVector dxq = ((FloatVector) roundedDxs.castShape(FLOAT_SPECIES, 0)).mul(alpha);
         sum =
             fma(

private static Vector<Integer> roundScaled(FloatVector dxc, float scale) {
  FloatVector y = dxc.mul(scale);
  Vector<Integer> truncated = y.convert(VectorOperators.F2I, 0);
  FloatVector t = (FloatVector) truncated.convert(VectorOperators.I2F, 0);
  VectorMask<Float> roundUp = y.sub(t).compare(VectorOperators.GE, 0.5f);
  return t.add(1f, roundUp).convert(VectorOperators.F2I, 0);
}
...
Vector<Integer> roundedDxs = roundScaled(dxc, scale);

Trade-offs

• For the quantizer's scaled values (non-negative, at most about 2^bits - 1) it computes exactly Math.round of the unfused product, so it writes the scalar provider's bytes, as 10.1.0 did (checked on the repro and the suites); no format change, no change to the scalar code.
• A few more lane operations per vector (multiply, two converts, subtract, compare, masked add, convert, instead of one fused multiply-add and a convert); not benchmarked.
• The recalculateScalarQuantizationOffset change is optional (that method writes no bytes) but keeps "same operations as in quantize above" true.
• The study's version of this fix adds the increment with IntVector.add(1, mask.cast(INT_SPECIES)); it passes the tests but fails :lucene:core:check (forbiddenApisMain25: VectorMask#cast is forbidden as "Potentially slow on some CPUs").

What the checks validated

Gradle in a scratch worktree of 645d4fe14a20, every run with -Ptests.defaultvectorization=true:

• the repro fails on stock main with assertions on and off and passes with the fix with assertions on and off;
• TestVectorUtilSupport, TestScalarQuantizer, TestScalarQuantizedVectorSimilarity and the repro at the host's vector size (256): 322 tests, 0 failures (16 skipped by the tests' own assumptions); the three classes at -Ptests.vectorsize=512 (the 512-bit species forced on the 256-bit host): 321 tests, 0 failures;
• five Lucene99 scalar-quantized backward-codecs suites: 196 tests, 0 failures;
• :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs, license headers) with the fix and the repro as a new test file: BUILD SUCCESSFUL (errorProne skipped by the build's default; eclint not configured).

Not benchmarked; x86 not run.

Change DefaultVectorUtilSupport.java:446 to (int) (scale * dxc + 0.5f) and drop FMA from the vector expression (dxc.mul(scale).add(0.5f)), so both compute the same two roundings. This is the approach open PR #16790 takes for its new kernels.

Trade-offs

Byte identity and slightly cheaper vector code, but both implementations then write 1 at 0.49999997f, so neither is Math.round or the nearest byte there, the scalar output changes from what 10.1.0 and earlier wrote, and the "nearest byte" javadoc needs a footnote.

What the checks validated

Not run.

Let TestVectorUtilSupport#testMinMaxScalarQuantize accept a difference of 1 where fl(scale * dxc) == 0.49999997f, and correct the code comment.

Trade-offs

No runtime change and no cost, but quantized bytes stay hardware-dependent, against the comment's stated intent.

What the checks validated

Not run.

• Method: floating-point SMT (Z3), the hunt's formal tier, in the earlier verification study's SIMD-equivalence unit: (int) fma(x, scale, 0.5f) and the unfused (int) (scale * x + 0.5f) against Math.round(scale * x). It found the witness 0x3b008080 at scale 255 and proved that the unfused form diverges only when fl(scale * x) is 0.49999997f; for the fused form it found the window [0.5 - 2^-25, 0.5 - 2^-26) but did not finish the "only there" query (unknown at 600 s), which the hand derivation and the executed boundary scans cover.
• Independent verifier: its own JUnit test searched witnesses itself (3,006 quantizer configurations, a 671M-element lattice through both real kernels) instead of reusing the original witness, and reproduced at 256 bits and at a forced 512-bit vector size, with and without FMA, on main and 10.5.2; exhaustive float sweeps (FixCheck, ScaleExhaustive) confirmed the single boundary and checked the exact-rounding fix.
• Later: reduced to the one-test repro (2026-10-29), re-verified on main 645d4fe14a20, and settled by a tie-break with two derived-witness checks (2026-11-01).