Dequantized FLOAT16 vectors overflow to Infinity when the raw vectors are dropped

Lucene104ScalarQuantizedVectorsFormat quantizes FLOAT16 vectors (since #16473). For read-only indexes whose raw float16 vectors were dropped to save storage, getFloat16VectorValues returns OffHeapScalarQuantizedFloat16VectorValues, which rebuilds each vector "with some precision loss relative to the original float16 vectors". It calls OptimizedScalarQuantizer.deQuantize(byte[], short[], ...), which narrows xi + centroid[h] with Float.floatToFloat16 and no clamp (OptimizedScalarQuantizer.java:294). The optimized interval can reach a little past the data, so a component at the fp16 limit rebuilds just beyond it and becomes Infinity: (65504, -65504, 0) reads back as (65472, -Infinity, 85.625) with the default 8-bit encoding. Full-precision rescoring then gives the document an infinite EUCLIDEAN distance (score 0.0), so an exact match can rank last; with -ea, VectorUtil.squareDistance trips its finiteness assert. The FLOAT32 sibling view stays finite on the same data. It needs the opt-in quantized format, raw vectors dropped by a file rewrite (as upstream's tests do), EUCLIDEAN or MAXIMUM_INNER_PRODUCT and components near +-65504 (tens of thousands for 1-2 bit encodings); plain kNN search is unaffected. Unreleased main and branch_11x only; saturating to +-65504 before narrowing fixes it.

• Dequantized view: OffHeapScalarQuantizedFloat16VectorValues (@lucene.internal). Javadoc: "Used for read-only indexes whose raw float16 vectors have been dropped to save storage: only the scalar-quantized bytes remain, so vectorValue(int) reconstructs float16 values by dequantizing them, with some precision loss relative to the original float16 vectors." (OffHeapScalarQuantizedFloat16VectorValues.java:43-45).
• Dropped raw vectors: an empty .vec file and a vector length of 0 in .vemf, written after IndexWriter closes, as TestLucene104ScalarQuantizedVectorsFormat#simulateEmptyRawVectors does (:614). The FLOAT32 view's javadoc names that method as the way to remove full-precision vectors (OffHeapScalarQuantizedFloatVectorValues.java:48-51).

1. Lucene104ScalarQuantizedVectorsReader.getFloat16VectorValues: when the raw float16 vectors have size 0 (:327), it returns the dequantized view paired with the quantized values (:331-343).
2. At index time OptimizedScalarQuantizer.scalarQuantize centres the vector on the centroid (:205), clamps the initial interval to the centred vector's [min, max] (:217-218), then optimizeIntervals (:219, defined at :336) can move an end past [min, max].
3. The view's vectorValue (:105-127) calls deQuantize(byte[], short[], ...) (:117-123), which computes xi = q * step + lowerInterval (OptimizedScalarQuantizer.java:293) and Float.floatToFloat16((float) (xi + centroid[h])) with no clamp (:294). Its javadoc: "narrowing each reconstructed value to float16" (:272-273). The view is its only production caller.
4. A rebuilt value of magnitude 65520 or more becomes +-Infinity. VectorUtil.squareDistance(short[], short[]) asserts a finite result (VectorUtil.java:144-151, assert :149); with assertions off, EUCLIDEAN scores the infinite distance as 1 / (1 + Infinity) = 0.0.
5. Affected readers: Float16VectorValues#vectorValue and #rescorer on the view, so RescoreTopNQuery.createFullPrecisionRescorerQuery(Query, short[], String, int) (RescoreTopNQuery.java:302-306) and, by code reading, the queries module's Float16KnnVectorFieldSource (:71). Plain kNN search scores in quantized space and was executed unaffected.

The gradle runs of the repro print only its message (-Infinity) with assertions on and off; the whole read-back above was printed by probes with -ea and -da.

Consequences executed on 645d4fe14a20 by probes and larger tests (not part of the repro). On the repro's two documents, rescoring doc 0's own vector with RescoreTopNQuery returns [doc=1 1.17e-10, doc=0 0.0] (-da). Default encoding, EUCLIDEAN, docs [65504, -65504, 65504, -65504, 1000, -1000, 0, 0], its negation and the zero vector: [doc=2 5.83e-11, doc=0 0.0, doc=1 0.0]. 1-bit encoding (HNSW variant), (60000, 49984, 0, 0), (20000, -49984, 0, 0), (40000, 0, 0, 0): doc 0 reads back as (Infinity, 33824, 1171, 1171) and ranks last, [doc=2 3.45e-10, doc=1 1.01e-10, doc=0 0.0]. In the controls with raw vectors kept, doc 0 ranks first. With -ea the rescoring stops at VectorUtil.java:149.

• Who is hit. Applications on unreleased main and branch_11x that index FLOAT16 vectors with Lucene104ScalarQuantizedVectorsFormat or Lucene104HnswScalarQuantizedVectorsFormat (both executed) and then drop the raw vectors for a read-only searcher index. No IndexWriter API does that today; open #15630 / #13158 would make the writer produce such files.
• Similarities. EUCLIDEAN and MAXIMUM_INNER_PRODUCT (both executed). COSINE vectors are scaled to unit length before quantization (Lucene104ScalarQuantizedVectorsWriter.java:903-905), so they never come near the limit. DOT_PRODUCT is documented to need unit-length vectors (VectorSimilarityFunction.java:53-57), but nothing enforces that; it was not tested.
• How large. There is no simple threshold: it depends on the optimized interval end, the centroid and the code a component gets. What the probes saw: with the default 8-bit encoding, 40 of 40 random 22-dimensional segments with about 25% of components at exactly +-65504 overflowed, and 0 of 200 with gaussian values of sigma 15000 capped at 60000 did. With 1 and 2 bits, values in the tens of thousands were enough (60000 at 1 bit, -50944 at 2 bits). All 5 encodings overflow on a witness at +-65504. Values exactly at 65504 arise, for example, when an application saturates float32 values while converting them to fp16.
• What happens. vectorValue() returns an infinite component with no error. Full-precision rescoring gives the document an infinite EUCLIDEAN distance (score 0.0), so an exact match can rank last; under MAXIMUM_INNER_PRODUCT the scores became NaN or +Infinity and moved such documents into the top 3. With -ea the rescore ends in an AssertionError.
• What does not happen. No index damage, no hang, no change to the inner query's match set; plain kNN search returns identical hits and scores with raw vectors kept and dropped.
• Scale. Opt-in on every axis. A single affected document is enough, in a segment that also holds other vectors; a one-document segment reads back exactly, because its centroid is the vector itself.

One JUnit class, one test method, public API only (it sits in org.apache.lucene.codecs.lucene104 like the format's own tests but uses no package-private member), ByteBuffersDirectory, an explicit codec, nothing randomized. 2 documents of 3 dimensions is the smallest default-encoding input found that overflows (1-dimensional vectors, single documents and 2-dimensional pairs read back finitely). The helper is a condensed copy of TestLucene104ScalarQuantizedVectorsFormat#simulateEmptyRawVectors (16 is DIRECT_MONOTONIC_BLOCK_SHIFT), because that method lives in core's test sources.

As the regression test: this same test, for example as a new method in TestLucene104ScalarQuantizedVectorsFormat, which already has simulateEmptyRawVectors(dir). With the fix it passes with assertions on and off, and :lucene:core:check -x test passes with the test file added. It compiles only where the float16 API exists (main from #16383) and fails from #16473 on.

cp TestFloat16DequantizedViewOverflow.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene104/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene104.TestFloat16DequantizedViewOverflow -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on (the same with -Ptests.asserts=false)
TestFloat16DequantizedViewOverflow > testDequantizedValuesAreFinite FAILED
    java.lang.AssertionError: expected a finite approximation of -65504, got -Infinity
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:8E7A13E97D1A25A2]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.codecs.lucene104.TestFloat16DequantizedViewOverflow.testDequantizedValuesAreFinite(TestFloat16DequantizedViewOverflow.java:60)

# probe (not part of the repro): the values behind it
centroid=[32752.0, -32752.0, 0.0]
ord=0 interval=[-32794.797, 32709.172] quantized=[255, 0, 128] floatReconstruction=[65461.168, -65546.8, 85.625] fp16ReadBack=[65472.0, -Infinity, 85.625]
ord=1 interval=[-32794.797, 32709.172] quantized=[0, 255, 128] floatReconstruction=[-42.796875, -42.83142, 85.625] fp16ReadBack=[-42.8125, -42.84375, 85.625]
floatToFloat16(65519.0) = 65504.0    floatToFloat16(65520.0) = Infinity

# with the saturating fix: the test passes with assertions on and off

• The view's contract. "reconstructs float16 values by dequantizing them, with some precision loss relative to the original float16 vectors" (OffHeapScalarQuantizedFloat16VectorValues.java:43-45; the "relative to the original" wording came from a review suggestion by msokolov in #16473).
• Upstream tests assert an error bound in this mode. testReadQuantizedFloat16VectorWithEmptyRawVectors "asserts they stay within the quantization error bound" (TestLucene104ScalarQuantizedVectorsFormat.java:444-449, added by #16473); the shared base test says formats "maintain accuracy within expected error bounds even when the original raw vector file is empty" (BaseKnnVectorsFormatTestCase.java:2187-2189). Both use small random vectors, far from 65504: the fp16 test unit-length vectors (TestLucene104ScalarQuantizedVectorsFormat.java:461), the base test unit-length vectors for COSINE and otherwise components in [0, 1) (BaseKnnVectorsFormatTestCase.java:2202-2204, :1903-1915).
• Finite in, finite out. KnnFloat16VectorField rejects non-finite values (checkFiniteFloat16, KnnFloat16VectorField.java:72); VectorUtil.squareDistance(short[], short[]) asserts a finite result (:149).
• The FLOAT32 sibling. On the finder's 1-bit data with raw vectors dropped, OffHeapScalarQuantizedFloatVectorValues rebuilds doc 0 as [73820.7, 33820.707, 1171.291, 1171.291] and ranks it first; on the repro's two documents it reads doc 0 back as [65461.168, -65546.8, 85.625], finite. For this data the difference that matters is that it keeps the float reconstruction, while the FLOAT16 view narrows it without saturating.
• A supported mode. CHANGES.txt: #14792 (introduced the fallback), #15476 ("Enforce fallback support for float vector retrieval in quantized KNN vector formats"), #16415 ("Fix scoring on scalar-quantized indexes whose raw float vectors have been dropped", 10.5.1).
• No sign of intent. None of the 41 #16473 PR texts mentions overflow, infinity, 65504, saturation, clamping, range or finiteness.

The reviews by a reviewer from another model family (OpenAI GPT-5.6) voted not real on these grounds, while reproducing the bug and agreeing on the cause and the fix: (a) deQuantize(byte[], short[], ...) is a public static method documented as "narrowing each reconstructed value to float16", and TestOptimizedScalarQuantizer#testDeQuantizeFloat16MatchesFloat asserts equality with Float.floatToFloat16 of the float overload, so Infinity is the helper's specified result; (b) the trigger state exists only after rewriting index files ("The raw vector file is emptied below, which CheckIndex rightly rejects", BaseKnnVectorsFormatTestCase.java:2275), the view is @lucene.internal, and SECURITY.md calls modified index files undefined behaviour; (c) 65504 is extreme.

Why it fails: (a) only decides where the clamp goes; the helper's only production caller is the view, whose contract is an approximation, and the helper's own test (values in [0, 1)) passes with the clamp. (b) The quantized data is exactly what IndexWriter wrote; only the raw-vector files are rewritten (.vec emptied, its metadata .vemf set to 0 vectors; .veq and .vemq untouched), which is the mode's purpose, and upstream tests, documents and bug-fixes that mode. The quoted CheckIndex comment is in a FLOAT32 test: on the repro's index CheckIndex reports no problem before or after the drop, and its vector test covers 0 fields, because at main it skips FLOAT16 vector fields (CheckIndex.java:2864-2867; see LFV-089). (c) The values are legal input the field accepts, and the FLOAT32 path handles them. Verdict: a defect; a maintainer may still call it hardening of an opt-in internal mode, which is why fix A pairs the clamp with a javadoc sentence.

Option A was validated through gradle in a scratch worktree of 645d4fe14a20; B, C and D were not run.

Clamp xi + centroid[h] to [-65504, 65504] before Float.floatToFloat16, and add to the javadoc: "The quantization interval can reach slightly past the range of the original values, so reconstructed values beyond the finite float16 range are saturated to +/-65504 instead of becoming infinite." Full diff:

--- a/lucene/core/src/java/org/apache/lucene/util/quantization/OptimizedScalarQuantizer.java
+++ b/lucene/core/src/java/org/apache/lucene/util/quantization/OptimizedScalarQuantizer.java
@@ -53,6 +53,8 @@
         {-3.611f, 3.611f},
         {-3.922f, 3.922f}
       };
+  // the largest finite float16 value
+  private static final float FLOAT16_MAX = 65504f;
   // the default lambda value
   private static final float DEFAULT_LAMBDA = 0.1f;
   // the default optimization iterations allowed
@@ -270,7 +272,9 @@
    * Dequantizes a quantized byte vector back to float16 values.
    *
    * <p>Behaves as {@link #deQuantize(byte[], float[], byte, float, float, float[])}, narrowing each
-   * reconstructed value to float16.
+   * reconstructed value to float16. The quantization interval can reach slightly past the range of
+   * the original values, so reconstructed values beyond the finite float16 range are saturated to
+   * +/-65504 instead of becoming infinite.
    *
    * @param quantized the quantized byte vector to dequantize
    * @param dequantized the output array to store dequantized float16 bit patterns
@@ -291,7 +295,8 @@
     double step = (upperInterval - lowerInterval) / nSteps;
     for (int h = 0; h < quantized.length; h++) {
       double xi = (double) (quantized[h] & 0xFF) * step + lowerInterval;
-      dequantized[h] = Float.floatToFloat16((float) (xi + centroid[h]));
+      double value = Math.clamp(xi + centroid[h], -FLOAT16_MAX, FLOAT16_MAX);
+      dequantized[h] = Float.floatToFloat16((float) value);
     }
     return dequantized;
   }

// the largest finite float16 value
private static final float FLOAT16_MAX = 65504f;
...
double xi = (double) (quantized[h] & 0xFF) * step + lowerInterval;
double value = Math.clamp(xi + centroid[h], -FLOAT16_MAX, FLOAT16_MAX);
dequantized[h] = Float.floatToFloat16((float) value);

Trade-offs

• A value at the limit rebuilds as 65504 instead of Infinity: an error no larger than the float reconstruction's own (42.8 in the repro).
• In-range values are unchanged bit for bit, so the helper's equality test still passes.
• It changes the documented behaviour of a public static method of an @lucene.experimental class (hence the javadoc sentence); the view is its only production caller.
• Cost: one clamp per component on the fallback read path. No on-disk change; existing indexes are fixed on read.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF, 2026-11-01: without the fix the repro fails with assertions on and off; with it, it passes with assertions on and off.
• With the fix: TestOptimizedScalarQuantizer (7), TestLucene104ScalarQuantizedVectorsFormat (65), TestLucene104HnswScalarQuantizedVectorsFormat (64), TestLucene104ScalarQuantizedMergeScorer (9), TestQuantizedVectorValuesPrefetch (3) and the repro: 149 tests, 0 failures, 4 skipped (two nightly testRecall, two empty-raw base tests the HNSW test class opts out of).
• ./gradlew :lucene:core:check -x test with the fix and the repro as a new test file: BUILD SUCCESSFUL (google-java-format, ecj lint, forbidden APIs, licenses; error-prone skipped).
• Earlier, the same clamp shadowed into the stock jars made the finder's, the independent reproducer's and the cross-model reviewers' tests pass. The full Lucene test suite was not run.

In OffHeapScalarQuantizedFloat16VectorValues.vectorValue, dequantize into a float[] scratch with the float overload and narrow with saturation.

Trade-offs

Keeps the helper's documented plain narrowing; costs a dimension-sized float scratch per view and duplicates the narrowing loop. Not validated.

Constrain the interval so that centroid + lower and centroid + upper stay within +-65504.

Trade-offs

Changes the quantized encoding and quantized scores of new segments and does not fix existing indexes. Not recommended, not validated.

A javadoc note on the view.

Trade-offs

Keeps the wrong rescoring order and the assertion failure, and contradicts the view's "precision loss" wording and the error-bound tests. Not recommended.

• Method: floating-point SMT precondition chaining (Z3), the hunt's formal tier, in the hunt unit on optimized scalar quantization. The obligation "the fp16 reconstruction of a legal in-interval fp16 component is finite" was refuted for 1, 4 and 8 bits (its violation is satisfiable); two more queries proved that fp16 narrowing overflows iff |y| >= 65520 and that a clamp keeps values finite.
• Witness: a real-jar probe of scalarQuantize + deQuantize(short[]) over random fp16 vectors hit +-Infinity for 1, 2, 4, 7 and 8 bits; then a JUnit test on a real index with the raw vectors dropped (read-back and rescoring) failed 2/2 with -ea and -da, and the clamp made it pass.
• Later: an independent reproducer, an intended-behaviour devil's advocate, cross-model reviews and an adjudication checked it (2026-10-25); the test was reduced to one method (2026-10-29) and re-verified on main 645d4fe14a20.