On Lucene 11, an old scalar-quantized segment with equal stored quantiles (e.g. identical vectors) decodes 127 units too low, so its exact matches rank last

Lucene99(Hnsw)ScalarQuantizedVectorsFormat stores each vector as small integer codes plus two numbers per segment, the lower and upper quantile. When a segment's two stored quantiles are equal, for example because every vector of the field is the same constant vector, the 9.9-10.3 writers store (q, q) and every code as 0, which the documented formula decodes as q; every release reader (9.9.2 to 10.6.0 tested, on the same or equivalent indexes) does so. On Lucene 11 (main and branch_11x, unreleased) the backward-codecs reader rebuilds the quantizer through the public ScalarQuantizer constructor, which since #14049 widens equal quantiles to (q - 127, q + 127) (int4: q - 15, q + 15). Code 0 then decodes 127 (or 15) units too low, and exact and near matches in that segment score about 0 and rank last: 0.0000153 instead of 1.0 for [1, 1, 1, 1]. No error; ordinary segments are unaffected and a merge should clear it. Keeping the lower quantile in that branch fixes it.

1. What the old writer stored. For a segment whose vectors are all [1, 1, 1, 1], the 10.3.2 writer stores minQuantile = maxQuantile = 1.0 and every code 0. The documented decode, float = (maxQuantile - minQuantile)/127 * byte + minQuantile (ScalarQuantizer.java:51), gives 1.0. The release constructor has no special case: scale = 127/0 = +Infinity and alpha = 0. Each value is clamped to [1, 1], so its offset from minQuantile is 0 and Math.round(Infinity × 0) = Math.round(NaN) = 0: every code is 0. With alpha = 0 every EUCLIDEAN query scores these documents 1.0 (#13480).
2. Lucene 11 reads both floats back unchanged in Lucene99ScalarQuantizedVectorsReader.FieldEntry.create and calls new ScalarQuantizer(minQuantile, maxQuantile, bits) (:481-483, :488-490).
3. The constructor widens them. Since #14049, if (minQuantile == maxQuantile) sets minQuantile - divisor and maxQuantile + divisor, with divisor = 127 for 7 bits and 15 for 4 bits (ScalarQuantizer.java:101-105). For (1, 1) that is (-126, 128), alpha = 2 and scale = 0.5 (:110-111). The comment says this "leads to alpha = scale = 1" (:103).
4. Code 0 now means -126. deQuantize computes alpha * code + minQuantile = 2 × 0 - 126 (:165-168). The scorer quantizes the query with the same parameters (Lucene99ScalarQuantizedVectorScorer.java:62-76): [1, 1, 1, 1] becomes codes 64 against stored codes 0, and the EUCLIDEAN score 1 / (1 + squareDistance × alpha²) (:163-165) is 1 / (1 + 4 × 64² × 4) = 0.0000153. With the Panama provider, Lucene99MemorySegmentScalarQuantizedVectorScorer uses the same quantizer and multiplier, so the scores are identical. A query [-126, -126, -126, -126] scores those documents exactly 1.0.

In every field the ordinary segment gets the same codes and scores on Lucene 11 as on its writer. On main this branch has no production writer behind it (the Lucene99 scalar-quantized writer is only in test sources since #15223, and no Lucene production code calls the public fromVectors factories), so its only production effect in Lucene is on segments written by releases.

• Who is exposed. Users of the opt-in Lucene99HnswScalarQuantizedVectorsFormat or Lucene99ScalarQuantizedVectorsFormat (any similarity) whose index was written by the releases that write the format, int7 by 9.9.0-10.3.x and int4 by 9.11.0-10.3.x (int4 needs format version 1, added by #13197), and who open it with Lucene 11 without reindexing. CHANGES says that is supported: "Indexes created with 9.x remain fully supported in Lucene 11, no reindexing required" (GITHUB#16741). Not the default codec.
• What it takes. A segment whose two stored quantiles are equal. The demonstrated case is a segment in which every vector of the field is identical: a constant vector (normalized, with equal components, for DOT_PRODUCT or COSINE), all-zero placeholder vectors, or a one-document segment with such a vector. One ordinary vector, or zeros mixed with 4-20% dense vectors, does not give equal quantiles. Near-constant data can: the quantile step clips the extreme values, so [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 5] at dimension 4 give (1, 1) on 10.3.2 and every code 0 (executed). No scale needed. A merge should clear it, since the legacy reader exposes the raw floats and the current writer re-quantizes them (code reading; no run merges such a segment).
• What goes wrong. Silent wrong scores: exact and near matches in that segment drop from first to last. Same with assertions on and off, and with the Panama and the default vector provider.
• Honest scope. On the writing release these documents already score 1.0 for every EUCLIDEAN query (#13480), so Lucene 11 is not worse for every query: for all-zero placeholders, ranking them low is arguably closer to the truth. The clear harm needs identical (or near-constant) non-trivial vectors queried near them. Hence likelihood contrived and priority Low. No release has it yet; the patch is small but changes a public constructor.

The index is embedded as a base64 zip (47 lines). It was written through the public API of the 10.3.2 jar (regenerated 2026-11-04 with neutral segment diagnostics: java.vendor = OpenJDK, os.version = Unknown): the default Lucene99HnswScalarQuantizedVectorsFormat() (int7) for a 4-dimension EUCLIDEAN field euc7, a stored id, NoMergePolicy, compound files; a0, a1, a2 = [1, 1, 1, 1], commit; b0 = [2, 1, 1, 1], b1 = [3, 3, 3, 3], b2 = [-1, 0, 2, 5], commit. The test searches KnnFloatVectorQuery("euc7", [1, 1, 1, 1], 6) and asserts that an exact match ranks first; the message prints every hit with its score.

The same test is the suggested regression test; the index could also live as a .zip resource next to the int7_hnsw.*.zip files. It passes checkGoogleJavaFormat, ecjLintTest and forbiddenApisTest. The 9.12.3-written copy of the same index fails on main the same way.

# gradle, main 5adff78bb168, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestOldUniformQuantizedSegment > testExactMatchRanksFirst FAILED
    java.lang.AssertionError: an exact match must rank first; hits: b0=0.492188 b2=0.124503 b1=0.0583900 a0=1.52586e-05 a1=1.52586e-05 a2=1.52586e-05
        at org.apache.lucene.backward_index.TestOldUniformQuantizedSegment.testExactMatchRanksFirst(TestOldUniformQuantizedSegment.java:111)
# the JUnit runner: the same failure on main with -ea and -da (Panama provider)
# 10.3.2, 10.5.2 and 10.6.0 jars on the same bytes: pass; hit list (10.3.2, 10.6.0) a0=1.00000 a1=1.00000 a2=1.00000 b0=0.492188 b2=0.124503 b1=0.0583900
# main with the recommended fix: pass, the same hit list as 10.6.0

• The stored quantiles are the parameters of the stored codes. The class javadoc formula (ScalarQuantizer.java:50-51) gives 1.0 for code 0 with (1, 1). Every release reader from 9.9.2 to 10.6.0 (on the same or equivalent indexes), and main's own parent commit a833887be67, compute that.
• Lucene 11 states that 9.x indexes stay supported. CHANGES: "Indexes created with 9.x remain fully supported in Lucene 11, no reindexing required" (CHANGES.txt:40-41). That is no promise of identical approximate scores for an experimental format; the point is that stored codes get a new meaning without notice. The #14049 entry describes a fix, "Fixes scalar quantization div-by-zero when all values are identical" (:342-343). No CHANGES or MIGRATE note mentions a decode change.
• The code does not do what its comment says (ScalarQuantizer.java:103). The #14049 discussion touches old indexes only in general: s1monw asked that the back-compat tests not be randomized, msokolov wondered whether a customized codec could end up "failing to be able to load an older index version" with no warning from those tests, and mccullocht later asked to backport the change to branch_10x. No comment considers how the widening changes the decode of codes already stored in old equal-quantile segments.
• Tests assert old int7 indexes keep their top hit. TestInt7HnswBackwardsCompatibility checks the top kNN hit on the 9.10.0-10.6.0 indexes (:105-106 after appending documents, :143-144 in testReadOldIndices), but its vectors {0.2, -0.1, 0.1 + 0.1 * id} (:57, :134) never make the quantiles equal.
• Precedent. #13867 (9.12 broke reading older Lucene99 scalar-quantized segments, fixed by #13874 and #13912) was treated as a back-compat bug.

The case for "intended" (built by an intended-behaviour check and by a reviewer from another model family, OpenAI GPT-5.6, who rated it moderate): the formats are @lucene.experimental; quantization is lossy and kNN approximate ("it's unclear what kinds of guarantees we can make about score values under quantization", msokolov on #14049); the widening was a deliberate, approved choice; and these degenerate segments already score oddly on the writing release (#13480), so Lucene 11 only produces different wrong numbers.

Why it fails: loss and approximation describe quantization error, not a reader that reinterprets persisted codes. Re-quantizing the raw vector on main gives code 64, not the stored 0, and an exact match ends below every other document. The choice was made for new quantizers (when it merged, main still wrote this format, so its own segments stayed consistent); no comment in the discussion considers the decode of codes already stored in old equal-quantile segments, and the wrong comment suggests the effect on them was not analysed.

One point to state carefully: Lucene 11 is not worse for every query. Persisted codes decode with other parameters, so exact and near matches go from first to last; #13480 is the related writer-side case. A maintainer may still triage the two together, since both start from uniform vectors: #13480 is about the writer's own scores, this is a newer reader giving stored codes a new meaning.

Three options. A was built and tested; B and C were not run.

One line in ScalarQuantizer's constructor (ScalarQuantizer.java:104), plus the comment. For stored (q, q) the quantizer becomes (q, q + 127) (int4: q + 15), alpha = scale = 1, and code 0 decodes to q again.

if (minQuantile == maxQuantile) {
  // avoid divide-by-zero by widening upwards only (alpha = scale = 1). Keep minQuantile: older
  // segments store min == max with every code 0, so code 0 must still decode to minQuantile.
  this.minQuantile = minQuantile;
  this.maxQuantile = maxQuantile + divisor;
} else {

Trade-offs

• It changes the constructor for every caller with equal quantiles. On main that is only this reader in Lucene's production code, plus the test-only Lucene99 writer, the public factories fromVectors and fromVectorsAutoInterval (no production caller in Lucene), and any code outside Lucene that uses them or the public constructor.
• Query values below q are clamped to q; there is no stored data below q.
• It restores the decode, not the writing release's query-independent 1.0 scores (that is #13480).
• One field shape stays imperfect: with a dynamic confidence interval (int4, confidenceInterval = 0) the 10.3.2 writer itself stored (0, 0) for [1, 1, 1, 1], so even its own decode is 0.0. With the fix the exact match there scores 0.2 and ranks second behind b0 (0.67); only option C ranks it first.

What the checks validated

Scratch worktree at 5adff78bb168, gradle: the minimal test fails without the fix and passes with it, with -Ptests.asserts=true and false. With the fix, :lucene:backward-codecs:test for backward_index.*, backward_codecs.lucene99.* and backward_codecs.lucene102.*, plus :lucene:core:test for util.quantization.* and TestLucene104ScalarQuantizedVectorsFormat: 27 suites, 1,939 tests, 0 failures (141 skipped), with assertions on and off; among them TestInt7HnswBackwardsCompatibility (105), TestLucene99ScalarQuantizedVectorsFormat (61), TestLucene99HnswScalarQuantizedVectorsFormat (59) and TestScalarQuantizer (8). checkGoogleJavaFormat, ECJ lint and forbidden-apis pass; the EditorConfig lint did not run (its tool is not installed here). The JUnit runner with the fixed class: pass with -ea and -da, and the hit list equals 10.6.0's. Re-run 2026-11-04 with the regenerated fixture in a clean scratch worktree: the minimal test fails without the fix and passes with it, with assertions on and off, and the format, ECJ lint and forbidden-apis checks pass; the 27-suite run used the earlier copy of the fixture.

In Lucene99ScalarQuantizedVectorsReader.FieldEntry.create (:481-490), build new ScalarQuantizer(min, min + ((1 << bits) - 1), bits) when the stored quantiles are equal.

Trade-offs

Same effect on read as A and leaves the constructor alone. The wrong comment stays, and code outside Lucene that rebuilds quantizers from stored quantiles keeps the shift.

What the checks validated

Not run.

Give the reader a legacy quantizer with alpha = 0 for stored min == max, so old segments score bit-identically to their writer.

Trade-offs

Also keeps #13480's query-independent 1.0 scores, and ranks the dynamic-interval case first. Needs a new constructor or scorer path: the largest change.

What the checks validated

Not run.

Method: formal, an SMT counterexample. A Z3 model in floating-point arithmetic (QF_FP, binary32, round to nearest even) encoded the release constructor and main's constructor applied to the same stored pair (q, q), and asked whether code 0 can decode differently. The query was satisfiable in 1.14 s: q = 321 decodes to 321 on the release and to 194 on main, with main's alpha = 2 (int4: 0 versus -15). The decodes agree again only for huge |q| (from about 2^31 for int7), where q ± divisor rounds back to q.

The model was then checked end to end: indexes written through the public API of the 10.3.2 and 9.12.3 jars, read on main by a JUnit test, gave the same shift, and the release readers on the same bytes did not.