PACKED_NIBBLE quantized vectors with an odd dimension fail once raw vectors are dropped

Lucene104ScalarQuantizedVectorsFormat and its HNSW variant can rebuild approximate float vectors from the quantized codes when a read-only index has had its full-precision vectors dropped (the fallback added by #15415 in 10.4.0). With ScalarEncoding.PACKED_NIBBLE, 4-bit codes are packed two per byte and an odd dimension is padded with one nibble: a 3-dim vector is 2 bytes, 4 nibbles.

The dequantizing view allocates its unpack buffer as new byte[dimension] (3), but VectorUtil.int4Unpack needs exactly twice the packed length (4), so every read of an odd-dimension field throws, and full-precision rescoring throws with it. The writer and the quantized scorer already size this buffer with the padded length, the data on disk is intact, and #16030's author fixed these lines on purpose ("it would occasionally compute the wrong size and tests would fail"); the fix was lost when #16030 was reverted.

int docPackedLength = encoding.getDocPackedLength(dimension);      // :91, (3 + 1) / 2 = 2 bytes
...
this.unpackedByteVectorValue = new byte[dimension];                 // :96, 3 slots
...
case PACKED_NIBBLE ->                                               // :123-124
    OffHeapScalarQuantizedVectorValues.unpackNibbles(byteValue, unpackedByteVectorValue);

• unpackNibbles is VectorUtil.int4Unpack (OffHeapScalarQuantizedVectorValues.java:191-193), which throws IllegalArgumentException("vector dimensions differ: " + unpacked.length + " != 2 * " + packed.length) unless unpacked.length == packed.length * 2 (VectorUtil.java:313-318). For odd d, d != 2 * ((d + 1) / 2), so every odd dimension fails and every even one is exact.
• The layout: PACKED_NIBBLE(1, (byte) 4, 4) (QuantizedByteVectorValues.java:39); getDiscreteDimensions rounds 3 up to 4 and getDocPackedLength gives 2 bytes (:136-139, :161-166). The writer quantizes into new byte[encoding.getDiscreteDimensions(dim)] (Lucene104ScalarQuantizedVectorsWriter.java:190) and packs with packNibbles (:551; OffHeapScalarQuantizedVectorValues.java:182-188): for dim 3, byte 0 holds codes 0 and 2, byte 1 holds code 1 and the padding nibble.
• The route: when the raw float vectors have size 0, the reader returns this view (Lucene104ScalarQuantizedVectorsReader.java:278-297), and ScalarQuantizedVectorValues.vectorValue delegates to it (:769-771). The float16 twin has the same lines (OffHeapScalarQuantizedFloat16VectorValues.java:92, :135-136; reader :326-344; main and branch_11x only).
• Everyone else pads: the writer (Lucene104ScalarQuantizedVectorsWriter.java:190, 243, 431, 942) and the quantized scorer (Lucene104ScalarQuantizedVectorScorer.java:226-230, new byte[OptimizedScalarQuantizer.discretize(values.dimension(), 2)] before its own unpackNibbles). So indexing, kNN search and merges work for odd dimensions while raw vectors are present.
• The other encodings are fine: the 1-bit and 2-bit unpackers stop at the output length (OptimizedScalarQuantizer.java:460-471, :527-551) and 8-bit / 7-bit need no unpack (OffHeapScalarQuantizedFloatVectorValues.java:129-139). An end-to-end matrix of 5 encodings x dims 1..40 fails in exactly the 20 PACKED_NIBBLE odd-dimension cases.
• The data is fine: reading the stored codes into a padded buffer and dequantizing the first dim of them works for odd dims 1..41 (worst absolute error 0.076).

• Who. Read-only searcher copies, for example NRT segment-replication replicas, that drop the full-precision vectors to save disk (the use case behind #14792 and the open issue #13158), with the Lucene104 scalar-quantized format, PACKED_NIBBLE, and an odd dimension.
• Opt-in. The default codec does not quantize; the Lucene104 scalar-quantized formats default to UNSIGNED_BYTE (Lucene104ScalarQuantizedVectorsFormat.java:144-147) and are @lucene.experimental (:97). Every odd dimension from 1 to 1023 is legal (maximum 1024, :168-170), and odd dimensions index, search and read fine while raw vectors are present.
• Reachability. No Lucene API writes the dropped-raw state; it takes file rewriting like TestLucene104ScalarQuantizedVectorsFormat.simulateEmptyRawVectors() (:614-682), and the result is an index that CheckIndex rejects (BaseKnnVectorsFormatTestCase.java:2274-2276). Two upstream efforts would make it easy to create: the open PR #15630 proposes writer support for empty full-precision vector files, and #16030 added a write-side mode that drops the float vectors (DATA_BLIND_WITHOUT_FLOATS; on main and branch_10x 2026-09-18 to 2026-09-25, never released; its feature request #16029 is still open).
• Effect. Every vectorValue(ord) of the field throws, and so do a search with a full-precision rescore query (RescoreTopNQuery.createFullPrecisionRescorerQuery) and, on main, the float16 view. Plain quantized kNN search works. Deterministic, the same with assertions on and off; no index damage, and a fixed reader reads the same files.
• Scale. One document; the flat and the HNSW format, dense and sparse fields, dims 1 to 1023.

• Smallest input. One document, dimension 3 (odd, so the 2 packed bytes hold codes in both halves plus one padding nibble; dimension 1 fails too), no compound file so the raw .vec/.vemf files can be rewritten, ByteBuffersDirectory, one thread; the same result with assertions on and off and with any seed.
• The drop helper is a copy of TestLucene104ScalarQuantizedVectorsFormat.simulateEmptyRawVectors() (:614-682; block shift 16 is that class's DIRECT_MONOTONIC_BLOCK_SHIFT). It cuts .vec to its header and footer and rewrites .vemf with the same header and field metadata but a vector data length of 0 and a vector count of 0; the quantized .veq/.vemq stay as written. The result is an index CheckIndex rejects, which upstream's own fallback tests allow for by switching CheckIndex off (BaseKnnVectorsFormatTestCase.java:2274-2276).
• Control (not part of the repro): the same test with a 4-dim vector passes on main, 10.4.0 and 10.6.0.
• Releases. The same file fails on 10.5.0, 10.5.1, 10.5.2 and 10.6.0; on 10.4.0 only the ScalarEncoding import differs (it was nested in Lucene104ScalarQuantizedVectorsFormat).
• As the regression test: this same method. Upstream it can live in TestLucene104ScalarQuantizedVectorsFormat and call that class's own simulateEmptyRawVectors(dir) (not run in that form). Let it check values too: the last line becomes assertArrayEquals(new float[] {0.1f, 0.2f, 0.3f}, v, 1f / 16), since a length check alone would also pass a broken fix that returns any 3 values. That line also fails before and passes with fix A and with fix B.

cp $REPRO/TestPackedNibbleOddDimensionWithoutRawVectors.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene104/
cd $LUCENE
./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene104.TestPackedNibbleOddDimensionWithoutRawVectors -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same exception)

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on
TestPackedNibbleOddDimensionWithoutRawVectors > testVectorValueWithOddDimension FAILED
    java.lang.AssertionError: expected the dequantized 3-dim vector, got java.lang.IllegalArgumentException: vector dimensions differ: 3 != 2 * 2
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:BF7D0792B33E1E60]:0)
        at org.apache.lucene.codecs.lucene104.TestPackedNibbleOddDimensionWithoutRawVectors.testVectorValueWithOddDimension(TestPackedNibbleOddDimensionWithoutRawVectors.java:57)

        Caused by:
        java.lang.IllegalArgumentException: vector dimensions differ: 3 != 2 * 2
            at org.apache.lucene.util.VectorUtil.int4Unpack(VectorUtil.java:315)
            at org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues.unpackNibbles(OffHeapScalarQuantizedVectorValues.java:192)
            at org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedFloatVectorValues.vectorValue(OffHeapScalarQuantizedFloatVectorValues.java:124)
            at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsReader$ScalarQuantizedVectorValues.vectorValue(Lucene104ScalarQuantizedVectorsReader.java:770)
            at org.apache.lucene.codecs.lucene104.TestPackedNibbleOddDimensionWithoutRawVectors.testVectorValueWithOddDimension(TestPackedNibbleOddDimensionWithoutRawVectors.java:54)
1 test completed, 1 failed
# gradle, assertions off (-Ptests.asserts=false): the same exception
# 10.6.0, assertions on and off: the same IllegalArgumentException
# 10.4.0, 10.5.0, 10.5.1, 10.5.2, assertions off: ArrayIndexOutOfBoundsException: Index 3 out of bounds for length 3
#   (at OffHeapScalarQuantizedVectorValues.unpackNibbles); assertions on: java.lang.AssertionError from unpackNibbles
# the same test with the 4-dim vector {0.1, 0.2, 0.3, 0.4}: passes (main, 10.4.0, 10.6.0)
# with fix A or fix B: BUILD SUCCESSFUL

• Documented capability. OffHeapScalarQuantizedFloatVectorValues.java:39-54: it dequantizes stored vectors "back to float vectors with some precision loss"; "This class is used for read-only indexes where full-precision float vectors have been dropped from the index to save storage space. Full-precision vectors can be removed from an index using a method as implemented in TestLucene104ScalarQuantizedVectorsFormat.simulateEmptyRawVectors()."
• Format contract. BaseKnnVectorsFormatTestCase.java:132 (supportsFloatVectorFallback()); :2187-2189: formats must "properly dequantize vectors and maintain accuracy within expected error bounds even when the original raw vector file is empty or corrupted"; :2255-2260: scores must not change once the raw file is emptied. These tests run for the flat format, which the repro uses; TestLucene104HnswScalarQuantizedVectorsFormat opts out (supportsFloatVectorFallback() returns false, :491-493), so the HNSW route is shown by execution only (the study's tests and a dim-1023 probe). CHANGES: #15415 under 10.4.0 New Features (lucene/CHANGES.txt:1378), #15476 (:1527), #16415 under 10.5.1 Bug Fixes (:894).
• This encoding is in scope. The fallback has an explicit PACKED_NIBBLE branch (OffHeapScalarQuantizedFloatVectorValues.java:123-124), and the Lucene104 fallback tests pick the encoding at random from all values (TestLucene104ScalarQuantizedVectorsFormat.java:81-86).
• Odd dimensions are in scope. No javadoc, CHANGES or MIGRATE entry limits parity, and upstream's own Lucene104 search and write/read tests draw dims from 4..64 (TestLucene104ScalarQuantizedVectorsFormat.java:98, :291).
• Why tests miss it. BaseKnnVectorsFormatTestCase.java:2196-2199, :2266-2269 (if (dim % 2 == 1) dim++); TestLucene104ScalarQuantizedVectorsFormat.java:452, :529 (dim = 2 * random().nextInt(1, 33)). The guard first appeared in the Lucene99 fallback test of #14792; the Lucene99 writer rejects odd dimensions for 4 bits ("is not supported for odd vector dimensions", test copy Lucene99ScalarQuantizedVectorsWriter.java:198-204). #15415 copied the guard and #15476 moved it into the shared base class, which still serves Lucene99.
• The PR author called the old size wrong. #16030 (mccullocht, merged 2026-09-18) changed both views to new byte[encoding.getDiscreteDimensions(dimension)] and the deQuantize loop bound to dequantized.length. In review, kaivalnp asked about the buffer line (2026-09-10): "I didn't get why this had to change? Is it required independently of this PR?"; mccullocht replied (2026-09-17): "This is an independent change -- it would occasionally compute the wrong size and tests would fail." #16030 was reverted on 2026-09-25 (c72bc6abf18), which restored the bug. The revert commit gives no reason; in the PR discussion dweiss named flaky CI tests and merge inputs whose wrappers defeat the data-blind mode's raw-vector detection (2026-09-23), and mccullocht agreed ("Rolling back is fine"); neither concerns this buffer.

Strongest case for "not a bug" (a cross-model reviewer, OpenAI GPT-5.6, who reproduced it and agreed on the cause): no production API drops raw vectors, only a test hook does; CheckIndex "rightly rejects" such an index (BaseKnnVectorsFormatTestCase.java:2275); the class is @lucene.internal, the format @lucene.experimental; and the fallback tests deliberately use even dimensions.

Why it fails: the committers designed the fallback knowing this. msokolov, reviewing #14792: "it provides the ability to read an index that we don't actually provide any way to create!", and they asked for the usage note that became the class javadoc ("full precision vectors can be removed from an index using a method like shown in the tests ... should be treated as a read-only index"). benwtrent on #14792: "When the vec file isn't present, the format should still return a FloatVectorValues, and when folks call float[] getVector(int ord), they get the de-quantized representation of the vector." mikemccand backed #15476 as "the test infra to confirm future KNN Codecs on 10.x also preserve this capability". Every upstream fallback test turns CheckIndex off and still requires correct reads; the failure comes through public LeafReader#getFloatVectorValues and RescoreTopNQuery; and the even-dimension guard comes from Lucene99's write-side restriction, not from a Lucene104 limit.

Two validated options, both applied in a scratch worktree of main 645d4fe14a20 and run with gradle; both also make the study's 5-encoding x 40-dimension matrix read back every vector. Not recommended: rejecting odd dimensions for PACKED_NIBBLE at write time (breaks a configuration that works with raw vectors present; existing segments still fail), or only enlarging the buffer (moves the failure into deQuantize, which would index dequantized[3] and centroid[3] of length 3). Test gap: the fallback tests could draw odd dimensions for the Lucene104 formats; the dim % 2 guard is needed only by Lucene99.

Allocate the unpack buffer with the padded length in both views, and loop deQuantize over the output length. These are the lines of #16030 by mccullocht (the deQuantize comment is theirs; the comment on the buffer line is not from #16030), without that PR's unrelated COSINE assertion changes, and the "wrong size" diagnosis is theirs too.

// OffHeapScalarQuantizedFloatVectorValues.java:96 and OffHeapScalarQuantizedFloat16VectorValues.java:92
-    this.unpackedByteVectorValue = new byte[dimension];
+    this.unpackedByteVectorValue = new byte[encoding.getDiscreteDimensions(dimension)];

// OptimizedScalarQuantizer.deQuantize, float[] and short[] overloads (:262, :292)
-    for (int h = 0; h < quantized.length; h++) {
+    // The quantized input may hold rounded-up dimensions for packed encodings; the output length
+    // defines how many are real.
+    for (int h = 0; h < dequantized.length; h++) {

Trade-offs

• Four changed lines plus comments; no extra copy; matches the writer and the scorer.
• Changes the loop bound of a public static helper. Lucene's own callers pass equal lengths; an outside caller passing an output longer than its input would now get an ArrayIndexOutOfBoundsException instead of a partly filled array.
• The 1-bit and 2-bit encodings also get the padded buffer; their unpackers fill it from the padded stored bytes and only the first dimension codes are read (the matrix check below covers them).

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: the minimal test fails before the change and passes after it, with assertions on and off.
• Seed DEADBEEF: TestLucene104ScalarQuantizedVectorsFormat (65 tests, 1 skipped), TestLucene104HnswScalarQuantizedVectorsFormat (64, 3 skipped), TestOptimizedScalarQuantizer (7): 0 failures.
• Seed 5EED1234: every test class in org.apache.lucene.codecs.lucene104.* and org.apache.lucene.util.quantization.*: 199 tests, 6 skipped, 0 failures.
• Standalone, the three patched files shadowed in: the study's 5 dropped-raw tests pass (float32 read, full-precision rescore, the float16 twin, two controls; 3 of them fail without the fix), and its matrix of 5 encodings x dims 1..40 reads back all 200 cases.
• Not run: the whole :lucene:core:test, check, or the backward-codecs suites.

Keep unpackedByteVectorValue = new byte[dimension] and add a PACKED_NIBBLE-only padded scratch in both views; deQuantize stays as it is. This is the study verifier's fix sketch, with the per-call allocation hoisted into a field.

// in both views
// PACKED_NIBBLE only: int4Unpack also writes the padding nibble of an odd dimension
private final byte[] paddedNibbles;
...
this.paddedNibbles =
    encoding == ScalarEncoding.PACKED_NIBBLE
        ? new byte[encoding.getDiscreteDimensions(dimension)]
        : null;
...
case PACKED_NIBBLE -> {
  OffHeapScalarQuantizedVectorValues.unpackNibbles(byteValue, paddedNibbles);
  System.arraycopy(paddedNibbles, 0, unpackedByteVectorValue, 0, dimension);
}

Trade-offs

• Touches only the two package-private views; no public method changes behaviour.
• One extra copy of dimension bytes per read, only on the PACKED_NIBBLE fallback path, and one more small array per view.

What the checks validated

• Gradle, seed DEADBEEF: the minimal test, TestLucene104ScalarQuantizedVectorsFormat and TestLucene104HnswScalarQuantizedVectorsFormat: 130 tests, 4 skipped, 0 failures.
• Standalone: the study's 5 dropped-raw tests pass and its 200-case matrix reads back every vector.

• The study's quantization unit found it by code reading while modelling; its report lists "code review + exhaustive layout check", and its own notes credit the even-dimension test guard as the pointer. No solver step produced it.
• An independent verifier wrote its own drop helper and 7 tests on the HNSW format: dim 3 fails for vectorValue() and for a search with a RescoreTopNQuery.createFullPrecisionRescorerQuery query, the float16 twin fails, the dim 4 control and plain kNN pass, the 200-index matrix isolates PACKED_NIBBLE x odd dims, and the stored codes decode correctly with a padded buffer (odd dims 1..41).
• The hunt then minimised it to the one-method test, re-ran it on main and on every release with the code, and tie-broke a cross-model review (section "Independent checks").