Scalar-quantized vectors with COSINE: two opposite vectors make the centroid zero, so flush or merge throws and the IndexWriter closes

For COSINE, the scalar-quantized (OSQ) vectors formats quantize every vector against one centroid per segment. At flush it is the mean of the segment's vectors, each scaled to length 1; a merge recomputes it from the segments' stored centroids or from the raw stored vectors; in every case the mean is then scaled to length 1. When the vectors cancel out exactly, that mean is {0, 0}. This happens with {1, 0} and {-1, 0} in one flush, or with one-document segments {0.6, 0.8} and {-0.6, -0.8} in one merge. VectorUtil.l2normalize then throws IllegalArgumentException: Cannot normalize a zero-length vector. At flush the exception aborts the in-memory segment and closes the IndexWriter, and the uncommitted documents are lost. At merge the writer closes, and merging the same segments again fails the same way, also in a fresh writer (a merge whose vectors do not cancel as a whole, for example one that also takes in a segment {1, 0}, succeeds). Each vector is legal on its own, and the default vectors format and the OSQ format with DOT_PRODUCT index and search the same documents. The opt-in formats are affected from 10.2.0, which introduced the centroid with the binary-quantized formats, through 10.5.2, 10.6.0 (by reading) and main. A small fix (+22/-5 lines) keeps a zero centroid instead, which the quantizer already handles for DOT_PRODUCT.

• OSQ formats. Lucene104ScalarQuantizedVectorsFormat (flat) and Lucene104HnswScalarQuantizedVectorsFormat (HNSW), opt-in per field through a codec subclass; in 10.2.x and 10.3.x the same code is in the Lucene102 binary-quantized formats. The default codec uses the unquantized Lucene99HnswVectorsFormat.
• Centroid. One mean vector per segment and field; each vector is quantized as its difference from it, and the query is quantized against the same centroid at search time.
• Tragic event. An exception after which IndexWriter closes itself (isOpen() == false, getTragicException() set).

1. The sum. Every added vector goes to accumulate (addValue, Lucene104ScalarQuantizedVectorsWriter.java:740; the columnar bulk path addDenseValues, :750). For COSINE, accumulateFloats divides it by its own norm and adds it to dimensionSums (:791-804): {1, 0} + {-1, 0} = {0, 0}.
2. Flush. flush calls computeCentroid() (:147), which divides by the count and calls VectorUtil.l2normalize(centroid) (:818). Its only guard is vectorCount > 0 (:813); its javadoc promises "the mean of the accumulated vectors, unit-length for COSINE".
3. Merge, stored centroids (flat format): the count-weighted mean of the segments' centroids, {0.6, 0.8} + {-0.6, -0.8} = {0, 0}, then VectorUtil.l2normalize(mergedCentroid) (:631); only totalVectorCount == 0 is guarded (:622, #15213).
4. Merge, recomputed centroid (any segment with deletions, and always for the HNSW format, whose reader does not expose a stored centroid, :569-575, :614): the mean of the live raw stored vectors, which are not scaled to length 1 (accumulateCentroid, :660-680), then VectorUtil.l2normalize(centroid) (:655); only count == 0 is guarded (:648). Since #16764 the sum skips deleted documents, so on main deleting {0, 1} from {1, 0}, {-1, 0}, {0, 1} leaves a zero mean.
5. The throw. VectorUtil.l2normalize(v) is l2normalize(v, true) (VectorUtil.java:206-209), which throws "Cannot normalize a zero-length vector" when the squared norm is 0 (DefaultVectorUtilSupport.java:479-487; PanamaVectorUtilSupport.java:1617-1624).
6. The writer closes. At flush, DocumentsWriterPerThread records an aborting exception, aborts the in-memory segment and fires the tragic event (DocumentsWriterPerThread.java:585-604); at merge, handleMergeException rethrows the exception (only an aborted merge is swallowed there) and IndexWriter.merge calls tragicEvent(t, "merge") (IndexWriter.java:5030-5057, :5109-5113).

On main, the 5-test JUnit fails 4 of 5 (flush; merge with the flat format; merge with the HNSW format; merge after a delete) with assertions on and off, and with the Panama and the scalar vector code; the control passes. The study's test class, re-run on main: after the failed flush the directory holds 0 of the 2 documents; after the failed merge the index still has its 2 committed documents, and a second, fresh writer fails forceMerge(1) the same way; at flush, all five ScalarEncodings fail, flat and HNSW, and so do a random 768-d v with -v and a signed one-hot set {3,0,0}, {-7,0,0}, {0,0.5,0}, {0,-2,0} (at flush the magnitudes do not matter, because each vector is scaled to length 1 first). The trigger is exact: {1, 0}, {-1, 1e-6} commits fine, and two mirrored 3-document segments broke the flat format's merge but not the HNSW format's merge, which recomputes the mean from the raw vectors one by one instead of from the two stored centroids, so float rounding leaves a non-zero residue. For the same reason magnitudes can matter at merge: {2, 0} and {-1, 0} make the flush throw, and as two one-document segments the flat-format merge, but the HNSW-format merge of those segments succeeds.

• Who is hit. Only fields with an OSQ (or, in 10.2-10.3, binary-quantized) format and COSINE; the default codec does not quantize. The trigger is a set of vectors that cancel exactly, in float32 and in every dimension. What has to cancel depends on the path: at flush, the segment's vectors, each scaled to length 1; at a merge with stored centroids (flat format, no deletions), the segments' unit centroids, weighted by their vector counts; at a merge that recomputes (always for the HNSW format, and for any merge with deletions), the raw live vectors of all the merged segments. Plausible: tiny segments (frequent NRT refreshes, small test corpora) holding a v, -v pair; sign-symmetric data (plus/minus one-hot vectors, feature differences, v/-v augmentation); on main and 10.6.0, a delete that leaves such a pair; on main, the new columnar addBatch bulk path too. Exact cancellation is expected to be rare in large segments of learned embeddings (no survey of real data was done).
• Default or opt-in. Opt-in format plus COSINE (supported; the javadoc prefers DOT_PRODUCT on normalized vectors). No assertions or flags needed: the same with -ea and -da, Panama and scalar vector code.
• Scale. Two documents: one flush, or one merge of two one-document segments.
• The damage. Flush: the commit throws, the writer closes and every document buffered since the last commit is lost. Merge: the writer closes; the committed index stays readable, and merging the same two segments again fails the same way, also in a fresh writer. A merge whose own vector set does not cancel succeeds: forceMerge(1) of {0.6, 0.8}, {-0.6, -0.8} and a third segment {1, 0} works with both formats. After the failed merge the index opened with its 2 documents and kNN found both; CheckIndex was not run. Other ways to recover were not tested.
• Workaround. Normalize vectors in the application and use DOT_PRODUCT, the javadoc's preferred way: the OSQ format with DOT_PRODUCT indexes and searches the same pair (control test).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected commit of two COSINE vectors to succeed, got java.lang.IllegalArgumentException: Cannot normalize a zero-length vector
       org.apache.lucene.codecs.lucene104.TestOsqAntipodalZeroCentroid.testAntipodalCosineVectorsCommit(TestOsqAntipodalZeroCentroid.java:46)
     Caused by: java.lang.IllegalArgumentException: Cannot normalize a zero-length vector
       org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport.l2normalize(PanamaVectorUtilSupport.java:1621)
       org.apache.lucene.util.VectorUtil.l2normalize(VectorUtil.java:229)
       org.apache.lucene.util.VectorUtil.l2normalize(VectorUtil.java:207)
       org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsWriter$FieldWriter.computeCentroid(Lucene104ScalarQuantizedVectorsWriter.java:818)
       org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsWriter.flush(Lucene104ScalarQuantizedVectorsWriter.java:147)
       [...]

• The input is legal. KnnFloatVectorField rejects a COSINE vector only when every component is 0 (KnnFloatVectorField.java:53-56); otherwise it checks only for null and empty vectors (:43-49) and non-finite values (:105). COSINE is supported: "The similarity score is normalised to assure it is positive" (VectorSimilarityFunction.java:76-82); DOT_PRODUCT on normalized vectors is "preferred", a recommendation, not a restriction. The OSQ formats (@lucene.experimental, Lucene104ScalarQuantizedVectorsFormat.java:97, Lucene104HnswScalarQuantizedVectorsFormat.java:47) state no limit on which vectors may share a segment, and computeCentroid's javadoc documents no exception (Lucene104ScalarQuantizedVectorsWriter.java:806-809).
• The sibling case was called a bug and fixed. #15209: a merge with zero samples gave a NaN centroid; benwtrent: "it does smell like a bug. The centroid is nan, which indicates something is wrong with the centroid merging logic somewhere." #15213 (CHANGES: "Don't normalize centroids with zero samples, which caused divide-by-zero") added only count == 0 guards. This bug is the remaining case: a non-empty set with a zero mean.
• A zero centroid is a valid state. In the #16030 review its author, mccullocht, wrote: "If the center is already a zero vector that is geometrically equivalent to our data blind configuration. Centering wouldn't subtract anything from the vector, and for angular similarity center_dot would be zero." #16030 (cab6cfcb604) relaxed exactly the two OptimizedScalarQuantizer COSINE assertions to || VectorUtil.dotProduct(centroid, centroid) == 0; it was reverted (c72bc6abf18, 2026-09-25) for flaky tests and reader-wrapping problems unrelated to that change. The sandbox DedupQuantizer quantizes normalized vectors "against a zero centroid" with DOT_PRODUCT, "without the unit-centroid requirement of a cosine-configured quantizer" (DedupQuantizer.java:116-118).
• In exact arithmetic the math works for any centroid. COSINE and DOT_PRODUCT share the score branch (Lucene104ScalarQuantizedVectorScorer.java, quantizedScore), and the study's SMT proof A5 shows the correction identities hold for any centroid (exact reals, dimensions 1 to 6; it says nothing about float rounding or recall). With the recommended fix the two documents score 0.8 and 0.19999999, the same as the unquantized format.
• History. The three unguarded calls came with the first centroid-based format, #14078 (10.2.0); no test, javadoc, CHANGES or MIGRATE entry says a cancelling COSINE set is unsupported.

The strongest case for "intended" or "won't fix": the formats are @lucene.experimental and opt-in; the COSINE javadoc tells users to normalize and use DOT_PRODUCT; a zero mean of unit vectors has no direction, so throwing is arguably the honest answer; Lucene's threat model leaves input sanitizing to the application; and real embeddings practically never cancel exactly.

Why it fails: every vector passes the field's own validation, and what breaks is a property of a set of documents in one segment or one merge, which flushes, merge selection and deletes decide, so the application can neither see nor validate it. The result is not a ranking trade-off but a closed writer, lost documents and a merge of the cancelling segments that keeps failing, while the same quantizer handles the same vectors with a zero centroid under DOT_PRODUCT. Upstream fixed the sibling count == 0 case as a bug, and the #16030 author treated a zero centroid as valid. @lucene.experimental allows incompatible API changes, not closing the writer on valid input. The cross-model refute review (OpenAI GPT-5.6) rated the intended-behaviour argument weak. A later cross-model fact check rated it moderate: a zero centroid means the segment is quantized uncentered (data-blind), which may cost recall; upstream has not adopted zero-centroid semantics, because #16030 was reverted; and the #16030 remark is one contributor's view. That bears on which fix to pick, not on whether closing the writer on legal input is acceptable.

Two fixes were built and validated through gradle in a scratch worktree of a9779ef3c193; two other options are explained. All three throw sites need the same guard; the question is what a COSINE centroid should be when the mean has no direction.

A helper normalizeCosineCentroid replaces the three VectorUtil.l2normalize calls in Lucene104ScalarQuantizedVectorsWriter. It sets the centroid to all zeros when its squared norm is 0, the exact condition on which l2normalize throws, and otherwise normalizes as before. The two COSINE assertions in OptimizedScalarQuantizer also accept an all-zero centroid, which is the assertion change of #16030. The patch is +22/-5 lines in 2 files.

static void normalizeCosineCentroid(float[] centroid) {
  if (VectorUtil.dotProduct(centroid, centroid) == 0) {
    Arrays.fill(centroid, 0f); // also clears a mean whose squared norm underflows to zero
  } else {
    VectorUtil.l2normalize(centroid);
  }
}

// OptimizedScalarQuantizer.multiScalarQuantize and scalarQuantize
assert similarityFunction != COSINE
    || VectorUtil.isUnitVector(centroid)
    || VectorUtil.dotProduct(centroid, centroid) == 0;

Trade-offs

• No change to the file format, and no change to the bytes or results of any segment that worked before. A cancelling segment gets centroid {0, ..., 0} and centroidDP 0, the same as DOT_PRODUCT quantization of those unit vectors.
• It changes the assertions of a public utility class, though assertions only.
• Not covered: a mean whose squared norm is subnormal but not 0 still normalizes with poor precision. That case is extreme and was not tested.
• A zero centroid means the cancelling segment is quantized uncentered (data-blind). The two-document probe shows exact scores, but recall on realistic cancelling or near-cancelling segments was not measured.

What the checks validated

• Stock: the test and its columnar companion fail 5 of 6, with assertions on and off.
• With the fix they pass 6 of 6, with assertions on and off, and 50 of 50 with seed CAFEBABE and tests.iters=10.
• A probe over all five encodings, flat and HNSW: the flat format stores centroid [0.0, 0.0] (the probe cannot read the HNSW format's centroid). For both formats, kNN {0.6, 0.8} scores 0.8 and 0.19999999 after the flush, and 1.0 and 0.0 after the merge, identical to the unquantized format.
• The closest suites pass: 25 classes and 314 tests (codecs.lucene104.*, util.quantization.*, TestVectorUtil, TestKnnGraph, TestHnswMergeAbort, the rescore and full-precision vector tests, among others), with assertions on and off.
• All of :lucene:core:test passes: 750 classes, 8,895 tests, 0 failures, 322 skipped. :lucene:core:check -x test passes too.

The same helper, but a zero mean becomes the first unit basis vector {1, 0, ..., 0}. OptimizedScalarQuantizer is left alone. The patch is +17/-3 lines in 1 file.

Trade-offs

It keeps the "COSINE centroid is unit length" invariant without touching the quantizer, but the choice of centroid is arbitrary. In the probe, scores came out slightly off the unquantized ones: 0.2000001 instead of 0.19999999, and up to 6e-8 instead of 0.0.

What the checks validated

The tests pass 6 of 6 with assertions on and off. The closest suites (25 classes, 314 tests) pass with assertions on; they were not run with assertions off for this option, and neither were the full core suite and check.

Switch the three calls to the non-throwing l2normalize(centroid, false) and change nothing else.

Trade-offs

By code reading (not executed), the zero centroid then trips the quantizer's unit-centroid assertions under -ea (OptimizedScalarQuantizer.java:122, :190), at flush and at query time. It also leaves a non-zero mean whose squared norm underflows unnormalized. It is the recommended option without the assertion change and the underflow handling. The withdrawn #16100 used this call for a different trigger, all-zero input vectors, but not alone: its diff also relaxed both quantizer COSINE assertions to accept a zero centroid (VectorUtil.isZeroVector(centroid) || VectorUtil.isUnitVector(centroid), and the same for the vector), switched three other l2normalize calls to the non-throwing form and skipped zero-length vectors when summing. Its centroid change is therefore close to the recommended option without the underflow handling.

Drop the final scaling and keep each path's un-normalized mean: at flush, the mean of the unit-scaled vectors; at merge, the weighted mean of the stored centroids, or the mean of the raw vectors.

Trade-offs

It changes the centroid, the quantized bytes and possibly the recall of every non-zero COSINE segment, to fix a corner case. It would need benchmarks.

Method: SMT (Z3) floating-point precondition analysis, the study's "formal" tier, run 3, experiment "quantization-smt", on main 7cecf79523.

• An AI tool read the OSQ writer and stated proof obligation PO-A2: "the mean of the unit-normalized vectors of a segment is non-zero", the precondition of l2normalize(throwOnZero=true) at the flush and both merge sites.
• Z3 refuted it with a pair that is not antiparallel, with extreme magnitudes, whose normalized sum is exactly 0 (sat, 491 s). The plain antipodal query returned UNKNOWN after 614 s, so a JUnit test covered {1, 0}, {-1, 0} and {0.6, 0.8} / {-0.6, -0.8}. The study verifier did not re-check the non-antiparallel witness, because the exact negations suffice.
• A separate AI verifier wrote its own 9-test class and ran it on main (all 9 reproduce) and on the 10.5.2 jars (8 of 9; the delete case needs #16764, which 10.5.2 lacks); a flush probe reproduced the bug on the 10.2.0, 10.3.2 and 10.4.0 jars.
• This hunt:
  • re-ran the bug on a9779ef3c193 and found the columnar route;
  • ran a cross-model refute review (OpenAI GPT-5.6), which wrote and ran its own test;
  • bisected the introducing commit;
  • searched for prior reports;
  • validated two fixes.