COSINE vectors whose float32 norm underflows to zero pass validation, then score +Infinity or NaN and break OSQ segments

KnnFloatVectorField rejects a COSINE vector only when every component is exactly 0, but every in-tree Java COSINE scorer and the quantized writer compute the squared norm in float32. For a vector whose components are all at most about 2.6e-23 (2^-75), such as {1e-30, 0}, each square rounds to 0, so the vector is accepted yet has a computed norm of exactly 0, like the zero vector the check is meant to keep out. Its COSINE score is then dot / 0: +Infinity, NaN or 0. On the default codec it takes the top hit above the identical vector (score 1.0), a NaN in a batch of up to 64 scored vectors makes the search skip the whole batch (a 4-document segment returns 0 hits), and CheckIndex at level 2 or higher (its logical integrity checks; the default level 1 checks only checksums) reports an index built only from accepted vectors as broken. On the opt-in Lucene104 scalar-quantized (OSQ) formats it turns the segment's centroid into NaN: every quantized score in the segment is NaN (a 3-document segment returns no hits), and every merge that includes it throws and closes the IndexWriter until the document is deleted. A slightly larger one-component vector has a subnormal, badly rounded norm, so its scores can exceed 1 (1.1103). A tiny query vector returns 0 hits too. The float32 norms date from the commit that added COSINE, so every release since 9.0.0 is affected.

• float32 squared norm: x1*x1 + x2*x2 + ... computed in float, as all of Lucene's in-tree Java COSINE code does. The smallest positive float is 2^-149 (about 1.4e-45), so a square below about 7e-46 rounds to 0.
• Subnormal: a float below Float.MIN_NORMAL (2^-126, about 1.2e-38). It has fewer significant bits, so a norm in that range is badly rounded.
• OSQ: the opt-in Lucene104 scalar-quantized vector formats (Lucene104HnswScalarQuantizedVectorsFormat, Lucene104ScalarQuantizedVectorsFormat). For COSINE they normalize each vector and quantize it around a per-segment centroid.

1. Validation checks exact zeros only. KnnFloatVectorField rejects COSINE vectors with VectorUtil.isZeroVector (KnnFloatVectorField.java:53-56, :146-150, :172-175), true only if every component == 0 (VectorUtil.java:515-523). Columnar addBatch does the same (ColumnValidation.java:364-381). Query targets get only checkFinite (KnnFloatVectorQuery.java:97).
2. The norms are float32. DefaultVectorUtilSupport.cosine (:79-117) and PanamaVectorUtilSupport.cosine (:177-199) accumulate fma(x, x, norm) in float and return (float) (sum / Math.sqrt((double) norm1 * (double) norm2)); so do the off-heap bulk scorers (MemorySegmentBulkVectorOps, java25, :180-206). (The optional native provider's external lucene_cosineFloat was not checked.) If every |x_i| <= 2^-75, the norm is exactly 0 and the result is sum / 0: +Infinity, -Infinity or NaN. For a one-component vector {x, 0} with 2^-75 < x < 2^-63 the norm is subnormal and badly rounded, up or down, so scores can exceed 1 (1.1103) or fall below the true value (0.947 for {4.735e-23, 0}).
3. The score transform passes them through. COSINE.compare is normalizeToUnitInterval(cosine(v1, v2)) (VectorSimilarityFunction.java:84-86), which is Math.max((1 + value) / 2, 0) (VectorUtil.java:461-463). The finiteness assert at VectorUtil.java:105 is off in production.
4. One NaN hides a batch. RandomVectorScorer.bulkScore returns the batch maximum via Math.max, NaN if any score is NaN (RandomVectorScorer.java:47-54). The exhaustive search scores 64 vectors at a time and keeps a batch only if that maximum is > minCompetitiveSimilarity() (Lucene99HnswVectorsReader.java:81, :396, :408), which is false for NaN; HNSW expansion uses the same test (HnswGraphSearcher.java:341-343). At level 2 or higher (CheckIndex.java:1114, :1141; the command-line default is level 1, checksums only), CheckIndex runs kNN searches with stored vectors and fails when one returns no hit (CheckIndex.java:3112-3137).
5. OSQ centroid. As each document is added, Lucene104ScalarQuantizedVectorsWriter.FieldWriter.addValue calls accumulate (:841-843), which divides the vector by (float) Math.sqrt(dotProduct(v, v)) = 0 and adds the Infinity or NaN to the in-memory centroid sums (:767-780). At flush, computeCentroid (:786-798) averages and normalizes them; l2normalize throws only when the squared norm == 0, false for NaN (DefaultVectorUtilSupport.java:479-498), so a NaN centroid is written with the segment and every quantized score in it is NaN (0 hits in the 3-document repro; the study's 2,001-document HNSW segment returned 1 hit, scored NaN). CheckIndex at level 2 or higher fails on the OSQ-HNSW segment; the flat OSQ format passes, because CheckIndex does not search flat formats (CheckIndex.java:3106-3110).
6. OSQ merge. The merge re-normalizes each raw COSINE vector (NormalizedFloatVectorValues.java:51, via Lucene104ScalarQuantizedVectorsWriter.java:317-318); for the tiny vector that throws IllegalArgumentException("Cannot normalize a zero-length vector") (DefaultVectorUtilSupport.java:483), and IndexWriter treats this merge exception as tragic and closes (IndexWriter.java:5109-5113; only a MergeAbortedException on a non-external merge is swallowed, :5043-5053). A buffered uncommitted document is lost; a second writer fails the same way; deleting the document lets the merge succeed (executed probe).

Main ea57360912ed, assertions off, one segment each:

In the study's larger runs: in one 3,000-document 16-d segment, a document with all components 1e-30 was the top hit, scored +Infinity, for 86 of 200 random queries (91 had a positive dot product with it); one Float.MIN_VALUE document added to a 200-document segment cut exhaustive recall@10 from 1.0 to 0.7 (in a 3,000-document HNSW graph: 0.600 without, 0.603 with, no loss); one document with a single 1e-25 component made all 2,001 quantized scores of its OSQ segment NaN (kNN returned 1 hit, scored NaN).

• Who is exposed. Every float32 COSINE field with the default codec (Lucene99 HNSW float), through the public KnnFloatVectorField constructor and default settings; fields that opt into the Lucene104 OSQ formats get the worse effects. Not exposed: byte vectors (integer arithmetic, so the exact-zero check is complete), FLOAT16 fields (the smallest float16 squared, 2^-48, is a normal float32), and DOT_PRODUCT, EUCLIDEAN or MAXIMUM_INNER_PRODUCT fields, which do not divide by the norm.
• The trigger. Every component at most about 2.6e-23 for +Infinity/NaN, or a norm below about 1e-19 for finite scores above 1. It is not expected from ordinary normalized embeddings (no survey of real models was done). Plausible sources: a buggy upstream pipeline, a computation meant to give zero that leaves rounding residue, an application that adds a tiny epsilon to get past the "zero vector not allowed" error, or input an untrusted user controls.
• Scale. One document is enough, at any index size. It scores +Infinity for every query whose dot product with it is positive and ranks first whenever the search reaches it: always in exhaustive search; for 86 of the 91 such queries (out of 200) in the verifier's 3,000-document HNSW graph. The NaN hiding affects the exhaustive search of small segments most (batches of 64, so up to 63 other documents); in a 3,000-document HNSW graph the verifier saw no recall loss (recall@10 0.600 without, 0.603 with).
• What you see. Nothing at index time and nothing at search time: wrong top hits (+Infinity or 1.1103 beat an exact 1.0), missing hits, and, at CheckIndex level 2 or higher, a CheckIndex error on an index built only from accepted input. On OSQ: a segment whose quantized scores are all NaN (no hits in the small repro), and an IndexWriter that closes on the first merge that includes it (buffered uncommitted documents lost); a second writer failed the same way, and deleting the document unblocked the merge.
• Existing indexes. Validation cannot repair segments already on disk; deleting the document lets the OSQ merge succeed again.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError
       org.apache.lucene.util.VectorUtil.cosine(VectorUtil.java:105)
       org.apache.lucene.index.VectorSimilarityFunction$3.compare(VectorSimilarityFunction.java:85)
       org.apache.lucene.codecs.hnsw.DefaultFlatVectorScorer$FloatVectorScorer.score(DefaultFlatVectorScorer.java:246)
       org.apache.lucene.util.hnsw.RandomVectorScorer.bulkScore(RandomVectorScorer.java:50)
       org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsReader.search(Lucene99HnswVectorsReader.java:408)
       org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsReader.search(Lucene99HnswVectorsReader.java:324)
       org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader.search(PerFieldKnnVectorsFormat.java:329)
       org.apache.lucene.index.CodecReader.searchNearestVectors(CodecReader.java:294)
       org.apache.lucene.search.KnnFloatVectorQuery.approximateSearch(KnnFloatVectorQuery.java:118)
       [...]

• The contracts. VectorSimilarityFunction.COSINE (VectorSimilarityFunction.java:76-81): "The similarity score is normalised to assure it is positive." normalizeToUnitInterval (VectorUtil.java:452-459): "Converts a dot product or cosine similarity value to a normalized score in the [0, 1] range ... @return a normalized score between 0 and 1". VectorUtil.cosine asserts Float.isFinite(r) (VectorUtil.java:105).
• The existing check exists for this reason. KnnFloatVectorField throws "zero vector not allowed with cosine similarity function" (:55, :174; :149 says "... cosine similarity configured in field type"); TestField.testKnnFieldZeroVectors and testKnnFieldSetVectorValueZeroVectors (TestField.java:738-812) assert it. No test covers a non-zero vector whose norm underflows, and no javadoc, CHANGES.txt or MIGRATE.md entry states a minimum magnitude.
• Upstream intent. On #15540 (open, type:bug) mikemccand asked: "Could we, if VectorSimilarity.COSINE vector distance is being used, validate that the vector's length is non-zero, and throw a clear exception up front, and prevent the vector from being indexed (so index does not become corrupt)?"; benwtrent: "I agree, cosine should be validated." The tiny vector's computed length is zero. On #16099 benwtrent requested changes: "this is a bad idea, adding unnecessary overhead to production paths for something that is data input validation and is obviously incorrect data."
• Precedent. #12700 / #12727 treated even slightly negative vector scores as a bug.

The strongest case for "intended" or "won't fix": the input is garbage that ordinary embedding models are not expected to produce; KnnFloatVectorField is marked @lucene.experimental (KnnFloatVectorField.java:38); Lucene's threat model leaves sanitizing input to the application (SECURITY.md line 28); the overflow mirror image, a 1e17 cap on float components (#12373), was closed for "no consensus to impose this limit"; COSINE is documented as the discouraged path ("the preferred way ... is to normalize all vectors to unit length, and instead use DOT_PRODUCT"); normalizeToUnitInterval calls its input "typically" in [-1, 1]; and a maintainer may object to rejecting mathematically non-zero vectors at a Float.MIN_NORMAL threshold.

Why it fails: Lucene already chose to validate COSINE inputs that cannot be scored (#15751, #16098, #16708), and the committers on #15540 asked for exactly "the vector's length is non-zero"; the check implements "all components are 0" instead, which is not what the arithmetic needs. The damage is not confined to the bad document: it hides up to 63 others, makes every quantized score of an OSQ segment NaN, blocks merges and makes CheckIndex (level 2 or higher) report corruption of an index built only from accepted input. #12373 was a fixed component cap for every similarity on the overflow side; this needs a COSINE-only check on the computed norm that costs one comparison for an ordinary vector, and rejecting only a computed norm of exactly 0 is a valid narrower choice. @lucene.experimental allows API changes; it does not set aside the documented score range. "Typically in [-1, 1]" does not cover +Infinity or NaN, and the code asserts finiteness itself. A cross-model refute review rated the intended-behaviour argument weak; the later cross-model fact check rated it moderate, not enough to make the behaviour correct. Verdict: a defect.

Validation is the layer upstream asked for (#15540, #16099). One option was built and validated in a scratch worktree at ea57360912ed (removed afterwards); the others are untested here or were tested by the study as shadow patches.

+27/-9 lines in VectorUtil.java, KnnFloatVectorField.java and ColumnValidation.java; git apply --check is clean at ea57360912ed. The new public VectorUtil.isZeroOrSubnormalNorm(float[]) replaces isZeroVector in the three KnnFloatVectorField COSINE checks and the exact-zero test in ColumnValidation.checkFloatVectorValue. The error messages are unchanged; isZeroVector is unchanged.

  public static boolean isZeroOrSubnormalNorm(float[] v) {
    double squaredNorm = 0;
    for (float value : v) {
      if (Math.abs(value) >= 0x1p-63f) {
        return false; // value * value >= Float.MIN_NORMAL on its own
      }
      squaredNorm += (double) value * value;
    }
    return squaredNorm < Float.MIN_NORMAL; // false for NaN
  }

Trade-offs

• Threshold. Rejecting only a norm of exactly 0 removes +Infinity and NaN but keeps subnormal norms, which still score above 1 ({4.57e-23, 0} vs {1, 0}: 1.1103). < Float.MIN_NORMAL on the double sum guarantees an exact squared norm of at least Float.MIN_NORMAL for every accepted COSINE vector (not quite a normal float32 norm; see Precision). It rejects slightly more than OSQ needs ({2^-74, 0} works on OSQ today).
• Cost. The loop stops at the first component with |x| >= 2^-63, usually the first one; only vectors that are zero or tiny in every leading component are summed. Off the search path. No benchmark was run.
• Precision. The sum is in double; the codecs' float32 fma sum rounds each square to a multiple of 2^-149 below 2 * Float.MIN_NORMAL, so it can be lower by up to about half a unit of 2^-149 per component. Executed counterexample: a 1,024-d vector the patch accepts (exact squared norm Float.MIN_NORMAL + 0.0078 x 2^-149) has a float32 squared norm of Float.MIN_NORMAL - 502 x 2^-149 and scores 1.000015 against a unit query in its direction. Small, but not zero; a margin such as < 2 * Float.MIN_NORMAL would close it (not built or tested).
• API. Adds one public VectorUtil method (ColumnValidation is in another package). TestField and TestColumnBatchVectorColumn check the substrings "zero vector not allowed" and "zero vector at batch doc N", so a rewording that keeps them ("... zero or near-zero vector not allowed ...") needs no test change.
• Scoring unchanged. VectorUtil.cosine and COSINE.compare called directly with a tiny vector still return +Infinity, NaN or scores above 1; the patched scoring test passes because it skips inputs the field rejects.
• Limits. Does not repair segments already on disk (the argument of #16100), and does not cover query vectors (option B).

What the checks validated

• Stock: the 7 repro tests and the new columnar case fail (28 tests with TestColumnBatchVectorColumn, 8 failed), assertions on and off.
• Patched: 28 tests, 1 failed, assertions on and off; the failure is the query-side test, which this option does not address.
• Patched, closest suites (org.apache.lucene.document.*, codecs.lucene104.*, codecs.lucene99.*, TestVectorUtil, TestKnnFloatVectorQuery, TestVectorSimilarityValuesSource, TestKnnGraph, TestCheckIndex): 985 tests, 0 failures, 41 skipped, assertions on and off (78 test classes match these filters, counted in the source tree).
• Patched, broader core suites *Knn*, *Vector*, *Hnsw*, *Quantiz*, assertions on: 1,508 tests, 0 failures, 55 skipped.
• :lucene:core:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• The study's variant of the same idea (float32 fma sum, a helper inside KnnFloatVectorField) passed the 6 index-side tests through the harness, and an independent reviewer's own test and TestField's testKnnField* tests.

Reject a COSINE query target with a zero or subnormal norm (or return no hits on purpose). The query does not know the field's similarity when it is built, so the check has to run per segment where the FieldInfo is known: in KnnFloatVectorQuery.approximateSearch and createVectorScorer, or once in CodecReader.searchNearestVectors (which misses exact search).

Trade-offs

Also changes today's answer for an exact-zero query {0, 0} (0 hits on the default codec; an IllegalArgumentException from the OSQ scorer). Query-time zero vectors were mentioned on #15540, so this part is partly known and needs a decision upstream.

Recompute in double when a float32 norm is below Float.MIN_NORMAL in cosine, guard the zero divisor in OSQ accumulate, and normalize in double in l2normalize.

Trade-offs

The study built this as three shadow patches: it fixed the default-provider runs but 0 of 5 index-side tests with the Panama provider, because every vector provider (default, Panama, off-heap bulk scorers, native) needs its own patch. It touches hot paths, which upstream rejected for the exact-zero case (#16099).

Ignore NaN when RandomVectorScorer.bulkScore and the bulk scorers compute the batch maximum, so one NaN cannot hide 63 other documents.

Trade-offs

Limits the blast radius on the default codec only: the bad document still scores +Infinity or NaN, and OSQ is unaffected.

Method: SMT solving with Z3's floating-point theory (the study's "formal" tier). 2026-10-22 study, run 3, experiment "quantization-smt" (main 7cecf79523).

• The proof obligation "accumulate divides by a positive divisor for every vector KnnFloatVectorField accepts" was refuted, with a witness in 10.7 s; "VectorUtil.cosine is finite for non-zero finite inputs" was refuted too; Z3 proved that no x > 2^-75 has fl(x*x) == 0, which gives the exact threshold.
• The default-codec effect was found as a side effect of the OSQ probe; the OSQ case came from chaining preconditions (the field's checks, then accumulate).
• AI verifiers wrote their own JUnit repros without reading the finder's tests: one for the default codec (12 tests, including the top-hit rate and CheckIndex), one for OSQ (10 tests, including the merge and a threshold scan over every float in [2^-80, 2^-60]). Both reproduced on main and 10.5.2. A follow-up merged the two into one root cause with a fix matrix and found the query side.
• This hunt re-ran it on ea57360912ed, ran a cross-model refute review (OpenAI GPT-5.6), traced its history across 33 releases, searched for prior reports and validated a fix.