The byte DOT_PRODUCT score is 0.5 + dot / (dimensions x 32,768). The divisor is multiplied in 32-bit int before it becomes a float, so it is right up to 65,535 dimensions, negative from 65,536 to 131,071 and zero at 131,072.
- A byte vector stores each dimension as a signed byte, -128 to 127, so the dot product of two byte vectors is a whole number.
- Lucene maps it into [0, 1]: divide by the largest possible magnitude,
dimensions x 2^15(2^15 = 2 x 128 x 128), and add 0.5. Higher scores mean closer vectors, and kNN keeps the highest. - Three places compute this:
VectorUtil.dotProductScore(behindVectorSimilarityFunction.DOT_PRODUCTand the default scorers) and, with the Panama vector code on memory-mapped files, a query-time and a graph-build scorer.
dimensions x 32,768 fits in an int up to 65,535 dimensions, wraps to -2,147,483,648 at 65,536 and to 0 at 131,072, and what that does to the scores of a vector's own copy and its opposite. Select the still to open it at full size. Generated with AI, checked against the code.- Take byte vectors of 65,536 dimensions: the query and document A are all
+1, document B is all-1. Both documents have the same norm, as byteDOT_PRODUCTrequires. - Score them with
VectorSimilarityFunction.DOT_PRODUCT.compare, or index them through a vectors format that raises the 1,024-dimension limit and run aKnnByteVectorQuery. - You expect
0.5 + dot / (65,536 x 32,768): A = 0.5000305 and B = 0.49996948, so A, a copy of the query, ranks first. - Instead 65,536 x 32,768 = 2,147,483,648 does not fit in an
intand wraps to -2,147,483,648: A = 0.49996948, B = 0.5000305, so B outranks A (a six-document kNN test came back in exactly reverse order). Nothing throws. - Up to 65,535 dimensions all is correct. Higher up it gets worse: at 131,071 dimensions A and B score -3.4999695 and 4.4999695, and at 131,072 the divisor wraps to 0, so scores are
Infinity,-InfinityorNaN.
Only byte DOT_PRODUCT vectors of 65,536 or more dimensions. Stock vectors formats accept at most 1,024, so ordinary indexes are safe. Hit: code calling the scoring API on such arrays, and fields indexed through a format that raises getMaxDimensions, the delegation pattern committers recommend for higher limits. No such deployment is known.
Mechanism, terms and numbers655 words
The byte DOT_PRODUCT score is documented as "scaled to be in [0, 1]" and computed as 0.5 + dot / (len * 2^15). Three places write the divisor as (float) (len * (1 << 15)): VectorUtil.dotProductScore, which VectorSimilarityFunction.DOT_PRODUCT.compare(byte[], byte[]) and the default scorers use, and the two Panama MemorySegment byte scorers (query time and graph build). The product is taken in int and only then cast, so it is exact up to 65,535 dimensions, wraps to -2^31 at 65,536 and to 0 at 131,072.
From 65,536 to 131,071 dimensions every score is mirrored around 0.5: a vector scores its own copy 0.49996948 and its opposite 0.5000305, so any two candidates compare the wrong way round (a six-document kNN test came back in exactly reverse order). In that band scores can also leave [0, 1] and go negative; at 131,072 they are infinite or NaN. Nothing throws. No stock vectors format accepts more than 1,024 dimensions, so ordinary indexes are safe; it is reached through the public scoring API on long arrays and through a format that raises getMaxDimensions, the delegation pattern committers recommend for higher limits. No deployment with byte vectors that large is known, so the priority is low. Casting to float before multiplying, at all three sites, fixes it and leaves every score below 65,536 dimensions bit-identical.
- The expression.
VectorUtil.dotProductScore(VectorUtil.java:427-438): javadoc "Dot product score computed over signed bytes, scaled to be in [0, 1]", comment "divide by 2 * 2^14 (maximum absolute value of product of 2 signed bytes) * len", thenfloat denom = (float) (a.length * (1 << 15));andreturn 0.5f + dotProduct(a, b) / denom;.VectorSimilarityFunction.DOT_PRODUCT.compare(byte[], byte[])returns it (VectorSimilarityFunction.java:65-68), andDefaultFlatVectorScorerscores throughcompare. - The copies. With the Panama provider on
MMapDirectory, the query-timeLucene99MemorySegmentByteVectorScorer.DotProductScorerreturns0.5f + raw / (float) (query.length * (1 << 15))(:127) and the graph-build scorer0.5f + raw / (float) (values.dimension() * (1 << 15))(Lucene99MemorySegmentByteVectorScorerSupplier.java:169). "OnMMapDirectory" is shorthand: these scorers are used only when the vector values have a non-null index slice that is aMemorySegmentAccessInput; otherwise, for example for the in-memory vectors at flush, scoring falls back toDefaultFlatVectorScorerandVectorUtil. The native dot-product option uses the same two classes and keeps the divisor. - The wrap. Java multiplies two
ints inintand drops the high bits; the cast tofloatcomes after.len x 2^15fits only whilelen <= 65,535(65,535 x 32,768 = 2,147,450,880). - Only the normalization is wrong. The signed dot product itself fits in an
intbelow 131,072 dimensions (|dot| <= len x 2^14). FloatDOT_PRODUCTand byteCOSINEare correct at 65,536 dimensions (a checker ran them); byteEUCLIDEANandMAXIMUM_INNER_PRODUCTdo not divide by the length. Only raw byteDOT_PRODUCTscores are affected: theshort[]overload normalizes differently, and the scalar-quantized scorers use their own formulas.
| dimensions | intended divisor | int divisor | vs itself | vs all -1 |
|---|---|---|---|---|
| 1,024 | 33,554,432 | 33,554,432 | 0.5000305 | 0.49996948 |
| 65,535 | 2,147,450,880 | 2,147,450,880 | 0.5000305 | 0.49996948 |
| 65,536 | 2,147,483,648 | -2,147,483,648 | 0.49996948 | 0.5000305 |
| 100,000 | 3,276,800,000 | -1,018,167,296 | 0.49990177 | 0.5000982 |
| 131,071 | 4,294,934,528 | -32,768 | -3.4999695 | 4.4999695 |
| 131,072 | 4,294,967,296 | 0 | Infinity | -Infinity |
| 196,608 | 6,442,450,944 | -2,147,483,648 | 0.49990845 | 0.50009155 |
For 65,536-131,071 dimensions the divisor is len x 2^15 - 2^32, negative, so every score is mirrored around 0.5. Scores leave [0, 1] once |dot| > (2^32 - len x 2^15) / 2 (from 65,537 dimensions with saturated components, from 131,065 with unit components). At 131,072 the divisor is 0 (NaN when the dot product is 0 too). The divisor's pattern repeats every 131,072 dimensions (positive but far too small for 131,073-196,607: order right, scores can leave [0, 1]); from 131,072 dimensions the dot product's own int accumulator can also overflow with saturated components, a separate limitation (LFV-138). Default and Panama providers print the same values.
Exposure and likelihood302 words
- Reachable through direct calls to
VectorUtil.dotProductScore,VectorSimilarityFunction.DOT_PRODUCT.compare(byte[], byte[])or the flat byte scorers on arrays of 65,536+ bytes, and through aKnnVectorsFormatthat raisesgetMaxDimensions, for example a thin delegating wrapper over the stock HNSW format. Committers recommend that pattern: #13150 was closed "Won't Do" with "Lucene prefers the delegation pattern", and #12436 (which made the limit codec-defined in 9.8.0) and a 2023 java-user thread say the same. Apache Solr ships such a wrapper with no upper bound (read from Solr's source, not run). - Not reachable with stock formats and no wrapper: every one declares 1,024, checked at index time (
IndexingChain.java:1386-1391), the only dimension check:KnnByteVectorField's "dimension > 1024" javadoc is not enforced. Elasticsearch caps vector fields at 4,096 dimensions although it keeps its own copy of the expression. - Once such a field exists, every query on it is affected, for every user: for 65,536-131,071 dimensions every score comparison is reversed whatever the data. A delegating-format kNN test returned its 6 same-norm documents as
[5, 4, 3, 2, 1, 0], the query's own vector last; recall and ranking on larger graphs were not measured. At 131,071 dimensions its kNN scores ran from about 250 down to -372.5 (the adjudication's re-run; random vectors, so the values depend on the seed), andMIGRATE.md(LUCENE-7996) says scorers may not return negative scores; at 131,072 scores are infinite orNaN. The HNSW graph is built from the same inverted similarities with either provider (effect on graph quality not measured). - Silent: no exception on main or 10.1.0 and later, with assertions on or off. On 9.x and 10.0.0 with assertions on, a negative score trips
TopScoreDocCollector'sassert score >= 0(executed on 9.12.3, read for 10.0.0), a test-time assertion removed in 10.1.0 (#13939). No crash, hang or index damage.
One JUnit test on the public API, no index: an all-+1 query of 65,536 bytes scored against itself and against all -1 with VectorSimilarityFunction.DOT_PRODUCT.compare. Deterministic, single thread, under a second.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.util;
import java.util.Arrays;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.apache.lucene.tests.util.LuceneTestCase;
public class TestByteDotProductScoreDenominator extends LuceneTestCase {
public void testDotProductScoreOrdersBySimilarity() {
byte[] query = new byte[1 << 16];
byte[] opposite = new byte[1 << 16];
Arrays.fill(query, (byte) 1);
Arrays.fill(opposite, (byte) -1);
float same = VectorSimilarityFunction.DOT_PRODUCT.compare(query, query);
float opp = VectorSimilarityFunction.DOT_PRODUCT.compare(query, opposite);
// 65,536 dims: expected the identical vector to score higher than its opposite
assertTrue("expected same > opposite, got same=" + same + ", opposite=" + opp, same > opp);
}
}
cd $LUCENE # an apache/lucene checkout at 645d4fe14a20
cp $REPRO/TestByteDotProductScoreDenominator.java lucene/core/src/test/org/apache/lucene/util/
./gradlew :lucene:core:test --tests org.apache.lucene.util.TestByteDotProductScoreDenominator -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false; Panama provider: add -Ptests.defaultvectorization=true (same failure)
- Expected
- The identical vector scores higher: 0.5000305 against 0.49996948 for the opposite one.
- Observed
AssertionError: expected same > opposite, got same=0.49996948, opposite=0.5000305, with assertions on and off.
More files, environment, full output202 words
65,536 is the smallest failing size, the first where len x 2^15 overflows. Components +1/-1 keep the dot product exact and give both documents the same norm, as byte DOT_PRODUCT requires. The test covers the VectorUtil site that compare uses; the two Panama scorer sites were checked separately when the fix was validated. It checks the ordering, not the values, so a VectorUtil-only patch passes it although both MemorySegment scorers stay wrong. As a regression test it is the same test; it fails on stock and passes with the fix. All runs were on Linux aarch64 with OpenJDK 25.0.3; the mechanism is plain int arithmetic.
| lucene | main 645d4fe14a20 (11.0.0-SNAPSHOT); the same failure on 10.6.0 |
|---|---|
| jdk | OpenJDK 25.0.3 |
| os | Linux aarch64 |
# gradle, stock 645d4fe14a20, seed DEADBEEF, assertions on
TestByteDotProductScoreDenominator > testDotProductScoreOrdersBySimilarity FAILED
java.lang.AssertionError: expected same > opposite, got same=0.49996948, opposite=0.5000305
# assertions off (-Ptests.asserts=false): the same message
# Panama provider (-Ptests.defaultvectorization=true): the same message
# 10.6.0 release jars, assertions on and off: the same message
# with the fix (float cast at the three sites): 1 test, 0 failures, assertions on and off; with Panama (assertions on), with the off-heap scorer test: 3 tests, 0 failures- The API promises scores "scaled to be in [0, 1]", higher for closer vectors, with no length limit.
- In #913 (2022) msokolov showed that the byte dot product's
intaccumulator is safe up to 2^17 dimensions; this divisor breaks at 2^16. - Float
DOT_PRODUCT, byteCOSINEand the Panama query scorer before 10.3.0 (alongproduct) are correct at these sizes.
The full argument267 words
- The contract.
VectorUtil.dotProductScore: "scaled to be in [0, 1]" (VectorUtil.java:428), with the intended divisor in the comment (:435).VectorSimilarityFunction.compare(byte[], byte[]): "Higher similarity scores correspond to closer vectors" (VectorSimilarityFunction.java:132-133), no length limit. msokolov on #12256: "the only requirement is that they sort the same as the dot-product score". uschindler on #12342: byte scores "can't get <0" thanks todotProductScore. The ordering statements fail from 65,536 dimensions; the [0, 1] range and non-negativity from 65,537 with saturated components, from 131,065 with unit components, and for every input at 131,072. - The authors' own bound. In the #913 review (2022), rmuir asked about overflow of the byte dot product's
intaccumulator, and msokolov answered that the limit is "2^31/2^14 = 2^17 and I don't think we would ever have a 2^17-dimensional vector". The remark is about the accumulator, not this divisor, and is not an API contract; it shows the authors assumedintarithmetic safe up to 2^17 dimensions, while the score's divisor overflows at half that. - The siblings are right. Float
DOT_PRODUCTand byteCOSINEare correct at 65,536 dimensions. The Panama query scorer's divisor was correct from 9.11.0 to 10.2.2 because it multipliedquery.byteSize(), along; #14874 replaced it withquery.lengthwithout discussing the type. - No source found makes it intended. No reviewed javadoc, test, CHANGES/MIGRATE entry, commit or searched discussion declares the behaviour at 65,536+ dimensions intended (the searches have gaps).
TestVectorUtilchecksdotProductScoreonly on 2- and 3-element vectors (with a reference that has the sameintproduct), and the byte-kernel tests stop at 4,098 dimensions.
The devil's-advocate check138 words
The strongest case for "intended or unsupported": every stock format caps at 1,024; KnnByteVectorField's javadoc says a dimension above 1,024 throws; the util package says its APIs are "primarily internal"; LUCENE-10471 called 16K dimensions "well beyond any stated use case"; and in #913 the author did not expect vectors this large. A checker rated this argument moderate.
Why it fails: the limit has been codec-defined since 9.8.0 and committers recommend raising it by delegation, so the KnnByteVectorField sentence is stale, not a contract; VectorSimilarityFunction lives in org.apache.lucene.index and documents no limit; the authors' own accumulator bound (2^17) is twice the size where this breaks; and the defect is operator order in one expression, fixed by moving one cast per site. The scale explains the low priority, not the behaviour. All checkers and the adjudication rejected the argument.
Recommended Cast to float before multiplying, at all three sites
Cast to float before multiplying, at all three sites: float denom = (float) a.length * (1 << 15); and the same in the two Panama scorers. Every score below 65,536 dimensions stays bit-identical. Validated with both vectorization providers; fixing only VectorUtil leaves the Panama scorers inverted.
- float denom = (float) (a.length * (1 << 15));
+ float denom = (float) a.length * (1 << 15);
- return 0.5f + raw / (float) (query.length * (1 << 15));
+ return 0.5f + raw / ((float) query.length * (1 << 15));
- return 0.5f + raw / (float) (values.dimension() * (1 << 15));
+ return 0.5f + raw / ((float) values.dimension() * (1 << 15));All options and trade-offs4 options ยท 465 words
The hunt's fix, unchanged, run through gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF) with the default and the Panama vectorization provider, together with the minimal repro and, for the two Panama sites, an independent checker's test of the off-heap byte scorers (65,536 dimensions, MMapDirectory).
In VectorUtil.dotProductScore (:436), Lucene99MemorySegmentByteVectorScorer (:127) and Lucene99MemorySegmentByteVectorScorerSupplier (:169). (float) len is exact for len <= 2^24, and multiplying by 2^15 only shifts the exponent, so every score up to 65,535 dimensions stays bit-identical; above that the divisor is the intended len x 2^15.
- float denom = (float) (a.length * (1 << 15));
+ float denom = (float) a.length * (1 << 15);
- return 0.5f + raw / (float) (query.length * (1 << 15));
+ return 0.5f + raw / ((float) query.length * (1 << 15));
- return 0.5f + raw / (float) (values.dimension() * (1 << 15));
+ return 0.5f + raw / ((float) values.dimension() * (1 << 15));Trade-offs
- One cast moves per site; no behaviour change below 65,536 dimensions (performance not benchmarked). Empty arrays passed directly still give
NaN(0 / 0), as before (executed). - All three sites are needed: with only the
VectorUtilchange, the Panama scorers still invert the scores (executed).
What the checks validated
Run through gradle on a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF), with the minimal repro and an independent checker's test of the two off-heap byte scorers (65,536 dimensions, MMapDirectory):
- Default provider (what gradle picks for this seed): stock, the repro fails with assertions on and off and both off-heap scorer tests fail; with the fix all pass, assertions on and off. The
VectorUtilhunk alone also passes here, because the default scorers callcompare. - Panama provider (
-Ptests.defaultvectorization=true, 256-bit): stock 3 of 3 fail;VectorUtilhunk only, 2 of 3 still fail (bothMemorySegmentscorers invert); full fix, 3 of 3 pass. - Closest suites with the fix (
TestVectorUtil, bothTestVectorScorerclasses,TestVectorUtilSupport,TestFlatVectorScorer,TestKnnByteVectorQuery,TestKnnByteVectorQueryMMap,codecs.lucene99.*; 13 classes, 752 tests, 38 skipped): 0 failures under each provider.
(float) (a.length * (1L << 15)), likewise at the three sites.
Trade-offs
The same scores as the float cast; a style choice.
The open suggestion #12263, for byte/float score parity.
Trade-offs
Removes the overflow as a side effect, but changes every byte DOT_PRODUCT score at every dimension: a separate design decision.
Throw for more than 65,535 elements.
Trade-offs
The API documents no limit, float DOT_PRODUCT works at any length, and committers recommend formats that raise getMaxDimensions.
A product that passes a schema-defined dimension straight to a delegating vectors format with no upper bound (Solr's wrapper does, read from its source, not run) would serve inverted rankings on byte DOT_PRODUCT fields of 65,536+ dimensions. Only configuration creates such a field; document authors could then use the inversion for rank placement.
Non-formal: while modelling the integer ranges of the byte-vector kernels, an AI tool read the score's divisor and saw that len * (1 << 15) is multiplied in int and reaches 2^31 at 65,536 dimensions. A JUnit test on the stock jars confirmed it, and independent checkers reproduced it with their own tests.
The method in full125 words
Method: non-formal (code reading plus an executed test; no model checker or solver produced the signal), a hunt unit on VectorUtil and the vectorization providers.
- The finder was bounding the
intranges of the byte kernels; a small arithmetic script for the sibling candidate (LFV-138) also prints the divisor's threshold,a.length = 2^16, by plain integer arithmetic. - A JUnit test on the stock jars confirmed it with assertions on and off and with the Panama and default providers; the fix was first checked by shadowing the patched classes onto the jars.
- A cross-model reviewer (OpenAI GPT-5.6) then corrected the first claim's expected scores (0.5 +/- 2^-15, not 1.0 and 0.0) and had the kNN test reframed, since it raises the stock format's declared limit.
No prior report found in GitHub issues and PRs, open PRs' diffs, the mailing lists, JIRA and downstream projects (2026-10-26, re-checked on GitHub 2026-10-31). Open #12263 proposes changing the same divisor for another reason, and never mentions overflow.
Searches6 searches
Closest items: #12263 (open, 2023) suggests dropping the length from the same divisor so byte and float scores match, which would remove the overflow as a side effect, but never mentions overflow; #12256 (closed) is the same reporter's scaling complaint, where msokolov states the ordering requirement this bug breaks; #913 (2022) discussed the int range of the dot-product accumulator, not the divisor; #12306 and #11507 carry general committer warnings that huge dimension counts may overflow, naming no code.
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits (apache/lucene) | 2026-10-26 | the hunt's search tool (12 GitHub issue/PR and 3 commit queries) plus 52 GraphQL searches over apache/lucene, Elasticsearch, OpenSearch, Solr, luceneutil and all of GitHub: dotProductScore, the scorer class names, overflow and denominator terms, 65536 / 131072 dimensions, negative or infinite byte scores | no report; related #12263, #12256, #12342, #1073, #1054, #13339, #14874, #12306, #11507 |
| Open PRs | 2026-10-26; re-checked 2026-10-31 | changed files of all 413 open PRs, with the diffs of the 8 that touch an affected file grepped; on 2026-10-31 the 96 PRs updated since 2026-10-26, three of which touch VectorUtil.java | none changes the divisor (#15285 moves the scorers and keeps it; #16914, #16915 and #16790 touch VectorUtil elsewhere: no dotProductScore or 1 << 15 in their diffs) |
| Mailing lists (dev@, java-user@, issues@, commits@) | 2026-10-26 | 20 tool queries and 33 direct lists.apache.org queries | no report; the 2023 threads on raising the dimension limit by delegation |
| JIRA archive | 2026-10-26 | 4 tool queries and 6 JQL searches | no report |
| Downstream and web | 2026-10-26 | Elasticsearch and Solr code and issues; 6 web queries; Stack Exchange | no report; Elasticsearch keeps its own copy of the expression but caps vectors at 4,096 dimensions |
| GitHub re-check | 2026-10-31 | dotProductScore, "dotProductScore overflow", "byte dot product overflow", "65536 dimensions", "131072 dimensions", "byte vector score negative", MemorySegmentByteVectorScorer, "dot product denominator overflow", as issues and as PRs | dotProductScore: issues #12256, #12263, #12342 and PRs #947, #1073, #12102, #12104 (the code's own changes, none a report); the other terms: no hits |
Introduced in 9.4.0 by #1054 (a693fe819b0, 2022-08-10, LUCENE-10577); copied into the Panama graph-build scorer in 9.11.0 (#13339) and the query-time scorer in 10.3.0 (#14874, which replaced a correct long). Fails on releases from 9.4.0 to 10.6.0 and on main; 9.3.0 has no such API. Not fixed upstream.
Versions tested7 rows
Introduced by a693fe819b0, LUCENE-10577 / #1054 "enable quantization of HNSW vectors to 8 bits" (msokolov, 2022-08-10; branch_9x 87a8f7d48f3), which added byte vectors with the divisor (float) (a.length * (1 << 15)); #1073 (798c02dd70d, 2022-08-20) fixed an offset in the same formula and kept the int product. The 1,024-dimension limit was then a global constant; since #12436 (9.8.0) it is the codec's getMaxDimensions.
Copies: #13339 (05f04aa08a4, 2024-05-21, first release 9.11.0) added the Panama graph-build scorer with values.dimension() * (1 << 15) in int, and the query-time scorer with query.byteSize() * (1 << 15), a long, which is correct. #14874 (b3f4011292c, 2025-07-29; backport #15010, first release 10.3.0) changed the latter to query.length. So with Panama on MMapDirectory, kNN query-time scores were right in 9.11.0-10.2.2 (read at every tag; executed with Panama forced on 9.12.3 and 10.2.2, since 9.11.0 cannot enable Panama on JDK 25) and wrong from 10.3.0, while the graph was still built from int-product scores; with the default provider they are wrong from 9.4.0.
Not fixed upstream (2026-10-31): main 2f424fd428da, branch_11x, branch_10_6 and the 10.6.0 tag still have all three expressions.
| Version or commit | Verdict | Result |
|---|---|---|
main 645d4fe14a20 | affected | the minimal test fails through gradle with assertions on and off and with both providers; with Panama both off-heap byte scorers also invert |
| 10.6.0 | affected | the minimal test fails, assertions on and off |
| 10.5.2, 10.3.0 | affected | the finder's test fails (10.5.2); a kNN query through a high-dimension format ranks the opposite vector first, also with Panama |
| 10.2.2, 9.12.3 | kNN query scores correct with Panama | with the default provider, kNN through a high-dimension format ranks the opposite vector first (and the scoring API fails on 9.12.3); with Panama forced, the query-time kNN scores are correct thanks to the long product (the graph is still built from int-product scores; not measured), and wrong again once that line is mutated to int (10.2.2) |
10.0.0, 9.5.0, 9.4.2, 9.4.0, a693fe819b0 | affected | the finder's test fails (the 9.4.x runs and the commit build with a BytesRef adaptation); the fix passes at 9.4.0 and the commit |
9.3.0, parent 59a0917e256 | not affected | no byte dotProductScore: the test does not compile |
fix on 645d4fe14a20 | not affected | the minimal test and the off-heap scorer test pass with the default and the Panama provider; 752 tests of the closest suites pass under each |
Confirmed with high confidence by the study's judge after an independent reproducer, an intended-behaviour devil's advocate and reviews by another model family (OpenAI GPT-5.6), none dissenting; re-run on two later mains and the 10.6.0 jars, and the fix validated through gradle with both vectorization providers.
Checks and verdicts398 words
Each checker that executed wrote its own test from the claim; the adjudication re-ran the independent reproducer's tests, the devil's advocate's kNN test and the fix.
| Check | Verdict | What it found |
|---|---|---|
| finder | candidate | 2 of 2 tests fail on main with assertions on and off, with the Panama (256- and 128-bit) and default providers, and on 10.5.2 and 9.12.3; the float-cast fix passes. |
| refute review (OpenAI GPT-5.6) | real, high; two corrections | Its own test fails 2 of 3 and its 65,535-dimension control passes; it corrected the expected scores (0.5 +/- 2^-15) and had the kNN test moved to an illustration. |
| Independent reproducer | real | Own public-API test fails 4 of 5 and an off-heap scorer test (both Panama scorers) 2 of 2; boundary sweep 65,535 correct, 65,536 mirrored, 131,071 out of [0, 1], 131,072 infinite; all pass with the fix. |
| Works-as-designed review (OpenAI GPT-5.6) | real, high | Intended argument moderate and rejected; float DOT_PRODUCT and byte COSINE pass at 65,536 dimensions, isolating the divisor. |
| Intended-behaviour devil's advocate | real | Through a delegating format over the stock HNSW format, kNN returns 6 documents in exact reverse order at 65,536 dimensions and scores from about 249 down to -374 at 131,071 (random vectors); no source makes it intended. |
| adjudication | confirmed, high | Re-ran the independent reproduction's tests (4 of 5 fail, assertions on and off; off-heap 2 of 2), the intended-behaviour check's kNN test (2 of 3; seed C0DE01: reverse order, scores 250 down to -372.5 at 131,071) and the fix (7 of 7 pass). |
| re-verification | still reproduces | Same failures on main a9779ef3c193 (2026-10-26); the one-test repro fails on 645d4fe14a20 and passes with the fix shadowed (2026-10-29). |
| this doc (2026-10-31) | real, fix validated | The 10.6.0 jars fail the minimal test (assertions on and off). Through gradle on 645d4fe14a20 stock fails with both providers; the full fix passes the repro, the off-heap scorer test and 752 tests of the closest suites under each; a VectorUtil-only fix leaves both Panama scorers inverted. |
| fact checks (2026-10-31) | real; wording fixes | An independent checker and a reviewer from another model family (OpenAI GPT-5.6) confirmed the bug, the mechanism, the code references, the score table, the repro and the fix, and asked for wording fixes: the 65,536-131,071 band, the 9.11.0 Panama tag, the score-range credit, the 10.0.0 assertion and several caveats. All were applied. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 472 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,665 words
- Registry title
- Byte DOT_PRODUCT scores (VectorUtil.dotProductScore and the two Panama byte scorers) compute len * 2^15 in int, so from 65,536 dimensions every score is mirrored around 0.5 (closer vectors score lower) and at 131,072 dimensions scores become +/-Infinity or NaN
- Area
- util / VectorUtil byte DOT_PRODUCT score x internal/vectorization / Panama MemorySegment byte scorers
- Priority
- Low, rank 986: From 65,536 dimensions byte DOT_PRODUCT scores are mirrored, which inverts every ranking on such a field, but no stock format accepts more than 1024 dimensions and no deployment with byte vectors that large is known.
- Impact class
- silent wrong results
- Impact note
- Only byte (int8) DOT_PRODUCT vectors of 65,536 or more dimensions are affected, which no stock KnnVectorsFormat accepts (all declare getMaxDimensions 1024): code that calls VectorUtil.dotProductScore, VectorSimilarityFunction.DOT_PRODUCT.compare(byte[], byte[]) or the flat byte scorers on such arrays, and fields indexed through a delegating KnnVectorsFormat that raises getMaxDimensions (the pattern committers recommend for raising the limit, and what Solr ships with no upper bound, read from its source, not run); there every byte DOT_PRODUCT score is mirrored around 0.5 for 65,536-131,071 dims so kNN returns the least similar documents first, scores leave [0, 1] (including negative kNN scores) near the top of that band, and at 131,072 dims they are +/-Infinity or NaN; -ea and -da, ordinary data, no deployment with such dimensions is known.
- Summary
- You score two byte vectors of 65,536 dimensions with VectorSimilarityFunction.DOT_PRODUCT (or VectorUtil.dotProductScore, or run a KnnByteVectorQuery on a byte DOT_PRODUCT field indexed through a format that raises getMaxDimensions) and expect the documented score 0.5 + dot / (len * 2^15): within [0, 1] and higher for more similar vectors. Instead the divisor len * (1 << 15) is multiplied in int before the cast to float, so from 65,536 dimensions it wraps to a negative number and every score is mirrored around 0.5: a vector identical to an all-ones query scores 0.49996948 and its opposite 0.5000305, and kNN returns the documents in exactly reverse order. At 131,072 dimensions the divisor wraps to 0 and the scores are +Infinity, -Infinity or NaN.
- Introduced
- a693fe819b04 (LUCENE-10577, apache/lucene#1054 'enable quantization of HNSW vectors to 8 bits', by msokolov, merged on main 2022-08-10; branch_9x backport 87a8f7d48f3d) added VectorUtil.dotProductScore with the denominator (float) (a.length * (1 << 15)), multiplied in int [read: git log -S/-G, git blame, git show; exec: a commit-level build of a693fe819b0 fails the repro with -ea and -da and passes with the float-cast fix, while its parent 59a0917e256 has no byte DOT_PRODUCT API, so the repro does not compile there]; 798c02dd70d (#1073, 2022-08-20, an offset fix) wrote today's 'float denom = (float) (a.length * (1 << 15))' and kept the int product; 45944002161 (#12102) changed only the signature; the same int expression was copied into the Panama graph-build scorer by 05f04aa08a4 (#13339, 2024-05-21; 9.x #13402) and into the Panama query-time scorer by b3f4011292c (#14874, 2025-07-29; 10.x backport #15010), which replaced a correct long product (query.byteSize()) there (2022-08-10 (a693fe819b0 on main; branch_9x backport committed 2022-08-24); Panama graph-build scorer 2024-05-21 (#13339); Panama query-time scorer 2025-07-29 (#14874))
- First affected version
- 9.4.0 [exec] (VectorUtil / VectorSimilarityFunction path); the Panama query-time scorer from 10.3.0 [exec, through the delegating-format illustration]; 9.3.0 and older do not have the API [exec: the repro does not compile]
- Affected versions
- VectorUtil.dotProductScore / VectorSimilarityFunction.DOT_PRODUCT byte path: every release from 9.4.0 to 10.6.0 (30 release tags) and main ea57360912ed (unreleased 11.0) [exec 2026-10-31: the minimal test fails on the 10.6.0 release jars (SHA-1 checked against Maven Central) with -ea and -da; exec: the finder repro fails on commit a693fe819b0, 9.4.0, 9.4.2, 9.5.0, 9.12.3, 10.0.0, 10.5.2 and main, also with -da on a693fe819b0, 9.4.0, 9.12.3, 10.5.2 and main; the intended-behaviour check's own test fails on 9.5.0, 9.12.3, 10.5.2 and main; the other releases have the same expression [read at every tag]]; Panama graph-build scorer (Lucene99MemorySegmentByteVectorScorerSupplier, int product): 9.11.0 to 10.6.0 and main [read; exec by the independent reproduction on 10.5.2 and main]; Panama query-time scorer (Lucene99MemorySegmentByteVectorScorer): long product, correct, in 9.11.0-10.2.2 [exec: the illustration passes with Panama on 9.12.3 and 10.2.2 and fails on 10.2.2 when only that line is mutated to int], int product from 10.3.0 to 10.5.2, 10.6.0 (by reading) and main [exec: the illustration fails with Panama on 10.3.0, 10.5.2 and main]; so with Panama on MMapDirectory the kNN query scores through a high-dimension format were correct in 9.11.0-10.2.2 (the graph-build supplier already had the int product there; its effect on graph quality was not measured) and wrong from 10.3.0, while the default provider is wrong in every release from 9.4.0; indexing such fields through a non-stock format is possible only from 9.8.0, when #12436 made the limit codec-defined [read]; 9.3.0 and older are not affected (no API) [exec]; not fixed: upstream main, branch_10x, branch_10_6 and branch_9_12 still have the int expression (2026-10-26) [read]; executed on aarch64 Linux with JDK 25 only (pure int arithmetic, platform-independent)
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- non-formal: code reading
- Upstream and prior reports
- none found (2026-10-26, read-only: a scripted prior-report search with 47 queries (12 GitHub issue/PR, 3 commit, 2 downstream-repo, 20 mailing-list over dev@, java-user@, issues@ and commits@, 4 JIRA, 6 web; 210 hits) plus the finder's earlier run (38 hits); 52 GitHub GraphQL searches over apache/lucene, elastic/elasticsearch, opensearch-project/OpenSearch, opensearch-project/k-NN, apache/solr, luceneutil and all of GitHub; 33 lists.apache.org queries (11 each on dev@, java-user@ and issues@); 6 JQL searches; web and Stack Exchange searches; the changed files of all 413 open apache/lucene PRs (8 touch an affected file: #83, #13370, #13981, #14371, #15285, #16200, #16640, #16790; none changes the denominator, and #15285 moves the scorers but keeps the int expression); the three files on upstream main, branch_10x, branch_10_6 and branch_9_12 (int expression present on all four); git history, CHANGES.txt and MIGRATE.md; plus the independent reproduction's and the intended-behaviour check's searches); gaps: a scripted prior-report search ran 12 of 30 planned GitHub queries (the other terms were covered by GraphQL), '1 << 15' cannot be searched on GitHub, and list search matches full text but caches only subjects and the first 400 characters; related prior items, none of them this bug: #12263 (open 2023 suggestion to drop a.length from the same denominator so byte and float scores match, which would remove this overflow as a side effect; it never mentions overflow or large dimensions), #12256 (closed 2023 scaling complaint on the same expression by the same reporter, who closed it as not a bug; msokolov states there the ordering requirement this bug breaks), #1073 and #1054/#947 (wrote the expression; reviews do not discuss the product type), #13339 and #14874/#15010 (copied it into the Panama scorers; #14874 replaced a correct long product), #12342 (uschindler: byte DOT_PRODUCT scores 'can't get <0', broken near 131,071 dims), #12306 and #11507 (general committer warnings that untested huge dimension counts may overflow; they name no code or symptom), #12436/#12466 (made the limit codec-defined), elastic/elasticsearch#90774 (downstream keeps its own copy of the expression but caps dense_vector at 4096 dims), apache/solr#1680 (downstream high-dimension wrapper; no score-arithmetic discussion)
- Lucene commit tested
- main
645d4fe14a20through gradle in a scratch worktree, with the default and the Panama vectorization provider: stock, with the fix and with aVectorUtil-only partial fix, plus 752 tests of the closest suites with the fix under each provider; the minimal test on the 10.6.0 release jars (assertions on and off); the checkers' tests on mainea57360912edanda9779ef3c193, 10.5.2, 9.12.3 and 9.5.0; the history runs on 10.3.0, 10.2.2, 10.0.0, 9.4.2, 9.4.0 and the introducing commita693fe819b0. All runs on Linux aarch64 with OpenJDK 25.0.3. Upstream main2f424fd428da(2026-10-30),branch_11x,branch_10_6and the 10.6.0 tag have the same code.
- Finding record
a Lucene main checkout lucene/core/src/java/org/apache/lucene/util/VectorUtil.java:427-438 (dotProductScore javadoc 'Dot product score computed over signed bytes, scaled to be in [0, 1]'; :435 comment 'divide by 2 * 2^14 (maximum absolute value of product of 2 signed bytes) * len'; :436 'float denom = (float) (a.length * (1 << 15));')a Lucene main checkout lucene/core/src/java25/org/apache/lucene/internal/vectorization/Lucene99MemorySegmentByteVectorScorer.java:120-127 (same comment; :127 'return 0.5f + raw / (float) (query.length * (1 << 15));') and Lucene99MemorySegmentByteVectorScorerSupplier.java:169 ('return 0.5f + raw / (float) (values.dimension() * (1 << 15));')a Lucene main checkout lucene/core/src/java/org/apache/lucene/index/VectorSimilarityFunction.java:52-58 (DOT_PRODUCT javadoc: 'byte vectors should simply all have the same norm') and :131-140 (compare(byte[], byte[]): 'Higher similarity scores correspond to closer vectors'; no length limit)a Lucene main checkout lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsFormat.java:41 (DEFAULT_MAX_DIMENSIONS = 1024) and :105-114 (getMaxDimensions: 'Codecs implement this method to specify the maximum number of dimensions they support'); lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1331-1336 (the only indexing-time dimension check, against the codec's getMaxDimensions)a Lucene main checkout lucene/core/src/java/org/apache/lucene/document/KnnByteVectorField.java:80, 99-100, 115-116, 129-130 ('@throws IllegalArgumentException ... dimension > 1024', stale since #12436 per the intended-behaviour check)a Lucene main checkout MIGRATE.md:1368-1372 (LUCENE-7996: scorers may not produce negative scores; per the intended-behaviour check)
- The finding in 30 seconds
lucene/core/src/java/org/apache/lucene/util/VectorUtil.java:427-438 (at 645d4fe14a20) (javadoc :428, comment :435, int product :436)lucene/core/src/java25/org/apache/lucene/internal/vectorization/Lucene99MemorySegmentByteVectorScorer.java:47-53, (at 645d4fe14a20) :112-129 (int product :127); Lucene99MemorySegmentByteVectorScorerSupplier.java:169lucene/core/src/java/org/apache/lucene/index/VectorSimilarityFunction.java:41-44, (at 645d4fe14a20) :65-68, :110-113; lucene/core/src/java/org/apache/lucene/codecs/hnsw/DefaultFlatVectorScorer.java:134 (ByteScoringSupplier), :285 (ByteVectorScorer)lucene/core/src/java25/org/apache/lucene/internal/vectorization/Lucene99MemorySegmentFlatVectorsScorer.java:66-81, (at 645d4fe14a20) :110-126; Lucene99MemorySegmentByteVectorScorer.java:47, :122-127; NativeVectorizationProvider.java:70-71; lucene/core/src/java/org/apache/lucene/index/VectorSimilarityFunction.java:70-73 (short[] DOT_PRODUCT)
- Who is affected
lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1386-1391, (at 645d4fe14a20) :1622-1633; lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsFormat.java:41, :105-114; lucene/MIGRATE.md:1368-1372a Lucene clone: git grep 'assert score >= 0' releases/lucene/9.12.3 releases/lucene/10.0.0 releases/lucene/10.1.0 -- lucene/core/src/java/org/apache/lucene/search/TopScoreDocCollector.java (10.0.0 :75, :153; 10.1.0 none); git log -1 ff03379eb6e (#13939, 2024-10-22)lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java:797-801; (at 645d4fe14a20) lucene/core/src/java/org/apache/lucene/codecs/hnsw/FlatFieldVectorsWriter.java:54-60 (flush scores in-memory vectors through the delegate scorer)
- Why it looks unintended
lucene/core/src/java/org/apache/lucene/util/VectorUtil.java:428, (at 645d4fe14a20) :435; lucene/core/src/java/org/apache/lucene/index/VectorSimilarityFunction.java:53-58, :132-133; lucene/core/src/java/org/apache/lucene/util/package-info.java:35; lucene/core/src/java/org/apache/lucene/document/KnnByteVectorField.java:80, :100, :116, :130lucene/core/src/test/org/apache/lucene/util/TestVectorUtil.java:266-283; (at 645d4fe14a20) lucene/core/src/test/org/apache/lucene/internal/vectorization/TestVectorUtilSupport.java:31-34
- TestByteDotProductScoreDenominator.java (one test, public API):
repro/LFV-143/src/org/apache/lucene/util/TestByteDotProductScoreDenominator.java(35 lines)
Made from the code at 645d4fe14a20 and the executed runs.
assets/LFV-143/bug.png(472 KB)