This flat reader's search checks only that k is not 0; it never checks that the field has vectors. An emptied field has no quantizer to score the query with.
Lucene104ScalarQuantizedVectorsFormat(opt-in, since 10.4.0) stores each vector quantized to a few bits. It has no graph: its reader'ssearchscores every vector. Before scoring, the query vector is quantized with the field's quantizer and centroid.- When a merge drops every vector document of a field while other documents survive, the merged segment keeps the field's vector
FieldInfo(dimension 4 in the repro) and writes 0 vectors. The reader then hands out empty vector values that have no quantizer and no centroid.
v listed with 0 vectors. The query passes the reader's k == 0 check, the empty values have no quantizer, and scoring throws; the HNSW reader stops at its size() == 0 check. Select the still to open it at full size. Generated with AI, checked against the code.- You index 2 documents with the flat
Lucene104ScalarQuantizedVectorsFormat: one with a 4-dimensional vector in fieldv, one without. You delete the vector document and callforceMerge(1). The segment still listsv, but holds 0 vectors. - You run
searcher.count(new FloatVectorSimilarityQuery.Adaptive("v", {1, 2, 3, 4}, 0f))and expect 0, asKnnFloatVectorQuerygives on this index and the default codec and the HNSW formats give after the same steps. - Instead the reader's
searchreturns early only when k is 0, then builds a scorer over the empty vector values, whose quantizer is null:NullPointerException ... because "quantizer" is null, with assertions on and off. - The HNSW reader returns early when the field has 0 vectors. The same two-line guard in this reader's two
searchmethods makes the repro pass.
Only indexes that use the flat Lucene104ScalarQuantizedVectorsFormat for a float (or, on main, float16) vector field, once a segment lists the field with zero vectors, for example after all its vector documents were deleted. Then every FloatVectorSimilarityQuery at the default decay and every 4-argument searchNearestVectors fails, until that segment is merged with segments that hold vectors.
Mechanism, terms and numbers393 words
You use the flat Lucene104ScalarQuantizedVectorsFormat for a float vector field, delete every document that has a vector while other documents stay, and merge. The merged segment still lists the field but holds no vectors, the same state Lucene's own testDeleteAllVectorDocs reaches. A FloatVectorSimilarityQuery on it should match nothing; instead IndexSearcher.search throws NullPointerException ("quantizer" is null). The reader's search returns early only when k is 0 and then builds a scorer over the empty vector values, whose quantizer is null. The HNSW reader guards this case with size() == 0; KnnFloatVectorQuery and the 5-argument searchNearestVectors are not affected because they clamp k to the field's size first.
- The query reaches the reader unguarded.
FloatVectorSimilarityQuery.Adaptiveat its default decay (0.5) takes the approximate path for each leaf (search/AbstractVectorSimilarityQuery.java:190-206) and calls the 4-argumentsearchNearestVectorswith no size check (search/FloatVectorSimilarityQuery.java:254-263).CodecReader.searchNearestVectorschecks only theFieldInfoand calls the vectors reader (index/CodecReader.java:283-295). - The reader checks only k.
Lucene104ScalarQuantizedVectorsReader.search(String, float[], ...):if (knnCollector.k() == 0) return;(codecs/lucene104/Lucene104ScalarQuantizedVectorsReader.java:358), thenexhaustiveBulkScore(getRandomVectorScorer(field, target), ...)(:359). The float16 overload is the same (:365-366). - The empty values have no quantizer. For a field with 0 vectors,
OffHeapScalarQuantizedVectorValues.loadreturnsEmptyOffHeapVectorValues(OffHeapScalarQuantizedVectorValues.java:210-211), built with a null centroid and a null quantizer (:503-518). - The scorer dereferences it.
Lucene104ScalarQuantizedVectorScorer.getRandomQuantizedVectorScorerreadsqv.getQuantizer()(:105) and callsquantizer.scalarQuantize(...)(:119):NullPointerException.exhaustiveBulkScorewould accept a null scorer (Lucene104ScalarQuantizedVectorsReader.java:376), but none is returned.
Lucene99HnswVectorsReader(the HNSW formats, includingLucene104HnswScalarQuantizedVectorsFormat):if (fieldEntry.size() == 0 || knnCollector.k() == 0) return;(codecs/lucene99/Lucene99HnswVectorsReader.java:359).KnnFloatVectorQuery(search/KnnFloatVectorQuery.java:115) and the 5-argumentLeafReader.searchNearestVectors(index/LeafReader.java:277-290) clamp k tosize()and return before the reader, so they give 0 hits on this index.- At decay
DECAY_MAX_QUALITY(1.0) the query goes straight to exact search, whose empty-valuesscorer(float[])returns null by design (OffHeapScalarQuantizedVectorValues.java:537-538): 0 hits. A filter that matches no live document returns before the search. - Executed controls: on this index the filtered
FloatVectorSimilarityQuery.Adaptive(filterMatchAllDocsQuery), the deprecatedFloatVectorSimilarityQuery.Explicitand the 4-argumentsearchNearestVectorsthrow the same NPE, whileKnnFloatVectorQueryand the 5-argument API return 0 hits;Lucene104HnswScalarQuantizedVectorsFormatandLucene99HnswVectorsFormatreturn 0 hits after the same steps;Float16VectorSimilarityQuery.Adaptiveon main throws the same NPE.
Exposure and likelihood162 words
- Opt-in format. The default codec and the HNSW scalar-quantized format return 0 hits after the same steps. The flat format,
FloatVectorSimilarityQueryand the 4-argument API are all@lucene.experimental. - No scale needed. 2 documents suffice, and no explicit
forceMergeis needed (with the default config, an NRT reader opened right after the delete already sees the emptied leaf and throws); it is data-dependent, so it tends to show up in production after deletes and a merge, not in development. - Loud, not silent. The whole search call throws (one such leaf is enough); nothing is written wrong and no result is wrong. It lasts until that segment is merged with segments that have vectors for the field: a merge keeps the field (the repro's own
forceMerge(1)is one), and adding new vector documents alone does not help. - Likelihood: rare. It needs the non-default format, a vector similarity query or the 4-argument API, and an emptied field. No public report was found since 10.4.0.
One JUnit test method on the public API: 1 vector document and 1 empty document, delete the vector document, forceMerge(1), assert the field has 0 vectors, then count a FloatVectorSimilarityQuery.Adaptive. No randomized input of its own, one test thread; it doubles as the regression test.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.codecs.lucene104;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.KnnFloatVectorField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.search.FieldExistsQuery;
import org.apache.lucene.search.FloatVectorSimilarityQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.util.LuceneTestCase;
public class TestFlatScalarQuantizedEmptyFieldSearch extends LuceneTestCase {
public void testSimilarityQueryOnFieldWithNoVectors() throws Exception {
IndexWriterConfig iwc =
new IndexWriterConfig()
.setCodec(
new Lucene104Codec() {
@Override
public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
return new Lucene104ScalarQuantizedVectorsFormat();
}
});
float[] vector = {1f, 2f, 3f, 4f};
try (Directory dir = new ByteBuffersDirectory()) {
try (IndexWriter w = new IndexWriter(dir, iwc)) {
Document doc = new Document();
doc.add(new KnnFloatVectorField("v", vector));
w.addDocument(doc);
w.addDocument(new Document());
w.deleteDocuments(new FieldExistsQuery("v"));
w.forceMerge(1);
}
try (DirectoryReader r = DirectoryReader.open(dir)) {
assertEquals(0, r.leaves().get(0).reader().getFloatVectorValues("v").size());
IndexSearcher searcher = new IndexSearcher(r);
// expected: 0 hits; actual: NullPointerException because "quantizer" is null
assertEquals(0, searcher.count(new FloatVectorSimilarityQuery.Adaptive("v", vector, 0f)));
}
}
}
}
cp $REPRO/TestFlatScalarQuantizedEmptyFieldSearch.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene104/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene104.TestFlatScalarQuantizedEmptyFieldSearch -Ptests.seed=DEADBEEF
# add -Ptests.asserts=false for production mode: the same NullPointerException
- Expected
searcher.count(...)returns 0.- Observed
NullPointerException: ... because "quantizer" is nullatLucene104ScalarQuantizedVectorScorer.getRandomQuantizedVectorScorer(Lucene104ScalarQuantizedVectorScorer.java:119), assertions on and off.
More files, environment, full output211 words
TestFlatScalarQuantizedEmptyFieldSearch (one method, extends LuceneTestCase, package org.apache.lucene.codecs.lucene104 of the lucene/core tests, Apache license header). It is the hunt's minimal test, unchanged. The test draws no random values (it uses new IndexWriterConfig() and ByteBuffersDirectory, not the randomized helpers); it was run with seed DEADBEEF, and the verifiers' own tests of the same steps failed with seeds 5EED1234, CAFEBABE and 12345678 too. It fails the same way with assertions on and off, and passes checkGoogleJavaFormat.
| lucene | main 5adff78bb168 (12.0.0-SNAPSHOT): fails; 10.6.0 and 10.5.2: fail; 10.5.1, 10.5.0 and 10.4.0: fail with the 3-argument constructor (no `Adaptive` before 10.5.2); 10.3.2 and 9.12.3: format absent |
|---|---|
| jdk | OpenJDK 25.0.3 (build 25.0.3+9-LTS) |
| os | Linux aarch64 (only this platform was run) |
cp $REPRO/TestFlatScalarQuantizedEmptyFieldSearch.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene104/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene104.TestFlatScalarQuantizedEmptyFieldSearch -Ptests.seed=DEADBEEF
# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF (assertions on; -Ptests.asserts=false gives the same NPE and frames)
TestFlatScalarQuantizedEmptyFieldSearch > testSimilarityQueryOnFieldWithNoVectors FAILED
java.lang.NullPointerException: Cannot invoke "org.apache.lucene.util.quantization.OptimizedScalarQuantizer.scalarQuantize(float[], byte[], byte, float[])" because "quantizer" is null
at __randomizedtesting.SeedInfo.seed([DEADBEEF:F58DACA4BCDA21F9]:0)
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer.getRandomQuantizedVectorScorer(Lucene104ScalarQuantizedVectorScorer.java:119)
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer.getRandomVectorScorer(Lucene104ScalarQuantizedVectorScorer.java:66)
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsReader.getRandomVectorScorer(Lucene104ScalarQuantizedVectorsReader.java:221)
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsReader.search(Lucene104ScalarQuantizedVectorsReader.java:359)
at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader.search(PerFieldKnnVectorsFormat.java:329)
at org.apache.lucene.index.CodecReader.searchNearestVectors(CodecReader.java:294)
at org.apache.lucene.search.FloatVectorSimilarityQuery.approximateSearch(FloatVectorSimilarityQuery.java:261)
at org.apache.lucene.search.AbstractVectorSimilarityQuery$1.scorerSupplier(AbstractVectorSimilarityQuery.java:202)
...
at org.apache.lucene.search.IndexSearcher.count(IndexSearcher.java:528)
at org.apache.lucene.codecs.lucene104.TestFlatScalarQuantizedEmptyFieldSearch.testSimilarityQueryOnFieldWithNoVectors(TestFlatScalarQuantizedEmptyFieldSearch.java:58)
# with the recommended guard: 1 test, 0 failures (assertions off); the repro plus 3 suites: 102 tests, 0 failures, 3 skippedtestDeleteAllVectorDocs, Lucene's shared format test, reaches this state and says "knn search doesn't fail on a field with all deleted docs", but checks only the 5-argument API.- LUCENE-10351 (#580, 2022) fixed the same kind of failure with a
size() == 0return in the HNSW reader. - The field keeps its vector
FieldInfo, so the javadoc's "undefined" case does not apply.
The full argument233 words
- The contract in Lucene's own test.
BaseKnnVectorsFormatTestCase.testDeleteAllVectorDocs(lucene/test-framework/src/java/org/apache/lucene/tests/index/BaseKnnVectorsFormatTestCase.java:893-929) adds one vector document (with aStringFieldid andDOT_PRODUCT) and one plain document, commits, deletes the first by term, runsforceMerge(1)and asserts the values have size 0; the comment at:918reads "assert that knn search doesn't fail on a field with all deleted docs".TestLucene104ScalarQuantizedVectorsFormatinherits it, but the test searches only through the 5-argumentsearchNearestVectors, which clamps k to 0 and never reaches the reader, so the gap stays untested. - The precedent. LUCENE-10351 (#580,
78da7030370, 2022-01-04, "Correct knn search failure with deleted docs") fixed the same kind of failure (a knn search on a field whose vector documents were all deleted then threwIllegalArgumentException) in the HNSW reader of the time with a reader-sidesize() == 0return, and added the knn-search assertion (the comment at:918) to the existingtestDeleteAllVectorDocs.Lucene99HnswVectorsReader.java:359keeps the guard. - The format is meant to be searched on its own. It is registered in
lucene/core/src/resources/META-INF/services/org.apache.lucene.codecs.KnnVectorsFormat:17, its reader overridessearchwith an exhaustive scan instead of the flat-reader no-op, andTestLucene104ScalarQuantizedVectorsFormat.testSearch(:95) queries it as the per-field format. - History. The guard, the empty values with a null quantizer and the scorer all came with #15169; no review comment discusses searching a field with no vectors, and no later commit adds a size guard to
search.
The devil's-advocate check117 words
The strongest counter-argument: the 4-argument API's behaviour is "undefined" for a field without vectors, the 5-argument API and KnnFloatVectorQuery show that callers clamp k to size() themselves, and everything involved is @lucene.experimental.
Why it fails. The javadoc carve-out is "doesn't have KNN vectors enabled on its FieldInfo" (index/LeafReader.java:401-402, codecs/KnnVectorsReader.java:112-113); this field keeps a vector FieldInfo, and the repro asserts non-null values of size 0. No javadoc asks callers for a size check, and Lucene's own FloatVectorSimilarityQuery calls the unguarded overload. @lucene.experimental concerns API stability, not an NPE on a writer-produced index. The cross-model refute review rated the intended-behaviour case weak; the devil's-advocate verifier considered six such arguments and rejected all of them.
Recommended Return early when the field has no vectors, in both search overloads (as the HNSW reader does)
Give this reader the HNSW reader's guard: in both search overloads, FieldEntry fi = fields.get(field); if (fi == null || fi.size == 0 || knnCollector.k() == 0) return;. The repro passes; with it the repro and 3 closest suites run 102 tests with 0 failures.
public void search(String field, float[] target, KnnCollector knnCollector, AcceptDocs acceptDocs)
throws IOException {
FieldEntry fi = fields.get(field);
if (fi == null || fi.size == 0 || knnCollector.k() == 0) return;
exhaustiveBulkScore(getRandomVectorScorer(field, target), knnCollector, acceptDocs);
}
// the same two lines in search(String field, short[] target, ...)All options and trade-offs3 options ยท 371 words
The recommended guard was built and checked through gradle on 5adff78bb168 in a scratch worktree: the repro with assertions off, then the repro with TestLucene104ScalarQuantizedVectorsFormat, TestFloatVectorSimilarityQuery and TestFloat16VectorSimilarityQuery, seed DEADBEEF, and checkGoogleJavaFormat. With the guard, a route probe returns 0 on every flat-format route, including the 4-argument API and Float16VectorSimilarityQuery on a FLOAT16 field. The hunt and the cross-model review ran the same guard with the same outcome. Options B and C were not run. The regression test is the one repro test, unchanged.
Return early when the field has no vectors, in both search overloads (as the HNSW reader does) (validated by the checks) (recommended)
In Lucene104ScalarQuantizedVectorsReader.search(String, float[], ...) and search(String, short[], ...) (:355-367), look up the field entry and return when it is missing, its size is 0 or k is 0 (+4/-2).
public void search(String field, float[] target, KnnCollector knnCollector, AcceptDocs acceptDocs)
throws IOException {
FieldEntry fi = fields.get(field);
if (fi == null || fi.size == 0 || knnCollector.k() == 0) return;
exhaustiveBulkScore(getRandomVectorScorer(field, target), knnCollector, acceptDocs);
}
// the same two lines in search(String field, short[] target, ...)Trade-offs
Local to this reader and the same shape as Lucene99HnswVectorsReader; one map lookup per search; no change for fields that have vectors.
What the checks validated
Without it the repro fails with assertions on and off; with it the repro passes with assertions off, and the repro plus the 3 suites run 102 tests, 0 failures, 3 skipped; :lucene:core:checkGoogleJavaFormat passes.
Return null from getRandomVectorScorer(String, float[]) and (String, short[]) (:216-238) when the field entry's size is 0; exhaustiveBulkScore already returns on a null scorer (:376).
Trade-offs
Also covers other callers of getRandomVectorScorer, but changes what that public method returns for an empty field, which every caller would see; needs a wider review. Not run.
Return no hits in FloatVectorSimilarityQuery.approximateSearch when the leaf's vector values have size 0, as KnnFloatVectorQuery does.
Trade-offs
Fixes the query but leaves the 4-argument searchNearestVectors and any other caller of the reader's search exposed; a reader-side guard is consistent with the approach of LUCENE-10351. Not run.
Downstream, Solr's flat ScalarQuantizedDenseVectorField (SOLR-18267) delegates to this format and offers a vector-similarity query parser, so its users can meet the same exception; no downstream report was found.
A proof-obligation audit of Lucene104ScalarQuantizedVectorsReader's contract: a search on a field whose vectors were all deleted must return no hits, as the HNSW reader's size() == 0 guard ensures. This reader's search does not discharge that obligation; a one-test public-API JUnit then witnessed the NullPointerException.
The method in full122 words
Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).
- The finder listed what the reader's
searchmust guarantee on every index the writer can produce, taking the contract fromtestDeleteAllVectorDocsand the HNSW reader's guard. The obligation "an emptied field yields no hits" is not discharged: the early return covers onlyk() == 0, and the empty values carry a null quantizer. - The finder then wrote a public-API JUnit test that fails on main with assertions on and off and on 10.6.0 and 10.5.2; a final repro check kept the minimal version as the one repro.
- An independent reproducer, an executed cross-model review, a devil's-advocate verifier and an adjudication reproduced it ("Prior reports, history and checks").
No prior report found in apache/lucene issues, PRs and commits (with the JIRA archive), the Lucene mailing lists, Elasticsearch, OpenSearch, Solr or the web (searched 2026-11-03). Nearest item: #580 / LUCENE-10351, the same failure fixed in the HNSW reader in 2022.
Searches3 searches
No matching public report was found in the documented searches. Related, not reports: #580 / LUCENE-10351 (same symptom class, HNSW reader, 2022; it sets the contract), #13195 (NPE for an absent field), #15355/#15359 (dimension check in the same scorer method), #16665/#16678 (another defect of the same reader), #14119 (empty vector values in older formats), #15257 (open, only reformats the k() == 0 line). Limits: keyword search; GitHub search returns at most 50 items per query; private reports are not visible.
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits (apache/lucene incl. the migrated JIRA archive) | 2026-11-03, re-checked the same day | two scripted prior-report searches (159 and 156 distinct hits) and about 50 manual searches on the reader, scorer, empty values, the similarity queries, the exception message and testDeleteAllVectorDocs | no report; #580, #13195, #15355/#15359 and #16665/#16678 are the nearest items |
| Open PRs | 2026-11-03 | the changed files of all 409 open apache/lucene PRs; the 27 updated since 2026-11-01 re-checked | 21 touch the involved files (the reader, scorer, empty values, writer, format, similarity queries, CodecReader or the format tests); none adds the guard, and only #15257 changes the reader (it only reformats the k() == 0 line) |
| Downstream trackers, mailing lists, web | 2026-11-03 | Elasticsearch, OpenSearch k-NN, OpenSearch, Solr and SOLR JIRA; dev@, java-user@, issues@ and commits@; web searches | no report of this NPE; solr#4698 shows Solr's flat field uses the format |
Since the format was added (#15169, 594eeb7e637, 2025-09-17; first release 10.4.0). Every release run from 10.4.0 to 10.6.0 fails; 10.3.2 and 9.12.3 do not have the format. Not fixed upstream.
Versions tested4 rows
Introduced by #15169 "Add a new codec to implement OSQ for 4 and 8 bit quantized vectors" (@mccullocht, co-author @benwtrent): main 594eeb7e637, branch_10x 07033dcd84c, both author-dated 2025-09-17. That commit created the reader with the k() == 0-only guard, the empty values with a null quantizer, and the scorer. No later commit adds a size guard to search. The float16 overload with the same guard came with #16473 (50c1e6acb7f, 2026-08-26, main only, unreleased). Not new in 10.6. Not fixed upstream: upstream main 1c2072f02843 (2026-11-02) still returns early only for k() == 0.
Related, separate fix site: the older flat Lucene102BinaryQuantizedVectorsReader.search has the same k() == 0-only guard (since #14078, first release 10.2.0); a probe fails the same way on 10.2.0 and 10.3.2. Its read-only copy in backward-codecs keeps that shape on main (Lucene102BinaryQuantizedVectorsReader.java:317-321, not run on 10.4 or later).
| Version or commit | Verdict | Result |
|---|---|---|
main 5adff78bb168 | affected | gradle and the standalone runner: fails with assertions on and off; passes with the guard |
| 10.6.0, 10.5.2 | affected | the doc's test: the same NullPointerException, assertions on and off |
| 10.5.1, 10.5.0, 10.4.0 | affected | the doc's test with the 3-argument FloatVectorSimilarityQuery constructor (Adaptive first appears in 10.5.2): the same NullPointerException |
| 10.3.2, 9.12.3 | not affected | the format does not exist (compile error) |
A finder, an independent reproducer, an executed cross-model review (OpenAI GPT-5.6, refute lens), a devil's-advocate verifier, an adjudication and the audit re-run reproduced it and agree on the root cause; a final repro check confirmed it. This doc's gradle runs reproduced it and validated the guard.
Checks and verdicts353 words
The cross-model review ran on main ea57360912ed, whose reader, scorer, empty values and query classes are byte-identical to 5adff78bb168. All other main runs used 5adff78bb168; the release runs used each release's own jars.
| Check | Verdict | What it found |
|---|---|---|
| finder | reproduced | Own test fails on main (assertions on and off), 10.6.0 and 10.5.2; the guard makes it pass and keeps TestLucene104ScalarQuantizedVectorsFormat green. |
| independent reproducer | real | Own test, written from the claim, fails on main, 10.6.0, 10.5.2 and 10.4.0; controls: the filtered query, the 4-argument API and the float16 query throw, KnnFloatVectorQuery, the 5-argument API and the HNSW formats return 0 hits. |
| cross-model review, refute lens (OpenAI GPT-5.6, executed) | real (high) | Own test fails with assertions on and off and with 3 seeds; a live-vector control passes; the guard makes it pass and the format suite stays green; the intended argument rated weak. |
| devil's-advocate verifier | real (high) | Six intended, documented, misuse or unreachable arguments considered and rejected; re-ran main and 10.6.0: fail. |
| adjudication | confirmed, Low | Re-ran the reproducer's test on main (assertions on and off): fails; corrected the introducing commit to the main hash. |
| audit re-run, final repro check | reproduced; confirmed | The one-test repro fails on main with assertions on and off; the final check kept the minimal repro (fails on main, passes with the guard). |
| this doc | reproduced; guard validated | Gradle in a scratch worktree: fails without the guard (assertions on and off), passes with it; 102 tests, 0 failures with the guard. |
| fact checks of this doc (independent check, and a cross-model check by OpenAI GPT-5.6) | real, Low; wording fixed | The independent check re-ran the repro through gradle (fails with assertions on and off, passes with the guard), every route the doc names, and the doc's test on 10.4.0 to 10.6.0 (all fail); the cross-model check re-read the code and the logs. Both agree the bug is real. Their corrections are applied: a merge with a segment that holds no vectors does not end the failure, and the release runs are now cited per version. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 476 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,514 words
- Registry title
- With the flat Lucene104ScalarQuantizedVectorsFormat, a FloatVectorSimilarityQuery (or the 4-argument LeafReader.searchNearestVectors) on a segment whose float vector field has no vectors left (every vector document deleted and merged away) throws NullPointerException ('quantizer' is null) instead of returning no hits (10.4.0 through 10.6.0 and main)
- Area
- codecs / Lucene104 scalar-quantized vectors (flat format search on a field with no vectors: Lucene104ScalarQuantizedVectorsReader.search)
- Priority
- Low, rank 693: With the opt-in flat Lucene104 scalar-quantized format, a FloatVectorSimilarityQuery or the 4-argument searchNearestVectors throws NullPointerException on a segment whose vector documents were all deleted and merged away, in releases 10.4.0 through 10.6.0. It needs that non-default format plus an emptied vector field, and the failure is loud with nothing written wrong.
- Impact class
- crash
- Impact note
- Only indexes that choose the opt-in flat Lucene104ScalarQuantizedVectorsFormat for a float vector field are hit, and only while a segment lists that field with zero vectors (every vector document of the merged segments deleted while other documents survived): IndexSearcher.search with a FloatVectorSimilarityQuery at its default decay (unfiltered, or with a filter that matches a live document), and every 4-argument LeafReader.searchNearestVectors, on that index then throws NullPointerException [exec, -ea and -da], while KnnFloatVectorQuery, the 5-argument searchNearestVectors, the default codec and the HNSW scalar-quantized format return no hits; no scale is needed, nothing is written wrong, and releases 10.4.0 through 10.6.0 and main are affected.
- Summary
- You index float vectors with the flat Lucene104ScalarQuantizedVectorsFormat, delete every document that has a vector while other documents remain, and merge, so the merged segment still lists the vector field but holds no vectors (the same index Lucene's own test testDeleteAllVectorDocs builds). You expect a FloatVectorSimilarityQuery on that index to return no hits, as it does with the HNSW formats and as KnnFloatVectorQuery does on the same index. Instead IndexSearcher.search throws NullPointerException ('quantizer' is null), because the reader's search() returns early only when k is 0 and then builds a scorer over the empty vector values, whose quantizer is null.
- Introduced
- 594eeb7e637c94d1018cdb938a5596de9034c0bc on main, cherry-picked to branch_10x as 07033dcd84ccbbcf13bc3a81a6430445d2e4d3fa (the same change): #15169 'Add a new codec to implement OSQ for 4 and 8 bit quantized vectors' by @mccullocht (co-author @benwtrent; merged 2025-09-17, milestone 10.4.0) [read: history clone; the commit created Lucene104ScalarQuantizedVectorsReader with search() guarding only k() == 0, EmptyOffHeapVectorValues with a null quantizer and centroid, and Lucene104ScalarQuantizedVectorScorer, which dereferences getQuantizer(); no later commit adds a size guard to search()]. The FLOAT16 overload search(String, short[], ...) with the same guard came with 50c1e6acb7f (#16473, author 2026-08-26, main only, unreleased). (2025-09-17 (author date 2025-09-17T15:37:18-07:00 of both the main and the branch_10x commit; first release 10.4.0))
- First affected version
- 10.4.0 [exec: the one-test repro with its only 10.4 adaptation, the 3-argument FloatVectorSimilarityQuery constructor because FloatVectorSimilarityQuery.Adaptive does not exist in 10.4.0, fails with the same NullPointerException]; 10.3.2 and 9.12.3 are not affected because the format does not exist there [exec: compile error, Lucene104Codec and Lucene104ScalarQuantizedVectorsFormat absent]
- Affected versions
- 10.4.0, 10.5.0, 10.5.1, 10.5.2, 10.6.0 (released 2026-10-29, 4ae946e5a9b) and main 5adff78bb168 (12.0.0-SNAPSHOT); not fixed anywhere (upstream main 1c2072f02843, 2026-11-02, still returns early only for k() == 0 in both search overloads). [exec, JDK 25.0.3, Linux aarch64] fail: main -ea and -da (finder, the independent reproduction, the intended-behaviour check, adjudication, follow-up), 10.6.0 -ea (finder, the independent reproduction, the intended-behaviour check, follow-up), 10.5.2 -ea (finder, follow-up) and with -ea and -da (the independent reproduction), 10.4.0 -ea with the adapted constructor (finder probe, the independent reproduction, follow-up); 10.5.0 and 10.5.1 -da with the adapted constructor (the doc's fact check, which also re-ran 10.4.0, 10.5.2 and 10.6.0: all fail). Not affected: 10.3.2 and 9.12.3 (format absent) [exec]. The FLOAT16 path (KnnFloat16VectorField with Float16VectorSimilarityQuery) throws the same NullPointerException on main [exec, the independent reproduction control], main only. Not new in 10.6 (10.4.0 and 10.5.2 fail).
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal: proof-obligation audit
- Upstream and prior reports
- none found (2026-11-03, on top of the finder's, the independent reproduction's, the intended-behaviour check's and the adjudication's searches; rechecked the same day): two scripted prior-report searches (the finder's, 156 distinct hits, 0 high; the follow-up's, 159 distinct hits, 0 high, 2 medium: #16665/#16678, another defect of the same reader) over apache/lucene issues, PRs and commits, Elasticsearch, OpenSearch and Solr, the dev@, java-user@, issues@ and commits@ lists, the JIRA archive and the web; about 50 manual GitHub searches (apache/lucene issues, PRs and commits; Elasticsearch, OpenSearch and Solr); the changed files of all 409 open apache/lucene PRs (15 touch the reader, scorer, empty values, similarity queries, CodecReader or BaseKnnVectorsFormatTestCase, none adds the guard; #15257 only reformats the k() == 0 line) and of the 27 open PRs updated since 2026-11-01 (#16965 and #16847 touch the query classes, not the guard); the commit history of the reader on upstream main (last change #16883, author 2026-10-24). Closest items, none the same bug: #580 / LUCENE-10351 (2022: the same symptom class fixed inside Lucene90HnswVectorsReader, which added testDeleteAllVectorDocs, the contract this breaks), #13195 (NPE in searchNearestVectors for a field absent from FieldInfos), #15355/#15359 (dimension check in the same scorer method), #16665/#16678 (COSINE merge scorer of the same reader), #14119 (EmptyOffHeapVectorValues accept ords in older formats), #13626 (merge-time NPE in another flat reader).
- Lucene commit tested
- main
5adff78bb168(2026-11-02) through gradle on 2026-11-03 in a scratch worktree: the repro with assertions on and off without the fix (fails), with the recommended guard (passes), and the repro plusTestLucene104ScalarQuantizedVectorsFormat,TestFloatVectorSimilarityQueryandTestFloat16VectorSimilarityQuerywith the guard (102 tests, 0 failures, 3 skipped), pluscheckGoogleJavaFormat. The hunt ran the same test through a standalone runner on main (-ea,-da). The fact check ran it on the release jars: 10.6.0 and 10.5.2 (-ea,-da) and, with only the constructor changed to the 3-argument one becauseAdaptivefirst appears in 10.5.2, 10.5.1, 10.5.0 and 10.4.0: all fail; 10.3.2 does not compile (the format is absent; 9.12.3 too, in the hunt's runs of an earlier test of the same steps). Probes on main also showed that the failure outlasts a merge with a segment that holds no vectors and the addition of new vector documents (only merging it with a segment that holds vectors ends it), and that the deprecatedExplicitvariant fails too. Only Linux aarch64 with OpenJDK 25.0.3 was run.
- Finding record
lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsReader.java:354-367 (at 5adff78bb168) (search(String, float[], ...) and search(String, short[], ...): 'if (knnCollector.k() == 0) return;' then exhaustiveBulkScore(getRandomVectorScorer(field, target), ...)), :214-222 (getRandomVectorScorer over loadQuantizedVectorValues(fi))lucene/core/src/java/org/apache/lucene/codecs/lucene104/OffHeapScalarQuantizedVectorValues.java:210-211 (at 5adff78bb168) (load returns EmptyOffHeapVectorValues for an empty ord-to-doc configuration), :503-518 (EmptyOffHeapVectorValues passes a null centroid and a null quantizer)lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorScorer.java:60-67 (at 5adff78bb168) (getRandomVectorScorer: QuantizedByteVectorValues branch), :102-119 (quantizer = qv.getQuantizer(); quantizer.scalarQuantize(...) at :119)lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsReader.java:359 (at 5adff78bb168) ('if (fieldEntry.size() == 0 || knnCollector.k() == 0)')lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityQuery.java:252-263 (at 5adff78bb168) (approximateSearch: 4-argument searchNearestVectors, no size check), lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java:198-206 (approximateSearch per leaf), lucene/core/src/java/org/apache/lucene/index/CodecReader.java:283-295 (4-argument searchNearestVectors checks only the FieldInfo)lucene/core/src/java/org/apache/lucene/search/KnnFloatVectorQuery.java:115 (at 5adff78bb168) and lucene/core/src/java/org/apache/lucene/index/LeafReader.java:277-290 (the guarded callers: min(k, size()) == 0 returns)lucene/test-framework/src/java/org/apache/lucene/tests/index/BaseKnnVectorsFormatTestCase.java:893-929 (at 5adff78bb168) (testDeleteAllVectorDocs; comment at :918 'assert that knn search doesn't fail on a field with all deleted docs'; checks only the 5-argument searchNearestVectors)lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene102/Lucene102BinaryQuantizedVectorsReader.java:317-321 (at 5adff78bb168) (older sibling with the same k() == 0-only guard; separate fix site)SECURITY.md:28 (Scope and Threat Model), :42 (Other out-of-scope reports), :50 (In-scope reports)
- The finding in 30 seconds
lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsReader.java:216-222 (at 5adff78bb168) (getRandomVectorScorer), :355-367 (both search overloads, guard at :358 and :365), :376 (exhaustiveBulkScore returns on a null scorer)lucene/core/src/java/org/apache/lucene/codecs/lucene104/OffHeapScalarQuantizedVectorValues.java:210-211 (at 5adff78bb168) (load returns EmptyOffHeapVectorValues), :503-518 (null centroid and null quantizer), :537-538 (scorer(float[]) returns null)lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorScorer.java:60-67, (at 5adff78bb168) :105 (qv.getQuantizer()), :119 (quantizer.scalarQuantize)lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsReader.java:359 (at 5adff78bb168) ('if (fieldEntry.size() == 0 || knnCollector.k() == 0)')lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityQuery.java:254-263 (at 5adff78bb168) (approximateSearch: 4-argument searchNearestVectors, no size check)lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java:51-52 (at 5adff78bb168) (DEFAULT_DECAY 0.5, DECAY_MAX_QUALITY 1), :190-206 (exact path at DECAY_MAX_QUALITY, else approximateSearch)lucene/core/src/java/org/apache/lucene/index/CodecReader.java:283-295 (at 5adff78bb168) (checks only the FieldInfo)lucene/core/src/java/org/apache/lucene/search/KnnFloatVectorQuery.java:115 (at 5adff78bb168) and lucene/core/src/java/org/apache/lucene/index/LeafReader.java:277-290 (clamp k to size(), return before the reader)
- Who is affected
lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsFormat.java:97 (at 5adff78bb168) (@lucene.experimental), lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityQuery.java:34 (@lucene.experimental)
- Why it looks unintended
lucene/test-framework/src/java/org/apache/lucene/tests/index/BaseKnnVectorsFormatTestCase.java:893-929 (at 5adff78bb168) (comment :918)lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsReader.java:359 (at 5adff78bb168)lucene/core/src/resources/META-INF/services/org.apache.lucene.codecs.KnnVectorsFormat:17; (at 5adff78bb168) lucene/core/src/test/org/apache/lucene/codecs/lucene104/TestLucene104ScalarQuantizedVectorsFormat.java:76, :95lucene/core/src/java/org/apache/lucene/index/LeafReader.java:401-402, (at 5adff78bb168) lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsReader.java:112-113
- Suggested fix
lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsReader.java:216-238, (at 5adff78bb168) :355-367, :376; lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsReader.java:359; lucene/core/src/java/org/apache/lucene/search/KnnFloatVectorQuery.java:115
- Diagrams
lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsReader.java:216-222 (at 5adff78bb168) (getRandomVectorScorer), :355-367 (both search overloads, guard at :358 and :365), :376 (exhaustiveBulkScore returns on a null scorer)lucene/core/src/java/org/apache/lucene/codecs/lucene104/OffHeapScalarQuantizedVectorValues.java:210-211 (at 5adff78bb168) (load returns EmptyOffHeapVectorValues), :503-518 (null centroid and null quantizer), :537-538 (scorer(float[]) returns null)lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorScorer.java:60-67, (at 5adff78bb168) :105 (qv.getQuantizer()), :119 (quantizer.scalarQuantize)lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsReader.java:359 (at 5adff78bb168) ('if (fieldEntry.size() == 0 || knnCollector.k() == 0)')
- TestFlatScalarQuantizedEmptyFieldSearch.java (lucene/core, package org.apache.lucene.codecs.lucene104, 1 test):
repro/LFV-1176/TestFlatScalarQuantizedEmptyFieldSearch.java(62 lines)
The diagram follows the repro's steps on main 5adff78bb168 (Lucene104ScalarQuantizedVectorsReader.java:356-360, OffHeapScalarQuantizedVectorValues.java:210-211, :503-518, Lucene104ScalarQuantizedVectorScorer.java:105, :119, Lucene99HnswVectorsReader.java:359).
assets/LFV-1176/bug.png(476 KB)