Both javadocs list a field's metadata entry as field number, then similarity; both writers store an int32 vector encoding in between. Since 10.3 the .vex list also calls the neighbour deltas vints, though they are group-varint.
- Each vector field of a segment has a metadata entry in two small files of the default codec:
.vemf(the raw vectors,Lucene99FlatVectorsFormat) and.vem(the HNSW graph,Lucene99HnswVectorsFormat). The entry says which field it is, how its values are stored (the encoding: BYTE or FLOAT32 in the releases; main also has FLOAT16), which similarity it uses, and where its data lives. - The graph itself is in
.vex: for each node, a neighbour count and the sorted neighbour ordinals stored as differences (deltas). - Each codec format documents its files in its class javadoc as a typed list (
[int32],[vlong],[vint]), and the default codec's overview sends readers there for the details.
![Diagram of the bug: the .vemf and .vem entry of a 2-dimensional COSINE float vector field, as the javadoc documents it ([int32] field number, [int32] vector similarity function ordinal, [vlong] offset, ...) and as the writer writes it, with the int32 vector encoding ordinal (1, FLOAT32) between the field number and the similarity (2, COSINE). Where the javadoc expects the similarity it reads 1, DOT_PRODUCT, and from there on every value is misread (the documented offset starts on the real similarity, 4 bytes early). A side box shows node 0's level-0 neighbour list in a separate 1000-vector index: Lucene's graph, and the same group-varint bytes read as vints.](../assets/LFV-941/bug.png)
- You write your own reader for the vector metadata files from the javadocs of
Lucene99FlatVectorsFormat(.vemf) andLucene99HnswVectorsFormat(.vem). Both say each field's entry is [int32] field number, [int32] vector similarity function ordinal, [vlong] offset, [vlong] length, [vint] dimension, [int] count, ... - You index 1 document with a 2-dimensional COSINE
KnnFloatVectorField(default codec), read field number 0, then the similarity. You expect 2: COSINE in the format's list EUCLIDEAN 0, DOT_PRODUCT 1, COSINE 2, MAXIMUM_INNER_PRODUCT 3. - You get 1 in both files, which means DOT_PRODUCT: a valid value, so reading it raises no error. It is the vector encoding (FLOAT32 = 1), an int32 both writers store between the two. The real similarity comes next, and from there on the parser is out of step (the documented offset starts on the real similarity, 4 bytes early), so every later value is misread: the dimension reads 0 instead of 2.
- Since 10.3.0 the
.vexgraph file is off too: the javadoc says the neighbour deltas are vints, but they are group-varint, with a flag byte before every 4 deltas. In a bigger index (the finder's test: 1000 random 4-dimensional vectors), node 0's neighbours 1, 2, 5, 9, ... read back as 0, 1, 2, 5, 9, ... Lucene's own readers match the writers: only the javadoc is wrong.
People who read or re-implement the vector files from these javadocs: ports, index inspection or recovery tools, format write-ups. For them every vector field the default codec writes goes wrong at the second value, and nothing flags it there, since the wrong similarity is a valid one. Lucene's own indexing, merging and kNN search are unaffected. Priority Low.
Mechanism, terms and numbers796 words
Lucene99HnswVectorsFormat is the vectors format of the default codec, and it stores the raw vectors with Lucene99FlatVectorsFormat. Their class javadocs document each field's metadata entry in .vemf and .vem as a typed list that goes from [int32] field number straight to [int32] vector similarity function ordinal. Since LUCENE-10577 (9.4.0, 2022) the writers store one more int32 between the two, the vector encoding ordinal, and the readers read it there; the list was copied without it into both Lucene99 classes when they were created in 9.9.0. A parser written from the javadoc therefore reads the encoding as the similarity (every FLOAT32 field comes out as DOT_PRODUCT, every BYTE field as EUCLIDEAN, valid values, so nothing flags the first misread), and from there on it is out of step (the documented offset starts on the real similarity, 4 bytes early), so every later value of the entry is misread. Separately, since #14932 (10.3.0, 2025) the .vex neighbour deltas are written with group-varint, while the .vex list still says array[vint], so a vint reader decodes wrong neighbours from the first list of 4 or more neighbours on. It is a documentation defect: Lucene's writers and readers agree, so nothing goes wrong at runtime; the fix is javadoc only.
* <h2>.vemf (vector metadata) file</h2> // Lucene99FlatVectorsFormat.java:48
* <li><b>[int32]</b> field number // :53
* <li><b>[int32]</b> vector similarity function ordinal // :54
* <li><b>[vlong]</b> offset to this field's vectors in the .vec file
* ...
* <h2>.vex (vector index)</h2> // Lucene99HnswVectorsFormat.java:43
* <li><b>[vint]</b> the number of neighbor nodes // :53
* <li><b>array[vint]</b> the delta encoded neighbor ordinals // :54
* <h2>.vem (vector metadata) file</h2> // :62
* <li><b>[int32]</b> field number // :67
* <li><b>[int32]</b> vector similarity function ordinal // :68Lucene99FlatVectorsWriter.writeMeta(:342-351):writeInt(field.number),writeInt(field.getVectorEncoding().ordinal()),writeInt(similarity ordinal),writeVLong(offset),writeVLong(length),writeVInt(dimension),writeInt(count).Lucene99HnswVectorsWriter.writeMeta(:615-621) writes the same first seven values.Lucene99FlatVectorsReader(:482-487) andLucene99HnswVectorsReader(:249-250) readreadVectorEncodingbeforereadSimilarityFunction. Ordinals:VectorEncodingBYTE 0, FLOAT32 1 (VectorEncoding.java:29,:32); similarities EUCLIDEAN 0, DOT_PRODUCT 1, COSINE 2, MAXIMUM_INNER_PRODUCT 3 (Lucene99HnswVectorsReader.java:224-229)..vex:Lucene99HnswVectorsWriterwriteswriteVInt(count)and thenwriteGroupVInts(deltas, count)whenversion >= VERSION_GROUPVARINT(writeGraph:591-597, used by the unsorted flush and by merges;reconstructAndWriteNeighbours:422-428, on the index-sorted flush path);VERSION_CURRENT = VERSION_GROUPVARINT = 1(Lucene99HnswVectorsFormat.java:92-94); the reader mirrors it (Lucene99HnswVectorsReader.java:605-619). A package-private constructor's javadoc in the same file already says "VarInt=0, GroupVarInt=1" (:261).
| bytes after the header | writer's meaning | javadoc's meaning |
|---|---|---|
00 00 00 00 | field number 0 | field number 0 |
01 00 00 00 | vector encoding 1 (FLOAT32) | similarity 1 = DOT_PRODUCT |
02 00 00 00 | similarity 2 = COSINE | offset 2, length 0, dimension 0, then count from the next bytes |
| ... | offset 4096 (a 2-byte vlong), length 8, dimension 2, count 1 | out of step: length, dimension and count are read from bytes that start 5 bytes before the real ones |
The documented offset starts on the real similarity int32, 4 bytes early, and the documented offset, length and dimension read its first three bytes, so the documented dimension reads 0 for every field (the similarity ordinal is at most 3); the finder's 4-dimensional field reads 0 too. The shift is not constant: in .vemf the real offset is a 2-byte vlong, so the later values start 5 bytes early; in .vem (a 1-byte offset here) 4 bytes early.
Node 0 has 19 neighbours 1, 2, 5, 9, 10, 17, ..., so the deltas are 1, 1, 3, 4 | 1, 7, 3, 8 | 3, 32, 10, 12 | 2, 117, 108, 112 | 137, 176, 7. Group-varint (GroupVIntUtil.java:144-173) writes each group of 4 as one flag byte with the four byte lengths, then the values in 1-4 little-endian bytes, and a tail of fewer than 4 as vints: 00 01 01 03 04 | 00 01 07 03 08 | .... A vint reader of 19 deltas reads 0, 1, 1, 3, 4, 0, 1, 7, ... (each flag byte is a delta of 0), so the running sums are 0, 1, 2, 5, 9, 9, 10, 17, ..., and it stops 6 bytes before the end of the 25-byte list. Lists of 1 to 3 neighbours look the same in both encodings; every list of 4 or more is misread, because the vint reader takes each flag byte for a delta.
On main VectorEncoding also has FLOAT16 (ordinal 2, #16383, in no release yet), whose documented similarity would read as COSINE (read in code, not run).
Exposure and likelihood282 words
- Who. Only people who decode
.vemf,.vemor.vexthemselves from these javadocs. Lucene's own code never consults the text. Outside ports exist: one closely transcribes the stale text, two others document the real layout, traced from the writer; none reported the javadoc. - Default codec, every vector field.
Codec.getDefault()isLucene104(Codec.java:59), whose default vectors format isLucene99HnswVectorsFormat(Lucene104Codec.java:123), which stores raw vectors withLucene99FlatVectorsFormat(Lucene99HnswVectorsFormat.java:155-156). The codec overview linksLucene99HnswVectorsFormatas "Vector values" (package-info.java:183,:313) and says "Details on each of these are provided in their linked pages" (:188). So everyKnnFloatVectorFieldorKnnByteVectorFieldwritten with the default codec has the extra int32 (in.vemsince 9.4.0, in.vemfsince 9.9.0, when that file was created), and every field whose segment gets a graph has group-varint.vexlists (tiny segments skip the graph). - What they get. A similarity that is wrong but plausible (DOT_PRODUCT for every float field, EUCLIDEAN for every byte field), then wrong offsets, lengths, dimensions and counts; and wrong neighbour ordinals from the first list of 4 or more neighbours on. A parser that checks later fields may notice the zero dimension, an implausible count or a missed footer; nothing flags the similarity itself.
- What nobody gets. Any runtime effect: writers and readers agree, and the six upstream
codecs.lucene99test classes, the vector format round-trip suites among them, pass with the fix applied (212 tests, 9 skipped). - Scale and priority. Every release from 9.4.0 (2022) for the metadata part and from 10.3.0 (2025) for
.vex, through 10.6.0 and main; nobody has reported it in four years, which fits the small audience. Impact minor, priority Low.
One JUnit class, one test, no randomness. It indexes 1 document with a 2-dimensional COSINE float vector (default codec, no compound file) and reads, in .vemf and .vem, the field number and then the int32 the javadoc calls the similarity. Put it in lucene/core's tests and run it with gradle.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.codecs.lucene99;
import java.io.IOException;
import java.util.List;
import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.KnnFloatVectorField;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.SegmentInfos;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.IOContext;
import org.apache.lucene.store.IndexInput;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.tests.util.TestUtil;
public class TestLucene99VectorsFormatsMetaJavadoc extends LuceneTestCase {
private static int similarityOrdinal(Directory dir, String ext, String codec, byte[] id)
throws IOException {
String suffix = "Lucene99HnswVectorsFormat_0";
try (IndexInput in = dir.openInput("_0_" + suffix + "." + ext, IOContext.READONCE)) {
CodecUtil.checkIndexHeader(in, codec, 0, Integer.MAX_VALUE, id, suffix);
assertEquals("field number", 0, in.readInt());
return in.readInt();
}
}
public void testDocumentedFieldSequence() throws Exception {
try (Directory dir = new ByteBuffersDirectory()) {
IndexWriterConfig iwc =
new IndexWriterConfig().setCodec(TestUtil.getDefaultCodec()).setUseCompoundFile(false);
try (IndexWriter w = new IndexWriter(dir, iwc)) {
Document doc = new Document();
doc.add(new KnnFloatVectorField("v", new float[] {1, 0}, VectorSimilarityFunction.COSINE));
w.addDocument(doc);
}
byte[] id = SegmentInfos.readLatestCommit(dir).info(0).info.getId();
int cosine = VectorSimilarityFunction.COSINE.ordinal();
// expected COSINE (2) in .vemf and .vem; actual 1 in both: the undocumented FLOAT32 encoding
assertEquals(
"[int32] vector similarity function ordinal in .vemf, .vem",
List.of(cosine, cosine),
List.of(
similarityOrdinal(dir, "vemf", "Lucene99FlatVectorsFormatMeta", id),
similarityOrdinal(dir, "vem", "Lucene99HnswVectorsFormatMeta", id)));
}
}
}
./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene99.TestLucene99VectorsFormatsMetaJavadoc -Ptests.seed=DEADBEEF -Ptests.asserts=true
- Expected
- The int32 after the field number, documented as the similarity ordinal, is 2 (COSINE) in both
.vemfand.vem. - Observed
- It is 1 in both (
expected:<[2, 2]> but was:<[1, 1]>): the FLOAT32 encoding, which reads as DOT_PRODUCT; with assertions on and off.
More files, environment, full output370 words
No package-private production code (IndexWriter, Directory.openInput, CodecUtil.checkIndexHeader, IndexInput reads, the default codec, plus the test framework; CodecUtil and SegmentInfos are @lucene.experimental), single thread, in the lucene99 codec test package like the upstream vector format tests. The file names assume the first segment _0 and the default per-field suffix Lucene99HnswVectorsFormat_0, which is what the default codec writes for this index. The same file fails unchanged on the 10.6.0 release.
The .vex part is not in this test on purpose (it needs a real graph, hundreds of vectors). The finder's separate three-check test shows it on main 645d4fe14a20: level 0 neighbours of node 0 expected:<[1, 2, 5, 9, 10, 17, 20, 28, 31, 63, 73, 85, 87, 204, 312, 424, 561, 737, 744]> but was:<[0, 1, 2, 5, 9, 10, 17, 20, 28, 31, 63, 73, 85, 87, 204, 312]>; on 10.2.2 and 9.12.3, before group-varint, the same check passes.
Regression test: optional. A javadoc fix writes the same bytes, so this test keeps failing after the fix (checked); it is evidence for the defect, not a regression test. If a test is wanted, use this same test with the corrected list: add import org.apache.lucene.index.VectorEncoding; and, in similarityOrdinal, add assertEquals("vector encoding ordinal", VectorEncoding.FLOAT32.ordinal(), in.readInt()); after the field number (optionally also read the offset, length, dimension 2 and count 1). That version passes on stock and with the fix.
| lucene | main 645d4fe14a20 (11.0.0-SNAPSHOT); the same failure on the 10.6.0 release |
|---|---|
| jdk | OpenJDK 25.0.3 (build 25.0.3+9-LTS) |
| os | Linux, aarch64 |
cp $REPRO/TestLucene99VectorsFormatsMetaJavadoc.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene99/
cd $LUCENE
./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene99.TestLucene99VectorsFormatsMetaJavadoc -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false (same failure)
# main 645d4fe14a20, ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene99.TestLucene99VectorsFormatsMetaJavadoc -Ptests.seed=DEADBEEF -Ptests.asserts=true
TestLucene99VectorsFormatsMetaJavadoc > testDocumentedFieldSequence FAILED
java.lang.AssertionError: [int32] vector similarity function ordinal in .vemf, .vem expected:<[2, 2]> but was:<[1, 1]>
at __randomizedtesting.SeedInfo.seed([DEADBEEF:AF4ECEFFEFFD713C]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.apache.lucene.codecs.lucene99.TestLucene99VectorsFormatsMetaJavadoc.testDocumentedFieldSequence(TestLucene99VectorsFormatsMetaJavadoc.java:59)
1 test completed, 1 failed
BUILD FAILED
# the same with -Ptests.asserts=false: the same AssertionError, 1 test completed, 1 failed
# 10.6.0 release (4ae946e5a9b9), the same file in a separate re-run on the release jars (no gradle), -ea, seed DEADBEEF
FAIL tests_run=1 failures=1: [int32] vector similarity function ordinal in .vemf, .vem expected:<[2, 2]> but was:<[1, 1]>- Both javadocs give exact typed lists, the codec overview sends readers to them, and the sibling
Lucene104ScalarQuantizedVectorsFormatdocuments this "vector encoding ordinal". - The lines were right when written; the 9.4.0 and 10.3.0 writer changes left them; a constructor comment in the file says "VarInt=0, GroupVarInt=1".
- #13548 refreshed these lists in 2024; upstream fixes format-doc drift as bugs (LUCENE-713, LUCENE-7532).
The full argument273 words
- Typed, ordered lists that read as an exact grammar. Both classes list the elements in order with a type per element (
[int32],[vlong],[vint],array[vint]). Nothing marks them as approximate, andpackage-info.java:188sends readers to them for the details. The siblingLucene104ScalarQuantizedVectorsFormatdocuments "int the field number / int the vector encoding ordinal / int the vector similarity ordinal" (:84-86). - They used to be exact.
[int32] field numbercomes from LUCENE-9844 (5b36af3cd79, 2021), the similarity line from6d4b5eaba35(2021), the.vexvint lines from #11860 (d0be9ab57c9, 2022), each correct when written; the study's lineage probe reads the documented prefix of a 9.3.0.vemcorrectly and fails from 9.4.0. - Oversights, not decisions. LUCENE-10577 (#1054) added the encoding write to the then-default HNSW writer and edited other lines of that javadoc, but not these; #12582 and #12729 created the two Lucene99 classes with the stale list; #14932 documented group-varint in the
@paramof a package-private constructor and in its 10.3.0 CHANGES.txt entry, not in the class list. No comment in the saved #1054, #12729 and #14932 threads mentions the class lists (the #12582 thread was not saved). - Maintained as exact layouts. #13548 (2024) refreshed exactly these lists ("Javadoc for Lucene99HnswVectorsFormat is a bit outdated") and kept the omission; on #870 a reviewer asked for the lists to be kept "in order ... more precise". Upstream has fixed format-doc drift as bugs (LUCENE-713, LUCENE-7532).
- Tests. No test that compares a format javadoc with the bytes was found in the
lucene99codec test sources reviewed; the format suites check writer/reader agreement, which is not in question.
The devil's-advocate check147 words
The case for "intended" (rated moderate by reviewers): the format classes are @lucene.experimental (Lucene99FlatVectorsFormat.java:74, Lucene99HnswVectorsFormat.java:83); the overview "attempts to provide a high-level definition of the Apache Lucene file formats" (package-info.java:54, the LUCENE-2946 policy); version 0 .vex files really do store vints; and the supported way to read vectors is the codec's own reader, whose round trip is tested.
Why it fails: these pages do not give a high-level description, they give typed, ordered lists, and a precise list that is wrong is worse than none. The LUCENE-2946 thread kept detailed docs for simple metadata files, which these are. @lucene.experimental allows the format to change, not the description of the current format to be wrong; the default writer always writes the group-varint version and the .vex list carries no version note. At most the argument makes the fix low priority, which it is.
Recommended Correct the three list items in the two class javadocs (validated)
Javadoc only, two files (+5/-1): add [int32] vector encoding ordinal after the field number in the .vemf and .vem lists, and describe the .vex deltas as group-varint (DataOutput#writeGroupVInts; vints only in version 0 files). Validated with gradle: the corrected list matches the bytes; the upstream codecs.lucene99 tests (212, 9 skipped) and the javadoc, format and ecjLint tasks pass.
All options and trade-offs4 options ยท 689 words
One fix was validated with gradle in a scratch worktree of 645d4fe14a20; the alternatives were not run. The .vemf [int8] -2/-1/0 line is left to open PR #15430, which rewrites it.
Two files, comments only, +5/-1: insert [int32] vector encoding ordinal after [int32] field number in the .vemf list (Lucene99FlatVectorsFormat) and the .vem list (Lucene99HnswVectorsFormat); in the .vex list, say the deltas are group-varint, written with DataOutput#writeGroupVInts, and that version 0 files store vints. The order is the writers' (Lucene99FlatVectorsWriter.java:342-344, Lucene99HnswVectorsWriter.java:615-617, :591-597) and the readers'.
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsFormat.java
@@ -51,6 +51,7 @@ import org.apache.lucene.store.IndexOutput;
*
* <ul>
* <li><b>[int32]</b> field number
+ * <li><b>[int32]</b> vector encoding ordinal
* <li><b>[int32]</b> vector similarity function ordinal
* <li><b>[vlong]</b> offset to this field's vectors in the .vec file
* <li><b>[vlong]</b> length of this field's vectors, in bytes
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsFormat.java
@@ -51,7 +51,9 @@ import org.apache.lucene.util.hnsw.HnswGraphBuilder;
* <li>For each node:
* <ul>
* <li><b>[vint]</b> the number of neighbor nodes
- * <li><b>array[vint]</b> the delta encoded neighbor ordinals
+ * <li><b>array[group-varint]</b> the delta encoded neighbor ordinals, written with
+ * {@link org.apache.lucene.store.DataOutput#writeGroupVInts}; files of version 0
+ * store them as <b>array[vint]</b>
* </ul>
* </ul>
* <li>After all levels are encoded, memory offsets for each node's neighbor nodes are appended to
@@ -65,6 +67,7 @@ import org.apache.lucene.util.hnsw.HnswGraphBuilder;
*
* <ul>
* <li><b>[int32]</b> field number
+ * <li><b>[int32]</b> vector encoding ordinal
* <li><b>[int32]</b> vector similarity function ordinal
* <li><b>[vlong]</b> offset to this field's index in the .vex file
* <li><b>[vlong]</b> length of this field's index data, in bytesTrade-offs
Smallest change that makes the two claimed items true, in the two current classes only. It leaves the same stale metadata lists in the frozen backward-codecs Lucene94HnswVectorsFormat / Lucene95HnswVectorsFormat javadocs (option B), three weaker gaps in the .vem text (option C) and the stale .vemf [int8] line, which open PR #15430 rewrites. The new .vex item does not itself say that a final group of fewer than 4 deltas is written as plain vints; the linked DataOutput#writeGroupVInts documents that (DataOutput.java:328-331).
What the checks validated
- Stock: the repro fails with assertions on and off; the corrected-list copy of the same test (encoding 1, similarity 2, offset, length, dimension 2, count 1) passes, so the fix describes the bytes Lucene already writes.
- With the fix: the original repro still fails the same way (a javadoc change writes the same bytes); every class of the
org.apache.lucene.codecs.lucene99test package plus the corrected-list test: 213 tests, 9 skipped, 0 failures (TestLucene99HnswVectorsFormat60,TestLucene99HnswVectorsFormatV057,TestKnnVectorsFormatCustomWriter57,TestLucene99SegmentInfoFormat17,TestVectorsMergeReadAdvice14,TestVectorAccessHints7, corrected test 1); the corrected-list test also passes with assertions off. :lucene:core:renderJavadoc,checkGoogleJavaFormatandecjLintMainpass; both rendered pages read "[int32] field number [int32] vector encoding ordinal [int32] vector similarity function ordinal", and the HNSW page shows "array[group-varint]" with thewriteGroupVIntslink (grep counts in the run log; the rendered HTML was not kept). The eclint tool was not configured, so no eclint check ran.- Not run: the whole
:lucene:core:testand./gradlew check.
Option A plus a History entry in the codec overview (for example "In version 10.3, HNSW graph neighbor lists were changed to use group-varint encoding") and the same encoding line in the backward-codecs Lucene94HnswVectorsFormat and Lucene95HnswVectorsFormat javadocs (:74-75 in each).
Trade-offs
Three more files, two of them frozen read-only formats that some reviewers prefer not to touch.
Read in code, not claimed as the headline: the -1 end-of-fields marker (Lucene99HnswVectorsWriter.java:243, Lucene99FlatVectorsWriter.java:171) is undocumented; the per-level node count and list are written only for levels above 0 (Lucene99HnswVectorsWriter.java:630-648); and the graph-offsets metadata after the levels (:650-664: an int64 offset, a vint block shift, the DirectMonotonicWriter metadata and an int64 length) is not listed.
Trade-offs
A larger rewrite of the .vem list; worth it if a reviewer wants the lists complete.
Drop the encoding int32 or go back to vints.
Trade-offs
Changes the on-disk format of every vector field to fix a documentation problem.
Products built on Lucene read vectors through the codec and are unaffected; only tools that parse .vemf, .vem or .vex themselves from these javadocs misread them.
A semi-formal differential check. Each default-codec file grammar was written down twice, from the class javadoc and from the writer, and one interpreter parsed the bytes of 64 generated indexes with both. The writer grammar parsed every file; under the javadoc grammar all 106 .vemf and 106 .vem files diverged, and so did every .vex with a graph.
The method in full137 words
- Hunt. The grammars are data in a small grammar language; the interpreter compares every parsed value with Lucene's own reader. 64 index configurations (4 fixed and 60 seeded random), 1,114 file checks, assertions on and off.
- Controls. The writer grammar parses every file exactly to the footer (0 failures). Each of 14 single-element mutants of the writer grammar is detected, including dropping the
.vemfor.vemvector encoding, so the check is not vacuous. - Result. 106 of 106
.vemfand 106 of 106.vemfiles diverge under the javadoc grammar; the 6.vexfiles that hold a graph diverge under vint reading and parse under group-varint. The same unit found the sibling file-format javadoc defects of.fnm,.si,segments_N,.nvmand the terms dictionary. - Minimized to the one-test repro of the Minimal repro section.
No public report found (GitHub issues, PRs and commits with the JIRA archive, the Lucene mailing lists, downstream trackers and the web; 2026-10-28, GitHub rechecked 2026-11-01). The nearest items are the changes that made the text stale (#1054, #12582, #12729, #14932) and the 2024 refresh that kept it (#13548).
Searches3 searches
Closest hits, none this bug: #13548 (refreshed these lists in 2024 and kept the omission); #1054 / LUCENE-10577, #12582, #12729, #14932 (made the text stale; their threads do not mention it); #16771 (the .vemf alignment NOTE only); #13158 (a user comment lists the real .vemf layout without saying the javadoc is wrong); open PR #15430 (rewrites only the adjacent [int8] line, left out of this claim); open PR #13525 (copies the stale .vemf list into a new class). One outside port closely transcribes the stale text, two document the real layout; none filed anything.
| Where | When | How | Found |
|---|---|---|---|
| Scripted prior-report search | 2026-10-28 | two runs, 205 queries: 80 GitHub issue/PR and 11 commit queries, 64 mailing-list queries over dev@, java-user@, issues@ and commits@, 12 JIRA queries, 10 downstream-tracker queries and 28 web queries (14 per run) | 23 hits read, none the same bug |
| GitHub and web, manual | 2026-10-28 | 22 searches on the identifiers and phrases, 3 code searches, 14 web searches, 8 other checks (252 queries and checks with the scripted ones), the changed files of all 413 open apache/lucene PRs and the diffs of the six that touch the format files | no report; only #15430 edits a grammar line (the excluded [int8] line) |
| GitHub, recheck | 2026-11-01 | 7 issue/PR searches (class names with "javadoc", "vector encoding ordinal", "vemf", "vex group varint", "delta encoded neighbor ordinals", "file format javadoc vector"), the 34 items created since 2026-10-28, and the two files' upstream history | only #16771, #13158, #15630, #15341 and #13844, none a report; upstream main f815e679bf6c still has the stale lines |
The metadata part is wrong since 9.4.0: LUCENE-10577 (a693fe819b0, 2022) added the encoding int32 and kept the old list, which #12582 and #12729 copied into both Lucene99 classes (9.9.0). The .vex part since 10.3.0: #14932 (a6c99125599, 2025). 9.3.0 and 10.2.2 are the last unaffected releases; 10.6.0 and main are affected.
Versions tested6 rows
- Introduced (metadata):
a693fe819b0"LUCENE-10577: enable quantization of HNSW vectors to 8 bits (#1054)" (Michael Sokolov, 2022-08-10; the branch_9x commit87a8f7d48f3shipped in 9.4.0), which addedmeta.writeInt(field.getVectorEncoding().ordinal())to the then-defaultLucene94HnswVectorsWriter. - Carried into the named classes:
f2bf5339e53/6623c42553b(#12582, Benjamin Trent, 2023-10-24) createdLucene99HnswVectorsFormat, anda47ba3369f4/448e6112954(#12729, Benjamin Trent, 2023-11-10) createdLucene99FlatVectorsFormat, both in 9.9.0 with the stale list. - Introduced (
.vex):a6c99125599"GroupVarInt Encoding Implementation for HNSW Graphs (#14932)" (2025-08-06; branch_10xa0de4374446, first in 10.3.0). - Later edits kept both elements stale: #13548 (
ceb4539609fon main, first in 10.0.0; backport2c2f7cca0b6in 9.12.0; 2024-07-08), #13688, #15598, #16771. - Also stale: the frozen backward-codecs
Lucene94HnswVectorsFormatandLucene95HnswVectorsFormatjavadocs. - Not fixed upstream: on upstream main
f815e679bf6c(2026-11-01) the newest commits to the two files are still8a8ef488218(#16771) and79674f38df7(#16682), both before645d4fe14a20. - Limits: 9.0-9.2 not run (their formats wrote no encoding int); all runs on Linux aarch64 with JDK 25.
| Version or commit | Verdict | Result |
|---|---|---|
main 645d4fe14a20 | affected | repro fails with gradle, assertions on and off; the finder's .vex check fails too (2026-11-01) |
10.6.0 (4ae946e5a9b9) | affected | repro fails (a separate re-run on the release jars, 2026-11-01); the finder's test, .vex check included, fails on the branch_10_6 head at the same commit (study) |
| 10.3.0 - 10.5.2 | affected | metadata and .vex checks fail (study) |
| 9.9.0 - 10.2.2 | metadata only | metadata checks fail; the .vex vint check passes (before group-varint) |
| 9.4.0 - 9.8.0 | predecessor classes | lineage probe: the then-default Lucene94 / Lucene95HnswVectorsFormat .vem list fails the same way; the Lucene99 classes do not exist yet |
| 9.3.0 | not affected | lineage probe: the documented prefix parses exactly; last unaffected |
Every check agrees it is real and documentation-only: the finder's grammar check (with a positive control and mutants), an executing cross-model refute review (OpenAI GPT-5.6), an independent reproducer, an intended-lens review, a devil's advocate, the adjudication and a skeptical audit. The brief re-ran the repro with gradle and validated the fix.
Checks and verdicts257 words
Each check ran separately. The intended-lens cross-model review could only read; a separate run re-ran the test for it.
| Check | Verdict | What it found |
|---|---|---|
| Finder (grammar check) | reproduced (doc) | 106/106 .vemf and 106/106 .vem files diverge; writer grammar 0 failures; 14 of 14 mutants killed; JUnit 3 of 3 fail on main, 2 of 3 on 10.2.2 and 9.12.3. |
| Cross-model refute review (OpenAI GPT-5.6) | real, high (executed) | Own public-API test: 5 of 6 checks fail with assertions on and off (two of them on lines outside this claim: the [int8] line and the level-0 node count); writer-layout control passes. |
| Independent reproducer | real | COSINE float and MAXIMUM_INNER_PRODUCT byte fields: 3 of 5 fail on main and 10.5.2, 2 of 5 on 10.2.2 and 9.12.3; own group-varint decode matches 2142 of 2142 nodes. |
| Intended lens (OpenAI GPT-5.6) | real, medium (reading only) | Works-as-designed argument moderate; a separate re-run of the finder's test fails 3 of 3. |
| Devil's advocate | real (doc) | Rejects the high-level-docs, experimental and version-0 defences; own test shows COSINE reading as DOT_PRODUCT. |
| Adjudication | confirmed, high | Documentation defect, impact minor; re-ran the independent reproduction's and the finder's tests. |
| Skeptical audit | confirmed | Re-ran the finder's test; impact minor, not an overclaim. |
| Brief gradle runs | fix validated | Repro fails on main (assertions on and off) and on 10.6.0; the corrected-list test passes on stock and fixed builds; with the fix 212 upstream codecs.lucene99 tests plus the corrected-list test, 0 failures; the javadoc, format and ecjLint tasks pass. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 432 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,663 words
- Registry title
- The file-format javadocs of the default vector formats (Lucene99FlatVectorsFormat and Lucene99HnswVectorsFormat) give a wrong .vemf/.vem/.vex grammar: both metadata lists leave out the int32 vector encoding that is written right after the field number, so every later element is misread, and since 10.3 the .vex list still calls the HNSW neighbour deltas vints although they are group-varint encoded (documentation defect; the writers and readers agree)
- Area
- codecs / vectors (Lucene99FlatVectorsFormat .vemf and Lucene99HnswVectorsFormat .vem and .vex file-format javadoc vs their writers and readers)
- Priority
- Low, rank 1286: The vector-format grammars, wrong since 9.4 for the metadata and 10.3 for .vex, touch only indexes with vector fields.
- Impact class
- minor
- Impact note
- Documentation only: Lucene's writers and readers agree, so indexing, merging, CheckIndex and kNN search are unaffected in every configuration; the people misled are those who read, inspect or re-implement the vector files from these class javadocs (the default codec's package docs link Lucene99HnswVectorsFormat as the Vector values format), and a parser that follows the text takes the encoding ordinal for the similarity (every float32 field reads as DOT_PRODUCT, every byte field as EUCLIDEAN, valid values, so nothing flags the similarity itself), misreads every later metadata field, and since 10.3 decodes wrong HNSW neighbours from the first list of 4 or more neighbours on (executed); the metadata part has been wrong in every release since 9.4.0 (2022) and the .vex part since 10.3.0 (2025), and no scale or special setting is needed.
- Summary
- You decode the vector metadata (.vemf, .vem) or the HNSW graph (.vex) files of the default codec by following the Lucene99FlatVectorsFormat and Lucene99HnswVectorsFormat javadocs, which list each field as '[int32] field number, [int32] vector similarity function ordinal, [vlong] offset, ...' and each neighbour list as '[vint] count, array[vint] delta-encoded ordinals'. Both writers instead store an extra int32, the vector encoding (BYTE or FLOAT32), between the field number and the similarity, and since 10.3 they write the neighbour deltas with group-varint, so a parser built from the text reads the encoding as the similarity (a COSINE float field comes out as DOT_PRODUCT), gets the offset, length, dimension and count wrong, and decodes wrong neighbours from the first list of 4 or more neighbours on. Lucene's own readers match the writers, so nothing breaks at runtime; only the documentation is wrong, the metadata part since 9.4.0 and the .vex part since 10.3.0.
- Introduced
- two parts [follow-up]: (1) the missing encoding int32: a693fe819b04f07942bb1bcbc28169838f1becfc (main) / 87a8f7d48f3 (branch_9x cherry-pick, the commit that first shipped, in 9.4.0), LUCENE-10577 'enable quantization of HNSW vectors to 8 bits' (apache/lucene#1054), by msokolov: it added meta.writeInt(field.getVectorEncoding().ordinal()) to the then-default Lucene94HnswVectorsWriter and readVectorEncoding to the reader, and edited the .vec and .vem level lines of the Lucene94 javadoc but not the field number / similarity lines [read: git show in the full-history clone; exec: the .vem documented prefix parses on 9.3.0 and fails on 9.4.0]; the stale text was carried into the two classes named here when they were created, by f2bf5339e53 / branch_9x 6623c42553b (apache/lucene#12582, Lucene99HnswVectorsFormat, by benwtrent) and a47ba3369f4 / branch_9x 448e6112954 (apache/lucene#12729, Lucene99FlatVectorsFormat, by benwtrent), both first released in 9.9.0; (2) the .vex neighbour encoding: a6c99125599 (main) / a0de4374446 (branch_10x), apache/lucene#14932 'GroupVarInt Encoding Implementation for HNSW Graphs', by aylonsk: VERSION_CURRENT = VERSION_GROUPVARINT and writeGroupVInts for the neighbour deltas, with no change to the class javadoc's .vex list [read; exec: the .vex vint check passes on 10.2.2 and fails on 10.3.0]. The stale lines were correct when written: '[int32] field number' in 5b36af3cd79 (LUCENE-9844, 2021-03-18), the similarity line in 6d4b5eaba35 (2021-04-25) and the .vex vint lines in d0be9ab57c9 (#11860, 2022-12-07) [follow-up, blame on main] (2022-08-10 (author date of a693fe819b0 and 87a8f7d48f3: the encoding int32); 2023-10-24 and 2023-11-10 (author dates of f2bf5339e53 and a47ba3369f4: the Lucene99 HNSW and flat classes created with the stale list); 2025-08-06 (author date of a6c99125599 and a0de4374446: group-varint .vex))
- First affected version
- 9.4.0 [exec] for the missing encoding int32, through the then-default Lucene94HnswVectorsFormat .vem javadoc (the last unaffected release is 9.3.0 [exec], whose Lucene92 writer wrote no encoding int); 9.9.0 [exec] for the two classes named in the title, which exist from 9.9.0 and were wrong from creation; 10.3.0 [exec] for the .vex group-varint part (last unaffected 10.2.2 [exec])
- Affected versions
- every release from 9.4.0 through 10.6.0 (10.6.0 run as its release revision, the branch_10_6 head 4ae946e5a9b) and main a9779ef3c193 for the metadata part, and 10.3.0 through 10.6.0 and main for the .vex part; not fixed upstream [exec: follow-up runs of every listed release with seed DEADBEEF and -ea: 9.4.0-9.8.0 with a lineage probe that reads the predecessor .vem javadoc prefix (fails; 9.3.0 passes), 9.9.0-10.5.2 with the finder's test or its release variants (meta checks fail everywhere; the .vex vint check passes 9.9.0-10.2.2 and fails 10.3.0-10.5.2), branch_10_6 and main (also -da), 3 of 3 failing on 10.3.0 and later; the independent reproduction's own test fails 3 of 5 on main (-ea and -da) and 10.5.2 and 2 of 5 on 10.2.2 and 9.12.3, its controls passing; the intended-behaviour check's own test fails 3 of 6 on main (-ea and -da) with its controls passing, and its meta checks fail the same way on 10.5.2]. Before 9.9 the stale lines live in Lucene94HnswVectorsFormat (9.4) and Lucene95HnswVectorsFormat (9.5-9.8), still present in backward-codecs on main. Caveats: upstream main 645d4fe14a20 (2026-10-28) still carries the lines [read]; all runs used JDK 25 on Linux aarch64
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal
- Upstream and prior reports
- none found (searched 2026-10-28: 205 queries from two scripted prior-report searches (252 queries and checks in all) over GitHub issues, PRs and commits, the dev@, java-user@, issues@ and commits@ lists, the JIRA archive, downstream repositories and the web, 22 manual GitHub searches including the last 48 hours, 3 GitHub code searches; all 413 open apache/lucene PRs scanned by changed files and every PR touching the format files diffed). Nearest items, not the same bug: #13548 (merged 2024-07, refreshed these javadoc lists after the flat-format split and kept the omission; its review comments do not mention it), #1054 / LUCENE-10577, #12582, #12729 and #14932 (the changes that made the text stale; their threads do not mention it), #16771 (merged 2026-10-16, edited only the .vemf alignment NOTE), #10883 / LUCENE-9844 (origin of the lists), #13158 (a user comment lists the real .vemf layout without saying the javadoc is wrong), #16211. Coordination, not prior art: open PR #15430 rewrites the adjacent .vemf '[int8] dense/sparse' line as part of an unmerged format change, so that line is excluded from this claim (it keeps the claimed omission), and open PR #13525 copies the stale .vemf list into a new class. Outside ports: one copies the stale text, two document the real layout from the writer; none filed anything. Recheck (2026-10-28, GitHub search and API): none of the 29 apache/lucene issues and PRs created since 2026-10-26 (including another contributor's ongoing javadoc sweep, #16920-#16922, #16927, #16928, #16933-#16935) is about these formats' grammar; 'vemf' gives only #15630, #13158, #15341, #13844, and open PR #15630 does not change either format class; 'delta encoded neighbor ordinals' and 'vex group varint javadoc' 0 hits; by author date the latest upstream commits are 8a8ef488218 (2026-10-16, #16771) for Lucene99FlatVectorsFormat.java and 79674f38df7 (2026-09-28, #16682) for Lucene99HnswVectorsFormat.java; upstream main 645d4fe14a20 (2026-10-28) still has the stale lines 53-54 of the flat class and 54 and 67-68 of the HNSW class. Not fixed upstream.
- Lucene commit tested
- main
645d4fe14a20: the repro with gradle in a scratch worktree, assertions on and off, then the javadoc fix with every class of thecodecs.lucene99test package and therenderJavadoc,checkGoogleJavaFormatandecjLintMaintasks (2026-11-01); the 10.6.0 release4ae946e5a9b9and the finder's three-check test (with the.vexcheck) on main645d4fe14a20in a separate re-run (2026-11-01). The study ran on maina9779ef3c193(thelucene99codec files,GroupVIntUtiland the codec overview are identical to645d4fe14a20) with the finder's three-check test on every release from 9.9.0 to 10.5.2 plus the branch_10_6 head, and with a lineage probe of the predecessor formats on 9.3.0-9.8.0.
- Finding record
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsFormat.java:48-63 (.vemf list; :53 field number, :54 similarity ordinal), :74 (@lucene.experimental); Lucene99FlatVectorsWriter.java:342-344 (writeInt number, encoding ordinal, similarity ordinal), :171 (writeInt(-1) end marker); Lucene99FlatVectorsReader.java:482-483 (readVectorEncoding, then readSimilarityFunction)lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsFormat.java:43-59 (.vex list; :54 array[vint] deltas), :62-80 (.vem list; :67 field number, :68 similarity ordinal), :83 (@lucene.experimental), :92-94 (VERSION_GROUPVARINT = 1 = VERSION_CURRENT); Lucene99HnswVectorsWriter.java:591-597 (writeVInt count, then writeGroupVInts when version >= VERSION_GROUPVARINT), :615-617 (writeInt number, encoding ordinal, similarity ordinal), :633-646 (node counts only for levels > 0), :650-664 (graph offsets meta); Lucene99HnswVectorsReader.java:224-229 (similarity ordinals EUCLIDEAN, DOT_PRODUCT, COSINE, MAXIMUM_INNER_PRODUCT), :249-250 (encoding read before similarity), :610 (GroupVIntUtil.readGroupVInts)lucene/core/src/java/org/apache/lucene/codecs/lucene104/package-info.java:54 ('a high-level definition'), :183 and :313 (link Lucene99HnswVectorsFormat as Vector values), :188 ('Details on each of these are provided in their linked pages'), :411 (History: 9.5 vint delta-encoding, no group-varint entry); Lucene104ScalarQuantizedVectorsFormat.java:85 ('int the vector encoding ordinal', the sibling format that documents it)lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene94/Lucene94HnswVectorsFormat.java:74-75 and lucene95/Lucene95HnswVectorsFormat.java:74-75 (the same field number / similarity omission in the frozen read-only copies)
- The finding in 30 seconds
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsFormat.java:48-63, :74lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsFormat.java:43-81, :83, :92-94, :261lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsWriter.java:342-351; Lucene99HnswVectorsWriter.java:422-428, :591-597, :615-621lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsReader.java:482-487; Lucene99HnswVectorsReader.java:224-246, :249-250, :605-619lucene/core/src/java/org/apache/lucene/index/VectorEncoding.java:29, :32, :38; lucene/core/src/java/org/apache/lucene/util/GroupVIntUtil.java:144-173
- Who is affected
lucene/core/src/java/org/apache/lucene/codecs/Codec.java:59; lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104Codec.java:123; lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsFormat.java:124-137, :155-156lucene/core/src/java/org/apache/lucene/codecs/lucene104/package-info.java:183, :188, :313
- Why it looks unintended
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsFormat.java:53-54, :74; Lucene99HnswVectorsFormat.java:53-54, :67-68, :83, :261lucene/core/src/java/org/apache/lucene/codecs/lucene104/package-info.java:54, :188; lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsFormat.java:84-86
- Suggested fix
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java:243, :630-664; Lucene99FlatVectorsWriter.java:171lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene94/Lucene94HnswVectorsFormat.java:74-75, lucene95/Lucene95HnswVectorsFormat.java:74-75
- TestLucene99VectorsFormatsMetaJavadoc.java (one test, lucene/core tests):
repro/LFV-941/src/org/apache/lucene/codecs/lucene99/TestLucene99VectorsFormatsMetaJavadoc.java(67 lines)
Made in parallel with this doc from the mechanism in the brief and an executed byte probe; the PNG is the summary frame.
assets/LFV-941/bug.png(432 KB)