KnnVectorValues.fromDISI's advance() leaves index() stale, so on Lucene 10 sparse byte vectors in segments written by Lucene 9.4 read back as zeros or as another document's vector

Lucene 10 still opens indexes created by Lucene 9 and reads their old segments through backward-codecs. For a byte vector field that only some documents have, in a segment written by Lucene 9.4.0, 9.4.1 or 9.4.2, the reader's iterator() returns the DocIndexIterator built by the protected helper KnnVectorValues.fromDISI. That iterator moves its ordinal only in nextDoc(); its advance() moves to the target document but leaves the ordinal where it was. So after advance(doc), index() is -1 or an earlier document's ordinal, and vectorValue(iterator.index()), the usual way to read one document's vector and exactly what the queries module's ByteKnnVectorFieldSource does, returns all zeros or another document's vector, silently. Lucene's kNN queries, the 9.4 float field and segments written by 9.5 or later are not affected, and a merge rewrites the segment correctly. Lucene 9.x reads the same index correctly: a 10.0.0 regression from #13779.

• Ordinal: vectors of a field are stored densely, one slot per document that has a vector. A document's ordinal is its slot: with vectors on docs 0, 2 and 4, doc 2 is ordinal 1. vectorValue(ord) reads a slot.
• DocIndexIterator: since 10.0 the way to walk a vector field (values.iterator()). It is a DocIdSetIterator plus index(), "the value index (aka "ordinal" or "ord") corresponding to the current doc" (KnnVectorValues.java:140).
• Sparse field: not every document has a vector, so ordinal and doc id differ.
• KnnVectorValues.fromDISI: a protected static helper that wraps a DocIdSetIterator of the documents with a value into a DocIndexIterator (KnnVectorValues.java:187-225).

1. fromDISI's iterator keeps its own int ord = -1 (:194); index() returns it (:202-204). nextDoc() does ord++ (:211), but advance(target) is only return docsWithField.advance(target); (:216-218). After a forward advance() onto a document, index() is -1 or the ordinal of an earlier document.
2. On main the only reader that returns this iterator is the Lucene 9.4 sparse byte reader: return fromDISI(disi); (backward-codecs lucene94/OffHeapByteVectorValues.java:194-197). Its float twin returns IndexedDISI.asDocIndexIterator(disi) (lucene94/OffHeapFloatVectorValues.java:188-191), whose index() is disi.index() and stays right after advance().
3. vectorValue(ord) reads only when ord != lastOrd, and lastOrd starts at -1 (OffHeapByteVectorValues.java:39, :71-77). So on fresh values vectorValue(-1) returns the allocated but never-filled buffer: all zeros. A stale ordinal >= 0 returns that earlier document's vector. After a by-ordinal read, vectorValue(-1) seeks one vector before the field's data: another field's bytes on NIOFSDirectory, an IllegalArgumentException on MMapDirectory (executed).
4. VectorFieldFunction.exists(doc) positions only with advance(doc) (VectorFieldFunction.java:44-58), and ByteKnnVectorFieldSource reads vectorValue(iterator.index()) (ByteKnnVectorFieldSource.java:70-76). So every document with a vector reads as zeros.

6 documents, vectors [doc, doc + 1] on docs 0, 2, 4. advance(2): ordinal -1, [0, 0] (expected 1, [2, 3]). nextDoc() then advance(4): ordinal 0, [0, 1] (expected 2, [4, 5]). ByteKnnVectorFieldSource on the segment written by the 9.4.2 release: [0, 0] for docs 0, 2 and 4. fromDISI over docs {1, 3, 5, 7}: advance(5) leaves index() -1 (expected 2).

• Who is exposed. Applications on 10.0.0 to 10.5.2 or main whose index still holds a segment written by Lucene 9.4.0, 9.4.1 or 9.4.2 with a sparse byte vector field, not merged since, read through advance() then index(): the queries module's ByteKnnVectorFieldSource and ByteVectorSimilarityFunction function queries (every document that has a vector reads as the zero vector, so those documents' similarity scores are wrong and all equal, while documents without a vector get null and score 0 as before; COSINE gives NaN, which FunctionQuery turns into 0), application code that reads one document's vector this way (a committer's java-user@ answer of 2025-02 recommends iterator.advance(doc) then values.vectorValue(iterator.index()), shown for float vectors; the same pattern on these byte vectors hits the bug), and any KnnVectorValues subclass built on the protected fromDISI. Code that also calls nextDoc() gets another document's vector. The fixtures were written by 9.4.2; 9.4.0 and 9.4.1 write the same Lucene94 format and were not run.
• Default vs opt-in. DirectoryReader.open reads 9.x-created indexes by default on 10.x and main (Version.MIN_SUPPORTED_MAJOR = 9, Version.java:308); the function-query path is the application's choice.
• Scale. Any size, deterministic; 6 documents are enough. The advance()-then-index() reads throw no exception, with -ea and -da. Only the rarer sequence with a by-ordinal read first can throw: IllegalArgumentException on MMapDirectory, another field's bytes on NIOFSDirectory (executed).
• Not hit. Lucene's exact, approximate and filtered kNN queries and DoubleValuesSource.similarityToQueryVector (executed: they read through scorer(), which uses IndexedDISI.index()); ByteVectorSimilarityValuesSource, ByteVectorSimilarityQuery and the join module's diversifying byte kNN query (code reading); the 9.4 float field (executed); dense 9.4 byte fields and segments written by 9.5 or later (code reading). Filter membership and deletions stay correct, and a merge rewrites the segment correctly, CheckIndex clean (executed).
• How common. Rare: the Lucene94 vector format was current only from 9.4.0 (2022-09-30) to 9.5.0 (2023-01-30), byte vectors first appeared in 9.4.0, and every merge removes such segments. Downstream, by code reading only: Elasticsearch 8.6.0 used Lucene 9.4.2, and its byte-vector script doc values call advance() then index(); whether its current releases still open such indices was not checked.
• On unreleased main, MIN_SUPPORTED_MAJOR was 10 from #15012 (2025-09-18) to #16741 (2026-10-15), so main refused 9.x-created indexes by default in that window; it is 9 again at a9779ef3c193.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[[1, 2]]> but was:<[[0, 0]]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.backward_codecs.lucene94.TestLucene94SparseByteVectorAdvance.testAdvanceThenIndex(TestLucene94SparseByteVectorAdvance.java:64)

• The contracts. KnnVectorValues.java:140: index() returns "the value index (aka "ordinal" or "ord") corresponding to the current doc". DocIdSetIterator.java:144-153: when target > current, advance() "behaves as if written" as a nextDoc() loop, which would move fromDISI's ordinal. The fromDISI javadoc (KnnVectorValues.java:187-190) has no nextDoc-only restriction, and MIGRATE.md makes DocIndexIterator the iteration API, "a DISI that also provides an index() method" (MIGRATE.md:1727).
• The code around it. createSparseIterator() in the same class implements advance() with slowAdvance(target) (KnnVectorValues.java:262-264). Iterators that cannot advance throw UnsupportedOperationException (KnnVectorsWriter.java:464-466, :562-564, :676-678).
• History. #13779 (6053e1e3137) gave the 9.4 float reader IndexedDISI.asDocIndexIterator and the 9.4 byte reader fromDISI in one commit; asDocIndexIterator even carries // can we replace with fromDISI? (IndexedDISI.java:429). Before it, the 9.4 sparse byte reader read vectorValue(disi.index()), correct after advance(). Inside the PR, commit d43785d replaced the inherited slowAdvance(target) default, which kept the ordinal right, with the forwarding advance() after a review comment about performance; the author wrote "I added implementations for advance()". Nobody discussed index() after advance(), and #13778 says the refactor "shouldn't introduce functional changes".
• Precedent. #14992 / #15553: KnnVectorsWriter.MergedByteVectorValues counted its ordinal in vectorValue() instead of nextDoc(), unlike its float twin, so stepping its iterator without reading every vector broke the next read (its advance() throws). A reviewer called the byte/float difference "an oversight"; fixed in 10.4.0. The same kind of mistake (ordinal bookkeeping that does not follow the iterator), different class and fix.
• Why tests missed it. BaseKnnVectorsFormatTestCase#testAdvance indexes float vectors only and checks docID(), not index() (BaseKnnVectorsFormatTestCase.java:1997-2042); no test calls fromDISI.
• Legal trigger. The fixtures use the public 9.4.2 KnnVectorField(BytesRef, BYTE) API and are opened with DirectoryReader.open; VectorFieldFunction calls advance(doc) only for doc > docID().

The strongest case for "works as designed": the vector-values API is @lucene.experimental (KnnVectorValues.java:29), #13779 explicitly allowed partial implementations, fromDISI is a protected helper whose core callers only call nextDoc(), and keeping the ordinal after a long skip costs a linear walk.

Why it fails: experimental means the API may change, not that it may return a wrong ordinal; Lucene signals partial support by throwing, while fromDISI implements advance() and answers wrongly; its first version in the PR inherited slowAdvance and was correct; the same commit used the ordinal-correct adapter for the float twin; Lucene's own queries module calls advance() then index() on this path; and 9.x read the same index correctly. The cross-model lenses, the independent reproducer, the devil's advocate and the adjudication rated the argument weak. Verdict: not intended, a defect.

Fixes A, B and A + B were applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF); option C and the hardening were not run.

A: lucene94/OffHeapByteVectorValues.java SparseOffHeapVectorValues.iterator() returns IndexedDISI.asDocIndexIterator(disi), the adapter its float twin uses (OffHeapFloatVectorValues.java:188-191; IndexedDISI is already imported). B: KnnVectorValues.fromDISI's advance(target) returns slowAdvance(target), like createSparseIterator() in the same class (KnnVectorValues.java:262-264).

// lucene94/OffHeapByteVectorValues.java, SparseOffHeapVectorValues
public DocIndexIterator iterator() {
  return IndexedDISI.asDocIndexIterator(disi); // was: fromDISI(disi)
}

// KnnVectorValues.fromDISI
public int advance(int target) throws IOException {
  // step with nextDoc() so that ord stays the ordinal of the current doc
  return slowAdvance(target); // was: docsWithField.advance(target)
}

Trade-offs

A keeps IndexedDISI's fast advance() on the only reachable reader. B makes the protected helper honour the advance() contract for any KnnVectorValues subclass; it does not touch nextDoc(), so nextDoc-only callers see no change. After A its remaining callers on main are write-side (BufferingKnnVectorsWriter) and the sandbox Faiss writer. No benchmark was run.

What the checks validated

• A + B: the repro passes 3 of 3 with -Ptests.asserts=true and =false.
• Closest suites with A + B: 8 test classes, 1,119 tests (18 skipped), 0 failures: TestLucene90HnswVectorsFormat, TestLucene91HnswVectorsFormat, TestLucene92HnswVectorsFormat, TestLucene94HnswVectorsFormat and TestBasicBackwardsCompatibility in backward-codecs, TestSimpleTextKnnVectorsFormat in codecs and TestKnnVectorSimilarityFunctions in queries, plus the repro.
• checkGoogleJavaFormat (core, backward-codecs) and ecjLint (backward-codecs) pass; fix.diff and fix-with-test.diff apply cleanly to a clean a9779ef3c193 tree.
• A + B shadowed onto the stock main jars: the finder's 9.4.2-release JUnit passes 3 of 3 and the independent reproducer's passes 9 of 9.

Closes every reachable path on main and keeps fast skipping.

Trade-offs

Leaves fromDISI itself wrong for third-party subclasses.

What the checks validated

The two reader tests pass; testFromDISIAdvanceKeepsOrdinal still fails (gradle, -Ptests.asserts=true).

One line in core; fixes the helper and therefore the 9.4 reader.

Trade-offs

The 9.4 sparse byte iterator's advance() becomes linear in the documents with a vector that it skips. For ByteKnnVectorFieldSource, which visits documents in order, that is about the same work; for sparse random targets it is slower.

What the checks validated

All 3 repro tests pass (gradle); in the hunt it also passed the reviewer's 5 tests, TestKnnVectorSimilarityFunctions (11 of 11) and the 10-test security probe.

Like the merged-values iterators in KnnVectorsWriter.

Trade-offs

Fails loudly instead of silently, but breaks any third-party caller that advances today. Not run.

Stops the read one vector before the field's data after a by-ordinal read.

Trade-offs

Only if it costs nothing on hot paths. Not run.

Method: bytecode delegation-completeness scan (semi-formal tier: a tool's structural finding, then witnessed by execution), in a hunt unit on filter readers and delegating wrappers, 2026-10-26, main ea57360912ed.

• The scanner interprets every method body of the stock main jars (6,946 classes, 1,005 wrapper classes) and flags a class in which one method of a group that must agree is customised while another is passed through unchanged. For KnnVectorValues$3, the iterator fromDISI returns, nextDoc() is customised (it moves the ordinal) and advance() is a pure forward.
• A grep for fromDISI callers found the one read-side user, the Lucene 9.4 sparse byte reader, and a JUnit on an index written by the Lucene 9.4.2 release jars confirmed the zero and wrong vectors.
• The scanner checks a structural rule and proves nothing about vector values, so this is not a find by a model checker, SMT solver or prover. The unit's differential wrapper harness and random interleaving tests missed it, because their indexes are written by main, whose readers never return fromDISI.