BufferingKnnVectorsWriter's flush-time vector values map each ordinal to itself, the wrong document for sparse or index-sorted fields

At flush, BufferingKnnVectorsWriter hands its subclass's writeField a values object over the buffered vectors of each field. Its iterator visits only the documents that have a vector, so ordinal k is the k-th such document, but ordToDoc(ord) returns ord. For a field held by documents 0, 2 and 3 it answers 0, 1, 2 instead of 0, 2, 3; when an index sort reorders the segment, the sorting wrapper inherits the same identity, so even a dense field's ordinals map to themselves instead of to their sorted documents; and getAcceptOrds, built on ordToDoc, filters the wrong vectors. Nothing throws, iteration stays correct, and FLOAT32, BYTE and FLOAT16 behave the same with assertions on and off. It is latent: SimpleText's writer and the backward-codecs test writers only iterate, so stock indexes are correct, and only a custom codec on this experimental base class that maps ordinals back to documents is hit. It came with the random-access API in 10.0.0 (#13779) and is in every 10.x release and main; a lazily built ordinal-to-document table fixes it.

• Flush-time values. flush(maxDoc, sortMap) wraps each field's buffered vectors and its docsWithField set in BufferedFloatVectorValues (BufferedByteVectorValues, BufferedFloat16VectorValues) and, when an index sort reorders the segment (sortMap != null), in a private Sorting*VectorValues, then calls the subclass's writeField (BufferingKnnVectorsWriter.java:90-134 at 645d4fe14a20).
• Contract. KnnVectorValues#ordToDoc: "Return the docid of the document indexed with the given vector ordinal. This default implementation returns the argument and is appropriate for dense values implementations where every doc has a single value." (KnnVectorValues.java:43-50).

1. addValue appends a vector only for documents that have one (BufferingKnnVectorsWriter.java:310-322), so the list index, the ordinal, counts only those documents.
2. The values' iterator is fromDISI(docsWithField.iterator()) (:353): index() counts nextDoc() calls, docID() is the real document. That part is right.
3. BufferedFloatVectorValues.ordToDoc(int ord) { return ord; } (:366-369), the same in BufferedFloat16VectorValues (:412-415); BufferedByteVectorValues (:433-472) inherits the identity default. For a sparse field, ordinal and document differ.
4. The sorting wrappers (:137-250) override no ordToDoc. Their iterator's index() is the delegate's ordinal for each sorted document (SortingCodecReader.java:244-266, :286-290), so the inherited identity maps it to itself, not to its sorted document, even for a dense field, whenever the sort reorders the segment (an already-sorted segment gets no sort map: IndexingChain.java:296).
5. The default getAcceptOrds returns acceptDocs.get(ordToDoc(index)) (KnnVectorValues.java:110-127) and inherits the error.
6. For contrast, the merge-time values that mergeOneField passes to the same writeField (:262-281) throw UnsupportedOperationException from ordToDoc (KnnVectorsWriter.java:498-500, :596-597, :698-699).

FLOAT32, BYTE and FLOAT16 give the same values. Answers stay inside [0, size): they are another vector's ordinal, never out of range. Only the first row is the minimal repro; the others come from the study's independent test, re-run for this doc.

• Who is hit. Authors of a custom KnnVectorsFormat whose writer extends the public abstract, @lucene.experimental BufferingKnnVectorsWriter (class javadoc: "Buffers up pending vector value(s) per doc, then flushes when segment flushes. Used for SimpleTextKnnVectorsWriter and for vectors writers before v 9.3 .") and whose writeField calls ordToDoc or getAcceptOrds at flush, for example to write an ordinal-to-document table or to keep only accepted documents' vectors. They silently get another vector's document for every field that some documents lack, and for every field when an index sort reorders the segment.
• Not hit: Lucene's own writers. The only subclasses in the tree are SimpleTextKnnVectorsWriter and the backward-codecs test writers Lucene90/91/92HnswVectorsWriter; all take document ids from the iterator (the Lucene92 test writer builds its own ordinal-to-document table from the iterated documents). The production vector formats do not extend this class. None of them calls ordToDoc or getAcceptOrds on these values (code read); consistent with that, with ordToDoc made to throw, the SimpleText and the three backward-codecs HNSW format suites still pass.
• Opt-in. It takes a third-party codec on an experimental base class; a read-only code search of Elasticsearch, OpenSearch, OpenSearch k-NN and Solr found no subclass (not exhaustive).
• Partly self-limiting. If such a writer's writeField also calls ordToDoc on the merge-time values, it hits UnsupportedOperationException at its first merge; avoiding that takes overriding mergeOneField or treating the merge-time values differently, which makes an undetected production user unlikely.
• Scale. Any segment size; 4 documents are enough; deterministic; FLOAT32, BYTE and FLOAT16; assertions on and off.

Public and protected API only: an anonymous BufferingKnnVectorsWriter subclass (public abstract, @lucene.experimental; writeField is protected) inside an anonymous KnnVectorsFormat, plugged into a FilterCodec over TestUtil.getDefaultCodec(); IndexWriter, KnnFloatVectorField, ByteBuffersDirectory, LuceneTestCase. The format is write-only (its reader throws) and the index is never opened. The IndexWriterConfig and the codec are not randomized (the test framework still randomizes settings such as locale and time zone, which this path does not use); two seeds gave the same failure. The package org.apache.lucene.codecs is only a convention. It passes once each document maps to itself, which the recommended fix gives; it can serve as the regression test.

It covers the sparse FLOAT32 case only. The index-sort, getAcceptOrds, BYTE and FLOAT16 symptoms come from the same identity and were shown by the study's independent test; they are not part of the repro.

On a 10.x release, delete the Float16VectorValues import and the empty writeField(FieldInfo, Float16VectorValues, int) override (FLOAT16 came with #16383, which is on main and branch_11x but in no release); nothing else changes.

Lint: ecjLintTest and forbiddenApisTest pass with the test in lucene/core/src/test; checkGoogleJavaFormat flags its last comment line (104 columns).

# gradle, scratch worktree of main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (the same with asserts=false)
TestBufferingKnnVectorsWriterOrdToDoc > testSparseFieldOrdToDoc FAILED
    java.lang.AssertionError: expected:<[0->0, 2->2, 3->3]> but was:<[0->0, 2->1, 3->2]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:E6A0A48A5C4E1EA8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.codecs.TestBufferingKnnVectorsWriterOrdToDoc.testSparseFieldOrdToDoc(TestBufferingKnnVectorsWriterOrdToDoc.java:103)

# standalone, same seed: the same message on main (-ea, -da) and on 10.0.0 to 10.6.0

# the study's independent test on main (not the repro): the other shapes, summarized from its JUnit failure messages
sparse getAcceptOrds(only doc):              [0->[0], 2->[2], 3->[]]          expected [0->[0], 2->[1], 3->[2]]
dense, reversing index sort, ordToDoc:       [0->3, 1->2, 2->1, 3->0]         expected [0->0, 1->1, 2->2, 3->3]
FLOAT32, BYTE and FLOAT16 give the same values

• The contract is unconditional. KnnVectorValues.java:43-47: "Return the docid of the document indexed with the given vector ordinal. This default implementation returns the argument and is appropriate for dense values implementations where every doc has a single value." The buffered values iterate a sparse docsWithField (BufferingKnnVectorsWriter.java:353) and still override ordToDoc with the identity.
• Nothing documents a restriction. The writeField hooks say only "Write the provided float vector field" (:283-293). No javadoc, MIGRATE or CHANGES entry says flush-time values are forward-only; MIGRATE's 10.0 note is headed "Vector values APIs switched to primarily random-access" (lucene/MIGRATE.md:1725-1727).
• The convention is to throw. #13779's description: "Some of the methods on KnnVectorValues have default impls that throw UnsupportedOperationException enabling subclasses to provide partial implementations and relying on testing to catch missing required methods." The merge-time values follow it; the flush-time values answer with a plausible wrong document.
• History. The identity override came in an intermediate #13779 commit (2bbf8f1feda6, "adding some ordToDoc", 2024-09-02) while the base ordToDoc still threw; a later commit (dce766c9751d) made the base default return ord. No review comment discusses the overrides. #16383 copied the identity to FLOAT16 (on main and branch_11x, in no release yet).
• Precedent. #13842 ("Fix simple text byte vector iteration", @benwtrent, merged 2024-10-01, milestone 10.0.0; its first attempt, the closed PR #13839: "SimpleText format wasn't overriding the byte vector's ordToDoc, causing multiple tests to fail") added ordToDoc(int ord) { return entry.ordToDoc[ord]; } to SimpleTextKnnVectorsReader. The writer was not touched.
• No test covers it. No test asserts ordToDoc or getAcceptOrds on writer-side values; with them made to throw, the related suites still pass.

The strongest case (rated moderate by every reviewer): the values given to writeField are forward-only in practice. The merge-time values for the same hook throw from ordToDoc, copy and scorer and enforce forward iteration ("only supports forward iteration with a single iterator", KnnVectorsWriter.java:476-485); the merge-time class javadoc says "View over multiple vector values supporting iterator-style access via DocIdMerger" (KnnVectorsWriter.java:311); every in-tree subclass only iterates; the class is experimental and documented as for SimpleText and pre-9.3 writers; and #13779's description accepts partial implementations ("Some of these readers would only ever be used iteratively." and "Can we live with a little less API purity and gain some simplicity and less boilerplate?").

Why it fails: the same description says partial implementations should throw so that tests catch them; these values instead answer with another vector's ordinal. Nothing tells a codec author that the hook's values are forward-only, ordToDoc's contract is unconditional, and the identity default is documented as right only for dense values. During review of #13779 its author wrote (2024-09-19): "I also added a test-and-throw ensuring that the vectorValues impls that require forward-iteration enforce it"; these values enforce nothing. The maintainers fixed the same kind of defect, an inherited identity ordToDoc on sparse values, in a reader as a bug (#13842); it is analogous, not identical, since that one was visible to stock code and broke tests. What survives is the severity: latent, custom codecs only. Verdict: most likely not intended, a minor latent defect.

Two fixes were built against 645d4fe14a20 and run with gradle.

Give Buffered{Float,Float16,Byte}VectorValues an ordToDoc that looks the document up in an int[] of the documents of docsWithField, in order, built on the first call; give the three Sorting*VectorValues the same, filled from the sorting iterator (docIds[it.index()] = doc). getAcceptOrds becomes correct with no change. One file, BufferingKnnVectorsWriter.java, +75/-2 lines.

// Buffered{Float,Float16,Byte}VectorValues; the Sorting* wrappers use
// sortedDocIdsOf(iteratorSupplier.get(), size()) instead
private int[] docIds; // ord -> doc, built on first use

@Override
public int ordToDoc(int ord) {
  if (docIds == null) {
    docIds = docIdsOf(docsWithField, size()); // the documents of docsWithField, in order
  }
  return docIds[ord];
}

Trade-offs

• No table and no extra pass for writers that never call ordToDoc (SimpleText and every in-tree writer), only an unused field per values instance; a caller pays one O(size) pass and 4 bytes per vector, once per values instance.
• Not thread-safe, like the rest of these values; copy() starts without a table.
• The helpers catch IOException only because nextDoc() declares it; neither iterator does I/O.
• Flush and merge still differ (the merge-time values throw); a javadoc line on writeField can say so.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (--offline, seed DEADBEEF, JDK 25.0.3). Stock: the repro fails with assertions on and off. With the fix: it passes with assertions on and off.
• TestSimpleTextKnnVectorsFormat (57 tests, 4 skipped) and TestLucene90HnswVectorsFormat, TestLucene91HnswVectorsFormat, TestLucene92HnswVectorsFormat (174 tests, 9 skipped) pass, 0 failures.
• Core lint: ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest pass; checkGoogleJavaFormat finds nothing in the fixed file (it flags only the test's long comment line); eclint is not configured here and was skipped.
• With the fixed class compiled ahead of the stock main jars (shadowed, not a rebuilt Lucene): the study's independent test passes 6 of 6 (sparse, sparse getAcceptOrds, reversing sort; FLOAT32, BYTE, FLOAT16; -ea and -da), and the hunt's 2 tests (sparse; sparse with an index sort) pass.

Make ordToDoc throw UnsupportedOperationException in the three Buffered* and three Sorting* classes, and document on the writeField hooks that the values support forward iteration only, at flush as at merge (one file, +43/-5).

Trade-offs

• Cheapest: ordToDoc, and with it the default getAcceptOrds, then fails at the first flush, as at merge, instead of writing wrong data. It does not make the buffered values forward-only in other respects: at flush vectorValue(ord) in any order and copy() still work, while the merge-time values reject both.
• Removes an answer that is correct today for dense, unsorted fields; a custom writer relying on it breaks (it would already break at merge if it calls ordToDoc on the merge-time values too).
• The minimal test then fails with UnsupportedOperationException, so as a regression test it would have to expect the exception.

What the checks validated

• Gradle: the repro fails with UnsupportedOperationException from BufferedFloatVectorValues.ordToDoc; TestSimpleTextKnnVectorsFormat and the three backward-codecs HNSW suites pass (57 + 174 tests, 0 failures), so nothing in the tree needs ordToDoc on these values.

Say in the writeField javadoc that ordToDoc and getAcceptOrds of the values are valid only for dense, unsorted fields.

Trade-offs

Leaves the silent wrong answer in place. Not run.

Method: bounded-exhaustive translation validation (the hunt's "semi-formal" tier), hunt unit (vector document/ordinal mapping), 2026-10-26, main ea57360912ed.

• The harness ran on the real classes of the main jars and checked them against the javadoc. Write side: every set of documents with a vector in segments of 1 to 8 documents and every call sequence of length up to 4 over the values BufferingKnnVectorsWriter hands to writeField, with and without an index sort (nextDoc only under the sort), for FLOAT32, BYTE and FLOAT16.
• The first signal was its check ordToDoc(index()) == docID() on the smallest sparse case: document set {1} after one nextDoc(), ordToDoc(0) = 0 while docID() = 1.
• The finder confirmed it with a hand-written JUnit through IndexWriter (sparse; sparse with an index sort), and the hunt later minimized it to the one sparse test. The harness also showed that this fix removes only its own failures, and that with the fix for fromDISI.advance() (the root cause of another finding) as well, the write side is clean.