Re-sorting with SortingCodecReader and addIndexes throws when a segment lacks a quantized vector field

SortingCodecReader's javadoc describes how to re-sort an existing index: wrap all its readers and add them to a fresh IndexWriter with addIndexes(CodecReader...). With a quantized vector format (Lucene104[Hnsw]ScalarQuantizedVectorsFormat on 10.4+ and main, the Lucene102 BBQ formats on 10.2-10.3) this fails when one segment has vectors for some other field but none for a float32 or fp16 field that uses the quantized format (byte fields skip the centroid code). To recompute the merged centroid, Lucene104ScalarQuantizedVectorsWriter asks every source reader for the field and skips a source only when the answer is null; it never checks that source's FieldInfos, although KnnVectorsReader#getFloatVectorValues is undefined for such a field and "never null". The per-field reader of a plain segment happens to answer null, so ordinary merges pass, but SortingCodecReader wraps the null and fails: NullPointerException at SortingCodecReader.java:252 with assertions off, AssertionError at :318 with assertions on (fp16 fields: the NullPointerException either way). addIndexes throws and adds nothing; the source index is unchanged. It is a new instance of the bug class #13641 fixed for the other vector writers in 9.12.0. Introduced in 10.2.0 by #14078; affects 10.2.0 to 10.6.0 (10.6.0 by reading) and main.

• Centroid: the quantized formats (Lucene104ScalarQuantizedVectorsFormat, Lucene104HnswScalarQuantizedVectorsFormat; on 10.2-10.3 the Lucene102 BBQ formats) quantize each vector relative to the segment's average vector. When segments merge, the writer computes the merged segment's centroid from the source segments.
• Re-sort recipe: SortingCodecReader's javadoc: "This can be used to re-sort and index after it's been created by wrapping all readers of the index with this reader and adding it to a fresh IndexWriter via IndexWriter#addIndexes(CodecReader...)". addIndexes merges all the readers into one segment by default.
• FieldInfos: each segment's list of fields. A segment where no document has vec has no vec entry.
• The reader contract: KnnVectorsReader#getFloatVectorValues: "The behavior is undefined if the given field doesn't have KNN vectors enabled on its FieldInfo. The return value is never null." So a merge must check the source's FieldInfos first; the generic merge and the Lucene99 writers do (MergedVectorValues.hasVectorValues, added by #13641).

1. Lucene104ScalarQuantizedVectorsWriter.mergeOneFlatVectorField calls mergeAndRecalculateCentroids for a float32 or fp16 field (Lucene104ScalarQuantizedVectorsWriter.java:351-360).
2. Its first loop asks every source through floatingPointVectorValues and skips a source only when the reader or its answer is null (:597-605). A SortingCodecReader-wrapped source has no stored centroid (:569-575), so the loop stops and recalculates (:614-617, :625): calculateCentroid skips only a null reader (:642-646), accumulateCentroid only a null answer (:663-666). Neither loop looks at mergeState.fieldInfos[i]; the helper's javadoc states the assumption, it returns "null when the field is absent from this reader" (:577-591).
3. A plain segment's reader is PerFieldKnnVectorsFormat.FieldsReader, which returns null for a field not in its FieldInfos (PerFieldKnnVectorsFormat.java:280-287, fp16 :311-318). That is why ordinary flushes and merges pass.
4. SortingCodecReader's vectors reader wraps the answer unconditionally: new SortingFloatVectorValues(delegate.getFloatVectorValues(field), docMap) (SortingCodecReader.java:631; byte :636, fp16 :641). The constructor asserts delegate != null (:318), then iteratorSupplier calls values.iterator() (:252). SortingFloat16VectorValues has no assertion (:402-406), so fp16 fields get the NullPointerException even with -ea. For points the same class passes a null through (:602-607).
5. When the lacking segment comes first, the first loop fails (:602); otherwise calculateCentroid does (:646). Both were executed.

• a quantized format on a float32 or fp16 field that some segment lacks (quantized only on a field every segment has passes; a byte field skips the centroid code, Lucene104ScalarQuantizedVectorsWriter.java:356-358);
• the lacking segment goes into one addIndexes call as a SortingCodecReader, with at least one other segment that has the field. In the javadoc recipe every segment is wrapped, so that means two or more wrapped segments; strictly only the lacking one must be wrapped, and the destination needs no index sort (executed: wrapping only the lacking segment fails, wrapping only the other passes);
• the lacking segment has some other vector field (executed: byte, float32, or float32 in the non-quantized format all fail; other encodings by reading), so it has a vectors reader; a segment with no vector field at all has none and is skipped;
• the sort reorders that segment; an already sorted segment (for example a one-document segment) is wrapped in a plain FilterCodecReader (SortingCodecReader.java:458-460), which answers null.

IndexWriter's own reorder paths (one SortingCodecReader over a single SlowCompositeCodecReaderWrapper, IndexWriter.java:3561-3564, :5557-5576) and BpVectorReorderer.reorderIndexDirectory (one addIndexes per segment) are not affected by code reading; they were not executed.

Segment 1: 3 documents with vec (3 dimensions) and other (byte). Segment 2: 3 documents with only other. Sort by a long doc-values field, inserted in descending order so the sort reorders both segments.

With assertions off the writer stays open (a commit after the failure shows 0 of 6 documents); with assertions on, a float32 field's AssertionError closes it as a tragic event, so the target gets no commit, while an fp16 field's NullPointerException leaves it open. With the default ConcurrentMergeScheduler, the merge thread also reports the exception as an uncaught MergeException.

• Who. Applications that re-sort an existing index with the documented recipe (for example to add or change an index sort, or to reorder documents) and use a quantized vector format for a field that some segment lacks while that segment has another vector field. Realistic shapes: a second embedding field added later, so old segments have only the first; or an optional embedding that a whole segment happens to lack.
• Default vs opt-in. The default codec uses the non-quantized Lucene99HnswVectorsFormat, which is not affected; the quantized formats are opt-in. From 10.4.0 the older int8 formats, which have the check, are read-only legacy, so every writable quantized format in core is affected. SortingCodecReader and the Lucene104 quantized formats are @lucene.experimental.
• Scale. No size threshold: 3 documents per segment are enough, as long as the sort reorders the lacking segment. The failure is deterministic (10 of 10 runs; the compact test also fails the same 4 tests in each of 5 iterations with another seed).
• Effect. A loud failure: addIndexes throws, nothing is added, nothing is corrupted, and the source index is unchanged. With assertions off the writer stays open, and each failed attempt leaves its partly written files in the target directory until the writer is closed (measured: 25 failed attempts, up to 326 files and 281,850 bytes, removed at close). With assertions on, a float32 field's AssertionError closes the writer as a tragic event; an fp16 field's NullPointerException leaves it open in both modes.
• One way to lose data. An in-place re-sort (OpenMode.CREATE on the source directory, all segments in one call) that closes the writer after the failure with the default commitOnClose commits an empty index over the original (measured: 40 to 0 documents). That is documented close() behaviour applied to any failure in that pattern, not something Lucene commits wrongly.
• Workaround. One addIndexes call per segment (measured: the in-place per-segment re-sort keeps all 40 documents, as 2 segments rather than one merged segment). Writing into a separate directory does not avoid the exception; it only protects the source from the in-place outcome above.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError
       org.apache.lucene.index.SortingCodecReader$SortingFloatVectorValues.<init>(SortingCodecReader.java:318)
       org.apache.lucene.index.SortingCodecReader$5.getFloatVectorValues(SortingCodecReader.java:631)
       org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsWriter.floatingPointVectorValues(Lucene104ScalarQuantizedVectorsWriter.java:584)
       org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsWriter.mergeAndRecalculateCentroids(Lucene104ScalarQuantizedVectorsWriter.java:602)
       org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsWriter.mergeOneFlatVectorField(Lucene104ScalarQuantizedVectorsWriter.java:360)
       org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsWriter.mergeOneFlatVectorField(Lucene104ScalarQuantizedVectorsWriter.java:335)
       org.apache.lucene.codecs.hnsw.FlatVectorsWriter.mergeOneField(FlatVectorsWriter.java:63)
       org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:132)
       org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:141)
       [...]

• Contract. KnnVectorsReader#getFloatVectorValues: "The behavior is undefined if the given field doesn't have KNN vectors enabled on its FieldInfo. The return value is never null" (KnnVectorsReader.java:64-69; the byte and fp16 getters say the same, :82-87, :89-94). The application never calls the getter; the writer does.
• The stated rule. #13641 (74e3c44063a, 2024-09-11): "During segment merge we must verify that a given field has vectors and exists. The typical knn format checks assume the per-field format is used and thus only check for null. But we should check for field existence in the field info and verify it has dense vectors". CHANGES 9.12.0: "Unify how KnnFormats handle missing fields and correctly handle missing vector fields when merging segments". It added KnnVectorsWriter.MergedVectorValues.hasVectorValues, used by the generic merge (KnnVectorsWriter.java:350), Lucene99HnswVectorsWriter (:520) and the int8 writer.
• Documented use. SortingCodecReader's javadoc prescribes exactly this re-sort (SortingCodecReader.java:47-51); nothing requires every segment to carry every vector field.
• History. #14078 (10.2.0) wrote the BBQ centroid loops six months after #13641, with the null-only check; #15169 (10.4.0) copied them byte for byte; #16473 (main) routed fp16 through them. None of the reviews discusses absent fields or SortingCodecReader.
• Tests. TestSortingCodecReader uses a single vector field. BaseKnnVectorsFormatTestCase.testMergingWithDifferentKnnFields asserts that such merges work, but passes for these formats because their own readers answer null. No CHANGES or MIGRATE entry, documentation or discussion accepts the crash.

The intended-behaviour verifier, a review by a model from another family and the cross-model fact check of this doc built the strongest case for "works as designed": the contract makes the lookup undefined, so SortingCodecReader's assertion is allowed; SortingCodecReader and the formats are @lucene.experimental; the format is opt-in, the path uncommon and the failure loud with the source unchanged; and the writer documents its own assumption ("null when the field is absent from this reader"), which holds for the per-field reader and the raw Lucene104 reader.

Why it still is a bug: the undefined-behaviour clause binds the caller, and the caller is the writer, not the application, which follows the documented recipe. @lucene.experimental lowers severity, not validity. The helper's javadoc contradicts the abstract contract and #13641's rule; it explains how the bug arose, not that the crash is intended. Which class owns the fix is genuinely open: the writer breaks the caller precondition #13641 made a merge rule, while SortingCodecReader turns the null its delegate actually gives into an exception, unlike its own points path. Both fixes are validated. Verdict: a defect (the earlier verifiers and reviews rated the intended-behaviour argument weak; the cross-model fact check rated it moderate, mainly over which class should own the fix).

Two fixes were applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF) with the four repro classes and the closest upstream suites. They fix the bug at different sites; A follows the rule #13641 stated. Which one, or both, is the maintainers' call. Both fix only this defect: other registered bugs on the same re-sort recipe (LFV-112 with document blocks, LFV-278 and LFV-404 with a reordering merge policy) are separate.

In Lucene104ScalarQuantizedVectorsWriter.mergeAndRecalculateCentroids and calculateCentroid, skip a source when KnnVectorsWriter.MergedVectorValues.hasVectorValues(mergeState.fieldInfos[i], fieldInfo.name) is false, as #13641 did for the generic merge and the Lucene99 writers. One file, +7/-2.

import static org.apache.lucene.codecs.KnnVectorsWriter.MergedVectorValues.hasVectorValues;
...
// in mergeAndRecalculateCentroids and in calculateCentroid
KnnVectorsReader knnVectorsReader = mergeState.knnVectorsReaders[i];
if (knnVectorsReader == null
    || hasVectorValues(mergeState.fieldInfos[i], fieldInfo.name) == false) {
  continue;
}

Trade-offs

• Matches #13641 and the sibling writers; up to two FieldInfos lookups per source and field per merge (one in each centroid loop).
• It does not change what SortingCodecReader returns, so any other caller that asks it for an absent field would still fail (none known in Lucene).
• The 10.2/10.3 line needs the same two lines in Lucene102BinaryQuantizedVectorsWriter (the release shadow runs show the guard fixes 10.2.0 and 10.3.2); on main the Lucene102 formats in backward-codecs are read-only, and their test-only writer copy has the same null-only checks. Open PR #16839 (ready for review since 2026-10-26) and draft PR #16819, which rewrite this code, would need it too.
• Not done: the full core suite, check, a CHANGES entry.

What the checks validated

• Stock: the 4 repro classes run 15 tests with 9 failures, assertions on and off.
• With A: 15 of 15 pass, and so do the 9 closest suites (core TestLucene104ScalarQuantizedVectorsFormat, TestLucene104HnswScalarQuantizedVectorsFormat, TestLucene104ScalarQuantizedMergeScorer, TestQuantizedVectorValuesPrefetch, TestSortingCodecReader, TestFilterReadersForwardVectorPrefetch, TestIndexSorting, TestAddIndexes; misc TestBpVectorReorderer): 13 classes, 272 tests, 4 skipped (2 nightly testRecall, 2 assumptions that apply only to other formats), 0 failures, with assertions on and with assertions off.
• Both loops are needed: with the guard only in calculateCentroid, testHnswScalarQuantizedLackingSegmentFirst still fails, in the first loop.
• checkGoogleJavaFormat, forbiddenApisMain, forbiddenApisTest, ecjLintMain and ecjLintTest of lucene/core pass with A and the compact test.
• Earlier, on main ea57360912ed: the same guard passed the finder's test and 128 upstream tests, the reproducer's 7 tests, and the cross-model reviews' tests plus 125 Lucene104 format tests; in each release's own writer it turns 10.2.0, 10.3.2, 10.4.0 and 10.5.2 into passes.

In SortingCodecReader.getVectorReader(), return null when the delegate answers null for float32, byte and fp16 values, as getPointsReader already does. One file, +9/-3.

FloatVectorValues values = delegate.getFloatVectorValues(field);
return values == null ? null : new SortingFloatVectorValues(values, docMap); // same for byte and fp16

Trade-offs

• Fixes every caller that relies on a null answer, including, by reading, a downstream copy of the BBQ writer.
• Makes a reader return null where the KnnVectorsReader javadoc says "never null" (the per-field reader already does).
• Leaves the writer's precondition violation in place: a wrapper or reader that does not answer null for an absent field would still fail.

What the checks validated

• With B: 15 of 15 repro tests pass, and the same 13 classes run 272 tests, 4 skipped, 0 failures, with assertions on and with assertions off.
• Not format- or lint-checked.

A as the fix that follows the stated rule, B as hardening of the wrapper.

Trade-offs

Each was validated on its own; the combination was not run.

Say that the re-sort recipe does not support quantized fields that a segment lacks.

Trade-offs

Leaves the documented recipe broken; not recommended.

Method: randomized differential testing (non-formal tier), in a hunt unit on index sorting.

• The harness builds the same documents with IndexWriterConfig.setIndexSort, through SortingCodecReader.wrap plus addIndexes, and in pre-sorted order, with randomized vector formats, and compares the three per stored id.
• Its assertions-on hunt hit an AssertionError at SortingCodecReader.java:318 via Lucene104ScalarQuantizedVectorsWriter.floatingPointVectorValues.
• The finder read the failing frames, found the missing FieldInfos check, minimised it to a 3-test JUnit on the stock jar, ran it on main and released versions, and wrote a fix sketch.
• Found 2026-10-25 on main ea57360912ed; then checked by an independent reproducer, a devil's-advocate verifier, reviews by a model from another family (OpenAI GPT-5.6) that ran their own tests, an adjudication, a follow-up (history, prior reports, security), the registration check and a re-run on main a9779ef3c193.