Merges with BpVectorReorderer throw NullPointerException and close the IndexWriter when a segment has no vectors for the field

During a merge, IndexWriter reads the segments being merged through one combined view (SlowCompositeCodecReaderWrapper) and hands it to OneMerge#reorder. For a vector field the view keeps one slot per segment, and a segment in which no document has the field gets an empty (null) slot. The iterator of the merged vector values skips empty slots and vectorValue()'s ordinal lookup handles them, but copy() calls sub.copy() on each slot and throws NullPointerException: Cannot invoke "org.apache.lucene.index.KnnVectorValues.copy()" because "this.sub" is null. BpVectorReorderer calls copy() in each thread that takes part in partitioning, which with the default settings starts once the merge holds at least 64 vectors, so with BPReorderingMergePolicy and a BpVectorReorderer a merge of 100 documents with vectors and 10 without fails. IndexWriter treats the failed merge as a tragic event: the writer closes, the changes since the last commit are lost, and a reopened writer fails again at the next merge of that segment. The committed index stays clean. The end-to-end failure is on main only (11.0.0-SNAPSHOT); the copy() defect itself came with #14092 and has shipped since 10.2.0, where only custom reorder code that copies the merged view reaches it.

• Merged view: during a merge, IndexWriter wraps the segments being merged in one CodecReader, SlowCompositeCodecReaderWrapper, and passes it to MergePolicy.OneMerge#reorder, which may return a new document order. With one segment there is no view (wrap() returns the segment, :49-57).
• Slot: the merged vector values keep one DocValuesSub per segment. A segment without vectors for the field gets a slot whose values are null.
• BP reordering: BPReorderingMergePolicy (lucene/misc) renumbers the documents of each merged segment. With a BpVectorReorderer (#14097, main only) it sorts documents so that similar vectors sit together, by splitting the vectors in halves recursively on several threads.

1. getFloatVectorValues of the merged view (SlowCompositeCodecReaderWrapper.java:824-841) asks every segment for its values (:830), which is null for a segment that indexed no vectors (CodecReader.java:235-237; LeafReader.java:215-221), and adds a slot also for null (:831). Byte (:924-941) and float16 (:944-961) are the same.
2. The iterator skips empty slots (:339, :357), and the ordinal lookup handles them (findSub, :1082 onwards). But DocValuesSub.copy() (:304-309) is return new DocValuesSub<>((T) (sub.copy()), docStart, ordStart); (:307) with no null check, and the three Merged*VectorValues.copy() methods call it for every slot (:903-909, :1003-1009, :1073-1079). (Separate from this bug: the merged values also inherit the dense default ordToDoc(), which returns its argument (KnnVectorValues.java:43-50), and iterator() returns one shared iterator (:887-889); the differential test flagged both, neither is a registered bug, and neither is needed to explain this failure.)
3. BpVectorReorderer.computeDocMap reads the merged FloatVectorValues (BpVectorReorderer.java:590) and, after a RAM-budget check that the default budget passes (:598-606), each thread that takes part builds one PerThreadState, whose constructor calls vectors.copy() (:110, :620-624). The first one is built in ReorderTask.compute (:217) right after if (halfLength < minPartitionSize) return; (:208-211): with the default minPartitionSize of 32 that needs 64 vectors.
4. BPReorderingMergePolicy's reorder catches only NotEnoughRAMException (BPReorderingMergePolicy.java:133-145); forced merges are always reordered (:103-105), natural merges from minNaturalMergeNumDocs documents, default 1 (:46).
5. IndexWriter.mergeMiddle builds the view and calls reorder (IndexWriter.java:5557-5558). The exception reaches merge()'s outer catch (Throwable t) and tragicEvent(t, "merge") (:5109-5113), which calls maybeCloseOnTragicEvent() and so rollbackInternal() (:6090-6106): the writer rolls back to the last commit and closes. addIndexes(CodecReader...) builds the same view (:3561-3562) but calls tragicEvent only for Error (:3390-3392).

The control, the 63/64 rows and the copy() row come from the repro tests, with default settings. The writer-state rows come from a probe that also uses the default codec, but in all four tests a ConcurrentMergeScheduler subclass whose handleMergeException only records merge-thread exceptions, and setMaxFullFlushMergeWaitMillis(0); the writer closes in IndexWriter.merge's outer catch (IndexWriter.java:5109-5113) before that handler runs, so the outcome does not depend on it. In that probe forceMerge throws IOException or, depending on timing, IllegalStateException: this writer hit an unrecoverable error; cannot complete forceMerge, with the NPE as the cause (IndexWriter.java:2284-2300).

• Who is hit. Applications on main that opt into the expert lucene/misc combination BPReorderingMergePolicy + BpVectorReorderer (CHANGES 11.0.0, #14097: "Binary partitioning merge policy over float-valued vector field") and whose vector field is optional: documents without embeddings, several document kinds in one index, or a batch without vectors flushed on its own. Applications that put a vector in every document, as the upstream TestBPReorderingMergePolicy does, never hit it.
• Trigger. A forced or natural merge that includes a segment with no vector for the field and holds at least 64 vectors, and addIndexes(CodecReader...) of such inputs, under the reproduced conditions: default reorderer settings (minPartitionSize 32) and RAM budget (a budget too small for the merge throws NotEnoughRAMException before copy() and the policy skips reordering), no index sort, no document blocks without a parent field. Default codec and IndexWriterConfig; no special values or timing. With the javadoc's suggested minNaturalMergeNumDocs of 2^18, small natural merges are not reordered, but larger ones and every forced merge still are.
• Effect. Indexing stops: the writer closes, the changes since the last commit are lost (114 documents visible before, 110 on disk after, an uncommitted delete undone), and each reopened writer fails again at the next merge that picks the same segment while the segment count grows. Under those conditions forceMerge(1) over such segments fails every time. The last committed index stays clean (CheckIndex) and searchable; what a near-real-time reader already saw of the uncommitted changes disappears. addIndexes(CodecReader...) throws but keeps the writer open.
• Scale. None: 64 vectors and one 10-document segment without the field.
• Released versions. The copy() defect has shipped since 10.2.0, but there no in-tree code calls copy() on the merged view; an application's own OneMerge#reorder that does fails on 10.2.0 and 10.5.2 and passes on 10.1.0.
• Encodings. float32, byte and float16 merged values all throw; only float32 has an in-tree caller.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected the two segments to merge, but forceMerge threw java.lang.NullPointerException: Cannot invoke "org.apache.lucene.index.KnnVectorValues.copy()" because "this.sub" is null
       org.apache.lucene.misc.index.TestBpVectorReorderSegmentWithoutVectors.testMergeSegmentWithoutVectors(TestBpVectorReorderSegmentWithoutVectors.java:51)
     Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.lucene.index.KnnVectorValues.copy()" because "this.sub" is null
       org.apache.lucene.index.SlowCompositeCodecReaderWrapper$DocValuesSub.copy(SlowCompositeCodecReaderWrapper.java:307)
       org.apache.lucene.index.SlowCompositeCodecReaderWrapper$SlowCompositeKnnVectorsReaderWrapper$MergedFloatVectorValues.copy(SlowCompositeCodecReaderWrapper.java:906)
       org.apache.lucene.misc.index.BpVectorReorderer$PerThreadState.<init>(BpVectorReorderer.java:110)
       org.apache.lucene.misc.index.BpVectorReorderer$1.initialValue(BpVectorReorderer.java:623)
       org.apache.lucene.misc.index.BpVectorReorderer$1.initialValue(BpVectorReorderer.java:620)
       [...]

• KnnVectorValues#copy (KnnVectorValues.java:95-99): "Creates a new copy of this {@link KnnVectorValues}. This is helpful when you need to access different values at once, to avoid overwriting the underlying vector returned." No precondition.
• LeafReader#getFloatVectorValues (LeafReader.java:215-221): "... or null if no {@link FloatVectorValues} were indexed. The returned instance should only be used by a single thread." So a reorderer that reads the vectors from several threads at once needs one copy per thread (or must serialize access), and a segment without vectors is an expected state.
• OneMerge#reorder (MergePolicy.java:311-325) "will be called when index sorting is disabled on a merged view of the {@link OneMerge}": this view.
• BPReorderingMergePolicy (:34-39): "When reordering doesn't have enough RAM, it simply skips reordering in order not to fail the merge."
• Sparse vectors are supported. The merged view stores null slots on purpose (since #12622, 2023) and handles them in its iterator and ordinal lookup. BpVectorReorderer sorts "docs with no value at the end" (BpVectorReorderer.java:743), its checkField returns null for an absent field instead of throwing (:699-713), and TestBpVectorReorderer.testIndexReorderSparse checks that documents without vectors "sort at the end" (:387-428), on a single segment, so without a merged view.
• History. #14092 (@msokolov, 2024-12-31) added the per-slot copy(); its PR body says the earlier shallow copy "was a bug introduced with #13779 ... I came across test failures when working on reviving a vector-based binary-partitioning merge policy". No review comments; segments without the field are not discussed.
• Precedents. #12858 / #12859 (2023): "Avoid null PointValues when merging points in SlowCompositeCodecReaderWrapper". #14123 / #14125 (2025): "Add null checks to SortingCodecReader" for a segment without vectors in a reordering or addIndexes merge.

The strongest case for "intended": the wrapper is package-private and "primarily targeted at merging, not searching" (SlowCompositeCodecReaderWrapper.java:43-47); KnnVectorValues, getFloatVectorValues and OneMerge#reorder are @lucene.experimental; KnnVectorsReader says "The behavior is undefined if the given field doesn't have KNN vectors enabled on its FieldInfo" (KnnVectorsReader.java:64-69); the vector values of SortingCodecReader throw UnsupportedOperationException from copy() (SortingFloatVectorValues.copy(), SortingCodecReader.java:345-348); the policy's javadoc names only BPIndexReorderer; and BpVectorReorderer is new and untested with such segments.

Why it fails: "experimental" allows API changes, not an NPE on legal data built with public API and default settings. The wrapper never asks a segment's KnnVectorsReader about an absent field; it calls CodecReader's guarded getter, which returns null by contract, and stores that null itself. An UnsupportedOperationException is a deliberate signal; this NPE is an accident in the method #14092 wrote for this caller. #14097 made BpVectorReorderer pluggable into BPReorderingMergePolicy, and TestBPReorderingMergePolicy plugs it in. The missing test is a coverage gap. Every reviewer rated the intended-behaviour argument weak.

One fix was built and validated in a scratch worktree; the other options are untested or not recommended. The null guard does not fix two related, separate hunt findings in the same code: LFV-278 (the merged iterator numbers ordinals by position, so a reordered addIndexes of two or more SortingCodecReader inputs writes vectors onto the wrong documents) and LFV-404 (BpVectorReorderer fails with UnsupportedOperationException on SortingCodecReader inputs, whose vector values' copy() throws on purpose, SortingCodecReader.java:345-348).

SlowCompositeCodecReaderWrapper.java only, +4 lines. The record is immutable and an empty slot has no state, so returning it is safe; slots with vectors are copied as before, so #14092's thread-safety fix stays. One place fixes float32, byte and float16 and every consumer of the merged view's copy(): BpVectorReorderer and custom OneMerge#reorder code, whether a merge or addIndexes(CodecReader...) reaches them.

private record DocValuesSub<T extends KnnVectorValues>(T sub, int docStart, int ordStart) {
  @SuppressWarnings("unchecked")
  DocValuesSub<T> copy() throws IOException {
    if (sub == null) {
      // this reader has no vectors for the field: nothing to copy
      return this;
    }
    return new DocValuesSub<>((T) (sub.copy()), docStart, ordStart);
  }
}

Trade-offs

• No behaviour change for segments with vectors; no extra allocation.
• The same change applies to the 10.x branches, whose DocValuesSub.copy reads the same (10.x has no float16 values). Add a CHANGES entry under Bug Fixes.
• A regression test should cover a merge, not only copy(): the end-to-end test checks the vector content and the document mapping after the reordered merge.

What the checks validated

• Clean scratch worktree at a9779ef3c193, gradle, seed DEADBEEF. Stock: TestSlowCompositeVectorValuesCopy 4 of 5 fail and TestBpVectorReorderSegmentWithoutVectors 1 of 2 fail. With the fix: both pass, 7 of 7, with assertions on and off.
• Closest suites with the fix: TestAddIndexes (39 tests), TestGetVectorCountCodecWrappers (1), TestSortingCodecReader (3) and the whole lucene/misc module (24 classes, 317 tests, 9 skipped, among them TestBPReorderingMergePolicy, TestBpVectorReorderer and TestBPIndexReorderer): 0 failures. TestBPReorderingMergePolicy with -Ptests.nightly=true: 3 of 3 pass, none skipped.
• With the fix and both tests in place, these pass: checkGoogleJavaFormat (core, misc), ecjLintMain and ecjLintTest (core), ecjLintTest (misc), forbiddenApisMain and forbiddenApisTest (core) and forbiddenApisTest (misc). The worktree was removed afterwards.
• Shadowed onto the stock jars: the repro passes 7 of 7 with assertions on and off; the writer-state probe passed 4 of 4.

subsCopy.add(sub.sub() == null ? sub : sub.copy()) in MergedFloatVectorValues, MergedFloat16VectorValues and MergedByteVectorValues.copy().

Trade-offs

Same behaviour; three places to keep in sync. Untested.

Skip segments whose values are null when the merged values are built.

Trade-offs

Removes the null state. The remaining slots keep their own docStart and ordStart, so the iterator and findSub look compatible, but that needs validation. The slot list would be empty only if no segment has the field, which the field-info check in CodecReader's getter normally prevents; a direct call on the vectors reader would then make MergedDocIterator's constructor call it.next() on an empty list (:322), so it needs a guard. Touches three getters for the same effect as the null guard. Untested.

Go back to the shallow copy of the slot list.

Trade-offs

Brings back the shared, thread-unsafe copy that #14092 fixed.

Avoid the copy, or catch the exception and skip reordering.

Trade-offs

Leaves the merged view's copy() broken for custom OneMerge#reorder code.

Method: randomized differential testing of the merged view against its source segments (no model checker, solver or prover found it).

• The test builds random multi-segment inputs (a target of 2-5 segments, which it can overshoot) in which some fields are missing from some segments, wraps them in the merged view and compares a broad set of merged-view APIs, the vector values' copy() included, with the source segments. Over 60 random iterations without empty segments it reported 592 mismatches of 5 kinds, among them 102 of the kind "vector-copy-exception: NullPointerException"; the others (getOffHeapByteSize exceptions, the inherited ordToDoc() and the shared iterator()) are not this bug.
• The failing frame (DocValuesSub.copy) led to the one in-tree caller that copies the merged view, BpVectorReorderer, and an end-to-end JUnit through BPReorderingMergePolicy reproduced it on the stock jars.
• A Z3 model of the same unit proved the merged view's ordinal lookup (findSub) correct with empty slots; it gave no signal for copy(), so this is not counted as a formal find.