addIndexes of a SortingCodecReader fails when the writer reorders merges with BpVectorReorderer

IndexWriter#addIndexes(CodecReader...) of a SortingCodecReader into a writer whose merge policy is BPReorderingMergePolicy with a BpVectorReorderer, and that has no index sort, throws UnsupportedOperationException and adds nothing. The policy reorders addIndexes as a forced merge; BpVectorReorderer copies the input's float vector values for its per-thread state on the first split (from 64 FLOAT32 vectors with default settings); SortingCodecReader's vector views throw from copy(). The reader from the documented BPIndexReorderer.reorder -> addIndexes recipe hits it when added to such a writer, while the postings BPIndexReorderer accepts the same input under the same policy. The failure is clean (writer open, not tragic, index unchanged). Making copy() work is not enough: BpVectorReorderer.valueMapToDocMap then reads the view's ordinals as doc order, so the reorder is applied to the wrong documents. Main and branch_11x only, since #14097 (2025-01-07); no release is affected.

1. BPReorderingMergePolicy.findMerges(CodecReader...): "addIndexes is considered a forced merge" (BPReorderingMergePolicy.java:199-203), with no minimum size (:103-105).
2. IndexWriter.addIndexesReaderMerge skips reordering only if the destination has an index sort (or blocks without a parent field), and otherwise calls merge.reorder(SlowCompositeCodecReaderWrapper.wrap(readers), ...) (IndexWriter.java:3548-3562). With one input, wrap returns the input itself (SlowCompositeCodecReaderWrapper.java:49-58).
3. BpVectorReorderer.computeDocMap takes reader.getFloatVectorValues(field) (BpVectorReorderer.java:590) and bisects the ordinal range. A range is split only if each half holds at least minPartitionSize vectors (:208-211, default 32), so with the default from 64 vectors on (a missing or non-FLOAT32 field returns null first, :586-589, :699-724). The first split creates the thread's state (:217, :619-624), and its constructor calls vectors.copy() (:108-111).
4. SortingCodecReader's SortingFloatVectorValues.copy() is throw new UnsupportedOperationException() (SortingCodecReader.java:346-347; byte :393-394 and float16 :436-437 likewise, latent because BpVectorReorderer handles FLOAT32 only). With several inputs the merged view's copy() copies each input (SlowCompositeCodecReaderWrapper.java:306-307, :903-909) and reaches the same throw.
5. The policy catches only NotEnoughRAMException (BPReorderingMergePolicy.java:136-141); addIndexes rethrows the merge's exception (IndexWriter.java:3380-3382), and only an Error would be tragic (:3390-3392). The reorder (:3562) runs before the SegmentMerger is created (:3568), so nothing is written.

BpVectorReorderer.valueMapToDocMap (BpVectorReorderer.java:735-789) returns the ordinal permutation itself as the doc map when every document has a vector (:737-739), and otherwise numbers ordinals by position in doc order (:744-764). A SortingCodecReader view keeps the wrapped reader's ordinals: its iterator's index() "may skip around, not increasing monotonically as iteration proceeds" (SortingCodecReader.java:264-266), and vectorValue(ord) reads in the delegate's ordinal space (:323-327). So once copy() works, the permutation computed over the vectors is applied to other documents.

• 64 documents, 2-dimensional vectors {i, 1}, one reversing SortingCodecReader: addIndexes throws, 0 documents added, with assertions on and off, 20 of 20 random seeds. With 63 documents nothing is split and nothing throws.
• The same failure for the reader BPIndexReorderer.reorder(leaf, dir, null) returns, for SortingCodecReader.wrap(leaf, Sort) into a writer without an index sort, with the default ConcurrentMergeScheduler, and for two 32-document sorting readers in one call.
• With only copy() fixed, a 512-vector index wrapped by a reversing view gets a doc map equal to the plain reader's at 512 of 512 positions and to the correctly re-based map at 0 of 512; cluster changes in doc order after BP: 383 (plain reader with BP: 35; no BP: 404).

• Who is exposed. Applications that combine BPReorderingMergePolicy(..., new BpVectorReorderer(field)) (lucene/misc, opt-in), a destination without an index sort (and not the document-blocks case IndexWriter also skips, IndexWriter.java:3550-3556), and addIndexes with a SortingCodecReader input: SortingCodecReader.wrap(reader, Sort), wrap(reader, docMap, null), or the output of BPIndexReorderer.reorder (its documented recipe). For example, an offline postings-BP pass whose output is added to a writer that also reorders by vectors, or a re-sort or rewrite job feeding such a writer.
• What it takes. At least 64 FLOAT32 vectors in the reordered field across the call's inputs (two times the default minPartitionSize of 32). Default settings, ordinary data, SerialMergeScheduler or ConcurrentMergeScheduler. No scale is needed. With several inputs, one without vectors for the field may hit a NullPointerException in the merged view's copy() instead (LFV-269).
• Not affected. A destination with an index sort (IndexWriter then skips the reorder); plain segment readers; the postings BPIndexReorderer; every release (the class is on main and branch_11x only).
• What the user sees. A clean, unchecked UnsupportedOperationException (no message) from addIndexes, within milliseconds. Nothing is added, nothing is lost; the writer stays open and is not tragic.
• Workarounds. Give the destination an index sort, write the sorted index first and add plain segment readers, or do not use the vector reorderer on that writer.
• How likely. Rare: the combination reorders the data twice, because the vector reorder discards the order the SortingCodecReader provides. Once on that path, it fails on every call.

Put the file in lucene/misc/src/test/org/apache/lucene/misc/index/ (the package of the upstream BP tests) and run the command. The formatter leaves it unchanged. 64 is the smallest input that reaches the throw (63 vectors are never split); the reversing doc map has the shape BPIndexReorderer.reorder returns, without needing postings; SerialMergeScheduler keeps everything in the test thread. The AssertionError states expected vs actual and keeps the cause. The same test is the suggested regression test; it does not check the doc-space half of the fix (see Suggested fix).

# stock 645d4fe14a20, the env command, seed 5EED0404; the same with -Ptests.asserts=false
TestBpVectorReorderSortingCodecReader > testAddIndexesOfSortingCodecReader FAILED
    java.lang.AssertionError: expected addIndexes to add (and reorder) the 64 documents, got java.lang.UnsupportedOperationException
        at org.apache.lucene.misc.index.TestBpVectorReorderSortingCodecReader.testAddIndexesOfSortingCodecReader(TestBpVectorReorderSortingCodecReader.java:74)
        Caused by:
        java.lang.UnsupportedOperationException
            at org.apache.lucene.index.SortingCodecReader$SortingFloatVectorValues.copy(SortingCodecReader.java:347)
            at org.apache.lucene.misc.index.BpVectorReorderer$PerThreadState.<init>(BpVectorReorderer.java:110)
            at org.apache.lucene.misc.index.BpVectorReorderer$1.initialValue(BpVectorReorderer.java:623)
            at org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:70)
            at org.apache.lucene.misc.index.BpVectorReorderer$ReorderTask.compute(BpVectorReorderer.java:217)
            at org.apache.lucene.misc.index.BpVectorReorderer.computeDocMap(BpVectorReorderer.java:591)
            at org.apache.lucene.misc.index.BPReorderingMergePolicy$1.reorder(BPReorderingMergePolicy.java:138)
            at org.apache.lucene.index.IndexWriter.addIndexesReaderMerge(IndexWriter.java:3562)
            at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3312)
1 test completed, 1 failed

# with the recommended fix: the same command, with and without assertions
:lucene:misc:test (SUCCESS): 1 test

# (some frames and the module prefixes are left out)

• Contracts. SortingCodecReader (SortingCodecReader.java:47-52): re-sort an index "by wrapping all readers of the index with this reader and adding it to a fresh IndexWriter via IndexWriter#addIndexes(CodecReader...). NOTE: This reader should only be used for merging." BPIndexReorderer.reorder (BPIndexReorderer.java:825-842): the returned reader "should typically be used in a call to IndexWriter#addIndexes(CodecReader...)"; its class example ends in w.addIndexes(reorderedReaderView) (:73-89). IndexReorderer.computeDocMap takes any CodecReader (IndexReorderer.java:25-35). The policy "simply skips reordering in order not to fail the merge" when RAM is short (BPReorderingMergePolicy.java:34-37).
• Tests. TestBPReorderingMergePolicy randomly plugs BpVectorReorderer into the policy, including testReorderOnAddIndexes (:49-57, :147-148), but only with SlowCodecReaderWrapper inputs (:189). No test, javadoc, CHANGES or MIGRATE entry excludes SortingCodecReader inputs.
• History. #13779 (6053e1e3137, 2024-09-28, by msokolov) made the sorting views' copy() throw on purpose; its body relies "on testing to catch missing required methods", because "some of these readers would only ever be used iteratively", and a review comment by benwtrent asked that sub-classes "purposefully indicate that this is unsupported" (on createIterator(), about the pattern in general, not on copy()). #14097 (02a09f6ab12, 2025-01-07, same author) added the copying BpVectorReorderer and "refactored to make BpVectorReorderer compatible with BPIndexReorderer so that either can plug into BPReorderingMergePolicy"; none of its comments mention copy(), SortingCodecReader or addIndexes.
• Precedent. #16705 (by goankur, merged 2026-10-01) replaced the throwing copy() of ExitableDirectoryReader's vector views with a delegating copy because a new consumer needed one; the same commit edited SortingCodecReader.java and left its copy() throwing.
• Framing.

Strongest case for "intended or harmless": the throw is deliberate (#13779); SortingCodecReader is @lucene.experimental and "should only be used for merging"; and the combination is pointless, because the vector reorder throws away the order the caller just built, so a user who wants vector BP should pass plain readers.

Why it fails: this is merging, the documented use, with the documented output of BPIndexReorderer.reorder. #13779's own rationale expects a method that a real consumer needs to be caught and filled in, and #14097, by the same author, says either reorderer should plug into the policy; the postings reorderer accepts the same input. For a reorder it cannot afford (not enough RAM) the policy skips the reorder rather than fail the merge; that rule is about RAM only, and nothing documents what should happen when the reorderer cannot read an input. Redundancy lowers the likelihood (rare), but it does not make an exception from an accepted call correct. Verdict: a defect. Expect the "why reorder twice?" question in review.

One fix (option A) was built and validated in a scratch worktree at 645d4fe14a20 (removed afterwards); the doc-space half was also checked with the verifiers' probes. Options B-D were not run. Option E is the first, incomplete fix, kept to show why it is not enough.

Two halves, +31/-15 lines in two files, no API change. (1) In SortingCodecReader, each sorting vector view gets a private constructor taking a delegate and the existing SortingIteratorSupplier, and copy() returns new Sorting*VectorValues(delegate.copy(), iteratorSupplier); the supplier's arrays are never modified after construction, so copies can share them, as iterator() already does. (2) In BpVectorReorderer.valueMapToDocMap, drop the "every document has a vector" shortcut and look up each document's new position by its own ordinal, it.index(). For a plain reader the result is identical, because its ordinals follow doc order.

// SortingCodecReader: each Sorting*VectorValues (float shown)
@Override
public FloatVectorValues copy() throws IOException {
  return new SortingFloatVectorValues(delegate.copy(), iteratorSupplier);
}

// BpVectorReorderer.valueMapToDocMap: no dense shortcut; ordinals from the iterator
KnnVectorValues.DocIndexIterator it = values.iterator();
for (int nextDoc = it.nextDoc(); nextDoc != NO_MORE_DOCS; nextDoc = it.nextDoc()) {
  // ... documents without a vector go to the end, as before
  int newOrd = valueMap.oldToNew(it.index()); // was: valueMap.oldToNew(ord++)

Trade-offs

• Land with LFV-278. With two SortingCodecReader inputs in one call, this fix makes addIndexes succeed, and then the merged view's iterator puts the wrong vector on 64 of 64 documents (LFV-278). With both fixes, all 5 failing shapes and 5 controls of the tie-break probe pass.
• Below 64 vectors nothing is split, and the doc map now follows the wrapped reader's ordinals instead of being the identity: in a 63-document control the output follows the underlying reader's order, not the reversed view's. No contract fixes the output order of a reorder that found nothing to split; mention the change in the PR and leave it to the maintainers whether it is acceptable. Starting the bisection from the view's doc order instead is an untested alternative.
• The dense case builds two int[maxDoc] arrays instead of reusing the permutation; small next to the bisection.
• copy() now delegates, so a sorting view is only as copyable as the vector values it wraps.
• The regression test checks only that addIndexes succeeds; the doc-space half is covered by the probes, not by the test; an upstream PR should add a test of the doc map.

What the checks validated

• Stock 645d4fe14a20: the test fails 1 of 1 through gradle, assertions on and off. With the fix: passes, assertions on and off.
• TestBpVectorReorderer, TestBPReorderingMergePolicy, TestBPIndexReorderer: 19 tests, 0 failures, 2 skipped (@Nightly); the first two with -Ptests.nightly=true -Ptests.iters=5: 55 tests, 0 skipped, 0 failures.
• TestSortingCodecReader and TestIndexSorting (core): 65 tests, 0 failures.
• :lucene:misc:test: 24 classes, 317 tests, 9 skipped, 0 failures, assertions on and off; :lucene:core:check -x test :lucene:misc:check -x test passes.
• Doc space (stock jars with the patched classes shadowed): a 512-vector index behind a reversing view gets the correctly re-based doc map at 512 of 512 positions; cluster changes after BP 35, the same as for the plain reader (copy()-only: 0 of 512 and 383).

Build each per-thread state from reader.getFloatVectorValues(field) instead of vectors.copy().

Trade-offs

Works for any reader without copy(), but costs a new doc mapping (an int[maxDoc] plus a bit set) per thread for sorting views, and still needs the ordinal half of option A.

BPReorderingMergePolicy catches it like NotEnoughRAMException and leaves the merge unreordered.

Trade-offs

Hides the incompatibility; the input silently stays unreordered (bp.reordered=false); the catch is broad.

Say that SortingCodecReader inputs are unsupported with BpVectorReorderer.

Trade-offs

Leaves the documented BPIndexReorderer.reorder -> addIndexes recipe broken for this policy.

Only half (1) of option A, the first fix the hunt wrote.

Trade-offs

The test passes, but the reorder is applied to other documents: the doc map equals the plain reader's at 512 of 512 positions and the correctly re-based one at 0 of 512; cluster changes 383 instead of 35.

Method: non-formal (the first signal was a hand-written JUnit).

• A hunt unit (vector doc/ordinal mapping, 2026-10-26) wrote a public-API test that re-sorts a segment with SortingCodecReader, adds it to a writer with BPReorderingMergePolicy(BpVectorReorderer) and checks that every document keeps its own vector; it failed earlier, with this UnsupportedOperationException.
• A hunt unit analysed it: a bytecode inventory of the 128 KnnVectorValues / RandomVectorScorerSupplier implementations and every production copy() call site, and a bounded check of every reachable (view, caller) pair (7,488 checks plus 576 scorer-supplier checks, assertions on and off). This was the only reachable throwing pair that was neither registered nor already known.
• Then: reviews by an independent reviewer from another model family (OpenAI GPT-5.6, reading only), an independent reproducer, a devil's advocate who measured the doc-space problem, an adjudication, a credibility audit, a re-verification on 645d4fe14a20, and a tie-breaker with its own plain main() probe.