Copying a ParallelLeafReader into an index stores a later reader's stored fields under another field's name

ParallelLeafReader joins indexes that hold different fields of the same documents; "Each document contains the union of the fields of all documents with the same document number". Reading through it is correct. Copying it into an index, with IndexWriter.addIndexes(SlowCodecReaderWrapper.wrap(parallelReader)) or from a MergePolicy.OneMerge.wrapForMerge that returns one, writes each stored value of a later reader under the name of whichever target field holds that reader's own field number (an earlier reader's field, or another field of the same reader) whenever the target gives the parallel reader's fields the parallel reader's numbers, for example an empty target. With index A storing title="Lucene in Action" and index B storing price="42", the copy holds title="Lucene in Action" and title="42" and no price, with no exception and a clean CheckIndex. storedFields() lets each sub-reader call the visitor with its own FieldInfo (price is number 0 in B), the parallel reader's getFieldInfos() numbers price 1, and StoredFieldsWriter.MergeVisitor skips its by-name remapping because those numbers match the target's, so it writes number 0, which in the target is title. In every tested case postings, doc values, points, vectors, norms and deletions were copied correctly. The merge shortcut came with LUCENE-6133 (5.0.0, 2014), a speed-up; 4.10.x copies correctly, and every release from 5.0.0 to 10.6.0 (10.6.0 run at its release revision) and main is affected.

• ParallelLeafReader: a LeafReader over several indexes with the same documents in the same order but different fields: "Each document contains the union of the fields of all documents with the same document number" (index/ParallelLeafReader.java:38-39).
• Field numbers: every index gives each field name a number, in the order it first sees the names, so independently built indexes all start at 0. A FieldInfo carries a field's name and number; stored fields are written on disk by number.
• Copying a reader: IndexWriter.addIndexes(CodecReader...) "Merges the provided indexes into this index"; SlowCodecReaderWrapper.wrap turns any LeafReader into the CodecReader it needs. A merge can also wrap each segment with OneMerge.wrapForMerge, "in order to add/remove fields or documents".

1. The view renumbers. ParallelLeafReader builds its own FieldInfos, first reader wins (ParallelLeafReader.java:154-159); a field keeps its number only if it is still free, else it gets the lowest free one (FieldInfos.java:425-433). B's price, number 0, becomes 1. The javadoc says the numbers "will likely not correspond to the actual field numbers in the underlying readers" (:242-244).
2. Stored fields are visited with the sub-reader's own numbers. storedFields().document(docID, visitor) hands the caller's visitor unchanged to each sub-reader (:291-296), and SlowCodecReaderWrapper passes it straight through (SlowCodecReaderWrapper.java:332-335). B's reader reports price as number 0.
3. The merge trusts numbers that look aligned. MergeVisitor turns its by-name remapping on only if a number in the source's getFieldInfos() names a different field in the target (codecs/StoredFieldsWriter.java:182-192, "if field numbers are aligned, we can save hash lookups"). For an empty target the merged numbering adopts the view's (title 0, price 1), so the check passes and remap() returns the visited FieldInfo as is (:237-243).
4. The codec writes the foreign number. The default stored-fields writer copies a non-codec source field by field through this visitor (Lucene90CompressingStoredFieldsWriter.java:678-685, :637-641) and records number 0, which in the new index is title.

If the target gives even one of the view's fields a different number, the constructor check fails, remapping by name is on and the copy is correct. If only the first reader stores fields, no stored value carries a foreign number; if the earlier readers have no fields at all, a later reader keeps its own numbers and nothing is renamed (read from FieldInfos.java:425-427, not run).

With several stored fields in the later reader, each value lands on the name that holds its own number in the target: A color="red", B weight=7, shape="circle" copies as {color=[red, 7], weight=[circle]}. Values move down by as many names as the earlier readers have fields, so shape lands on B's own weight; with two fields in A they move down by two.

• Who. Applications that copy a ParallelLeafReader, or the leaves of a ParallelCompositeReader (same result in one probe), built over two or more indexes: addIndexes(SlowCodecReaderWrapper.wrap(...)) to turn a parallel index into a single one, or a OneMerge.wrapForMerge that folds a side index into segments as they merge (the pattern of upstream's TestDemoParallelLeafReader). A rarely used API combination; everything else is default (IndexWriterConfig, codec, merge scheduler).
• Trigger. Two conditions. (1) A later sub-reader has a stored field whose own number differs from its number in the view; that is always so in the usual setup, where every index numbers its fields from 0 and an earlier reader has at least one field. (2) The target gives each of the view's fields the view's number: an empty target, or one that already holds the same fields under the same numbers. No special data and no scale; in the usual setup every copied document with a stored value from a later reader is hit.
• Effect. Silent and permanent. In the copy each stored value of a later reader carries the name of whichever field holds that reader's own number in the target, and the real field is missing: with one field in B that is an earlier reader's field (price becomes title); with several, values move down by as many names as the earlier readers have fields, so some land on other fields of the same reader. Postings, doc values, points, vectors, norms and deletions were copied correctly in every tested case, so queries match the same documents; what breaks is the stored data the application loads: displayed values, highlighting, and any later re-indexing from stored fields, which carries the wrong names on. The stored-only cases are CheckIndex-clean, so nothing reports it; if the source indexes are discarded after the copy, the original values are lost.
• Not hit when the target's numbering differs (the merge then remaps by name) or when only the first reader stores fields. The wrapForMerge path needs the stored-field side index after the segment reader; upstream's demo puts its side index first and takes the safe path.
• How often among small cases. A bounded-exhaustive run over 128,625 configurations found stored-field mismatches in 21,993 empty-target copies and none where the target's numbering differs; with fix A, none (8,040-configuration sample).
• Downstream, by reading only. Elasticsearch and OpenSearch use ParallelLeafReader on a read path only. Solr main (not in a release) has a wrapForMerge that returns SlowCodecReaderWrapper.wrap(new ParallelLeafReader(...)), but its side segments hold only doc values and its padding wrapper exposes no stored fields, so it is not hit as written. Nothing downstream was run.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: copied document: Document<stored<title:Lucene in Action> stored<title:42>> expected:<42> but was:<null>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.apache.lucene.index.TestParallelLeafReaderAddIndexesStoredFields.testAddIndexesOfParallelReader(TestParallelLeafReaderAddIndexesStoredFields.java:51)

• Contracts. ParallelLeafReader: "Each document contains the union of the fields of all documents with the same document number" (ParallelLeafReader.java:38-39). IndexWriter.addIndexes(CodecReader...): "Merges the provided indexes into this index" (IndexWriter.java:3255). SlowCodecReaderWrapper: "Wraps arbitrary readers for merging" (:38). OneMerge.wrapForMerge: "Wrap a reader prior to merging in order to add/remove fields or documents" (MergePolicy.java:302). The addIndexes path uses no experimental or internal API; the merge path does: MergePolicy, MergePolicy.OneMerge (which declares wrapForMerge) and the repro's FilterMergePolicy are @lucene.experimental (MergePolicy.java:69, :189; FilterMergePolicy.java:27).
• Lucene's own wrappers keep the invariant. SlowCompositeCodecReaderWrapper (#12622, 2023): "Remap FieldInfos to make sure consumers only see field infos from the composite reader, not from individual leaves" (SlowCompositeCodecReaderWrapper.java:118-122, used in every visitor callback at :162-205). The test framework's MismatchedLeafReader, which renumbers fields on purpose, also renumbers the FieldInfo its visitor passes on (MismatchedLeafReader.java:110-135); its issue, LUCENE-6135 (#7197), says merge shortcuts must be off when numbers do not line up, or values get mixed up across fields.
• History. Before LUCENE-6133 the merge loaded each document and wrote it through fieldInfos.fieldInfo(field.name()), by name; the removed code even had a TODO to "renumber the field number on the fly". LUCENE-6133 (5.0.0) was a speed-up, "Improve default StoredFieldsWriter.merge() to be more efficient" (lucene/CHANGES.txt:11324), and its issue (#7195) discusses only CPU cost. 4.10.4 copies correctly; builds of the commit fail and of its parent pass.
• Copying parallel readers is a tested path. TestParallelReaderEmptyIndex: "Adding this reader to a new index should not throw any exception", then addIndexes(SlowCodecReaderWrapper.wrap(pr)) (:34-36, :129-134); TestDemoParallelLeafReader returns SlowCodecReaderWrapper.wrap(ParallelLeafReader) from wrapForMerge (:644-650); LUCENE-1918 (2.9.0) fixed exceptions when a ParallelReader was "passed to IndexWriter.addIndexes(IndexReader[])" (lucene/CHANGES.txt:19473-19478); in a 2009 java-user answer a committer (@mikemccand) suggested "use addIndexes(IndexReader[]), passing in your ParallelReader". None of these tests has a stored field in a later reader with an aligned target.
• Precedent. LUCENE-3575 (#4649, 2011), "Field names can be wrong for stored fields / term vectors after merging", the same symptom in older merge code, was fixed as a bug.

The intended-behaviour verifier built the strongest case for "not a bug": field numbers are internal; getFieldInfos() warns its numbers "will likely not correspond" to the sub-readers'; StoredFieldVisitor is @lucene.experimental (StoredFieldVisitor.java:35); the class javadoc names searching as the use case (ParallelLeafReader.java:42-44) and warns that indexes not "created and modified the same way" give "undefined behavior" (:46-49); and wrapForMerge is experimental merge-policy API where the application supplies the reader, so maintainers could ask that reader to present consistent FieldInfos.

Why it fails for addIndexes: the NOTE came with LUCENE-4055 (2012, about codec metadata), two and a half years before the merge shortcut, and describes a numbering difference, not renamed persisted data. The addIndexes user calls only public, non-experimental API and never sees a FieldInfo. The "same way" warning's only example concerns documents and their order and it states no field-number requirement, while the same javadoc expects different fields per index, which always number from 0. Upstream tests and CHANGES copy parallel readers. The cross-model reviews and the verifier rated the case weak; a cross-model fact check of this doc rates it moderate for wrapForMerge, the experimental path. Verdict: a defect. Copying is supported by tests, CHANGES and a list answer rather than by the class javadoc.

Two fixes were applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF, assertions on): A, at the class that breaks the invariant, and B, a defensive check in the merge. Either one fixes both repros. Neither makes the expert-constructor aside copy correctly.

Wrap the caller's visitor so that needsField and the seven value callbacks receive the parallel reader's FieldInfo for that field name, falling back to the sub-reader's when the name is unknown. The same pattern as SlowCompositeCodecReaderWrapper. Patch: optionA-parallelleafreader-remap-visitor.diff, the finder's sketch, +60/-1 in ParallelLeafReader.java.

public void document(int docID, StoredFieldVisitor visitor) throws IOException {
  // Sub-readers report their own FieldInfo (their own field numbers); present this
  // reader's FieldInfo instead, so consumers that trust FieldInfo.number (e.g. the
  // stored fields merge) see numbers consistent with getFieldInfos().
  StoredFieldVisitor remapping = new RemappingStoredFieldVisitor(visitor, fieldInfos);
  for (StoredFields reader : fields) {
    reader.document(docID, remapping);
  }
}
// RemappingStoredFieldVisitor.remap(fi):
//   FieldInfo own = fieldInfos.fieldInfo(fi.name); return own == null ? fi : own;

Trade-offs

• Two hash lookups by name (FieldInfos.fieldInfo(String)) per loaded stored field (one in needsField, one in the value callback; one for a skipped field) and one wrapper-visitor allocation per document() call, on every read through a ParallelLeafReader, at search time too; it could be skipped for sub-readers whose numbering already matches. No benchmark was run.
• Search-time visitors now see numbers consistent with getFieldInfos(), arguably a fix too.
• It protects ParallelLeafReader only, not other wrapping readers.

What the checks validated

• Stock: the finder's test fails 2 of 2, the independent test 2 of 3 (its control passes), the expert test 1 of 1.
• With A: the finder's test 2/2 and the independent test 3/3 pass; the expert test still fails with the same rename.
• Closest suites with A: TestParallel* (4 upstream classes and the 2 new ones), TestDemoParallelLeafReader, TestAddIndexes, TestFieldsReader, TestFilterCodecReader, TestFilterLeafReader, TestStoredFieldsConsumer, TestSortingStoredFieldsConsumer, TestStoredFieldsMergeReadAdvice, TestLucene90StoredFieldsFormat* and codecs.lucene90.compressing.*: 20 classes, 196 tests, 8 skipped, 0 failures.
• Earlier: the adjudication's re-run passes 3/3 with it; a cross-model reviewer ran 28 related upstream tests with it (all pass); the exhaustive harness shows 0 stored-field mismatches with it (8,040-configuration sample).

When the constructor found the numbering aligned, remap() checks that the visited number names the same field in the merged FieldInfos, and otherwise looks the field up by name. Patch: optionB-mergevisitor-check-number.diff, +9/-1 in StoredFieldsWriter.java.

// in MergeVisitor.remap(), after the existing remapper branch:
FieldInfo merged = mergeFieldInfos.fieldInfo(field.number);
if (merged != null && merged.name.equals(field.name)) {
  return field;
}
return mergeFieldInfos.fieldInfo(field.name);

Trade-offs

• Protects every merge that copies stored fields through StoredFieldsWriter.MergeVisitor (the default merge and the Lucene90 VISITOR strategy) against any wrapping reader that hands it foreign FieldInfo, at the cost of one array lookup and one string comparison per stored field on that path only; the codec's bulk and per-document copy paths are untouched. No benchmark was run.
• A stored-fields format that overrides merge without MergeVisitor is not covered, and a name the target does not know still yields null (the NullPointerException of the expert aside).
• Search-time visitors of a ParallelLeafReader still see sub-reader numbers.

What the checks validated

• With B (A reverted): the finder's test 2/2 and the independent test 3/3 pass.
• The expert aside now fails with NullPointerException: Cannot read field "number" because "info" is null in the merge thread (Lucene90CompressingStoredFieldsWriter.writeField): B turns that silent rename into an exception, it does not make it copy.
• The same closest core suites plus TestSimpleTextStoredFieldsFormat (merges through the default StoredFieldsWriter.merge): 21 classes, 215 tests, 11 skipped, 0 failures.

A fixes the reader that breaks the invariant; B guards the merge against any other reader that does the same.

Trade-offs

Two changes to review. Each was validated alone through gradle and the closest suites; together they were run once, in this doc's fact check, through the JUnit runner with both patched files shadowed: the finder's and the independent repros pass 5 of 5 (not run through gradle or the suites).

Add their fields to the parallel FieldInfos, or reject copying such a reader.

Trade-offs

A design question for upstream: what that copy should contain is not documented. Not run.

Method: bounded-exhaustive metamorphic translation validation (a hunt unit, 2026-10-27). No model checker, solver or prover was involved.

• The relation. A ParallelLeafReader over readers A and B, and its addIndexes copies into an empty target, a reverse-numbered target, an aligned target and through the expert constructor, must equal a reference single index built from the javadoc's union and first-owner rules, compared per live document: live docs, FieldInfos, postings, norms, doc values, points, vectors, stored fields and term vectors.
• The space. 3 fields; in each of the 2 readers a field is absent or one of {stored, text with term vectors, numeric doc values, 1D int point, 2D float vector}, not absent in both, so 35 combinations per field; 35^3 x 3 deletion modes = 128,625 configurations, 3 documents per reader, 686,000 checks. Run on the stock jars of ea57360912ed, whose stored-field code on this path is identical to a9779ef3c193 (the only differences in the classes involved, in SlowCodecReaderWrapper and the stored-fields reader, are vector-hint and merge-stream code); local samples on a9779ef3c193 agree.
• The result. Stored-field mismatches in 21,993 empty-target copies, 21,993 aligned-target copies and 7,331 expert-constructor copies; 0 in reverse-numbered targets; 0 with the fix sketch (8,040-configuration sample). The only other mismatch kind was term vectors, a separate registered bug on the same path. Live docs, postings, norms, numeric doc values, 1D points and float vectors never differed; other doc-values types, byte vectors, multi-dimensional points, payloads and fields that are both stored and indexed were not in the model.
• Order of events. The hypothesis came from reading code while writing the model (the stored-fields visitor is passed through while getFieldInfos() renumbers); a probe confirmed it; the exhaustive run measured it. The finder (an AI tool) then reduced it to a public-API JUnit, ran the release bracket and wrote the fix sketch.
• Then cross-model reviews, independent verifiers, an adjudication, a history bisect, a prior-report search, a security screen and an audit (see Independent checks).