ParallelCompositeReader picks a shared field's owner separately in each segment, so a later index's values show through

ParallelCompositeReader documents that "matches for a query term are from the first index added that has the field" (ParallelCompositeReader.java:29-30 at e1faeba08bde). But prepareLeafReaders builds one ParallelLeafReader per segment position from that position's leaves only (:119-137), and ParallelLeafReader gives each field to the first sub-reader whose own leaf FieldInfos contain it (ParallelLeafReader.java:154-171). In a segment of the first index with no FieldInfo for the field (in the example, no document of that segment has it), the second index serves the field. With two 3-document indexes whose segments are [0, 1] and [2], the first having t=new on doc 0 only and the second t=old everywhere, t:old returns [2] instead of []; identical docs 1 and 2 answer differently, and after forceMerge(1) of both indexes the answer is []. Doc values behave the same (executed); norms, points and vectors use the same owner map. Wrong since 4.0 (LUCENE-3736, 2012); the 3.x ParallelReader chose owners per index and gives [].

• ParallelCompositeReader: "An CompositeReader which reads multiple, parallel indexes. Each index added must have the same number of documents, and exactly the same number of leaves (with equal maxDoc), but typically each contains different fields. Deletions are taken from the first reader. Each document contains the union of the fields of all documents with the same document number. When searching, matches for a query term are from the first index added that has the field." (ParallelCompositeReader.java:26-30). Its use case: "The smaller fields may be re-indexed in a new index and both indexes may be searched together." (:32-34).
• ParallelLeafReader: the same reader for single segments, with the same precedence sentence (ParallelLeafReader.java:38-40).
• FieldInfos: per segment: the fields of the documents written to that segment (documents deleted later and doc-values updates included). A field that no document written to a segment had is not in that segment's FieldInfos.

1. prepareLeafReaders checks that all readers have the same leaf count and leaf maxDoc (ParallelCompositeReader.java:105-115, validate at :142-162), then, for each leaf position i, builds new ParallelLeafReader(true, subs, storedSubs) from readers[j].leaves().get(i).reader() only (:119-137).
2. ParallelLeafReader's constructor walks its sub-readers in order and, for each FieldInfo of that sub-reader's own getFieldInfos(), records it in fieldToReader and termsFieldToReader unless an earlier sub-reader already has it: // NOTE: first reader having a given field "wins": (ParallelLeafReader.java:154-171).
3. Segment [2] of the first index (doc 2, no t) has no FieldInfo for t, so at position 2 the second index wins t. At position 0 the first index (doc 0 has t=new) wins.
4. terms("t") reads termsFieldToReader (ParallelLeafReader.java:258-262): t:old finds doc 2 in segment [2] and nothing in segment [0, 1]. Doc values, norms, points and vectors read fieldToReader (:396-503), the same per-segment choice.
5. After forceMerge(1) of both indexes there is one segment, the first index's segment has t, and t:old matches nothing.

3 documents per index, segments [0, 1] and [2], first index t=new on doc 0, second index t=old on docs 0-2: t:old returns [2] (expected []); t:new returns [0]. After forceMerge(1) of both: t:old returns [], t:new [0] (main, 10.6.0, 9.12.3 and 9.0.0). Doc-values probe, same layout with NumericDocValuesField("n") (n=1 on doc 0 of the first index, n=9 on every document of the second): FieldExistsQuery("n") returns [0, 2] instead of [0], n == 9 returns [2] instead of [].

• Who. Users of ParallelCompositeReader (an expert, rarely used class) whose parallel indexes share a field: the overlap the javadoc's precedence rule is written for. The class javadoc's own use case is re-indexing frequently changing fields in a second index (ParallelCompositeReader.java:32-34); a field kept in both, or re-indexed only for some documents, has this shape. Overlap probably occurs in practice: a 2016 java-user post ("Faceted Search with Parallel Indexes", unanswered) used a ParallelCompositeReader over two indexes with taxonomy facets and got facet results only from whichever index came first. That both indexes carried the same facet field (taxonomy facets share a default $facets field) is an inference; the post shows no schema. What they describe matches the documented first-index rule, not this bug.
• When. Some segment of an earlier index has no FieldInfo for the shared field (no document written to it had the field) while a later index has the field in the same segment. Small segments and sparsely set fields make this more likely (no frequency data), and it changes as segments flush and merge.
• Effect. Silent extra hits and wrong counts on the shared field; in a MUST_NOT clause the same extra matches remove documents instead (code reading, not run). Two documents with identical data in every index answer differently depending on their segment, and an ordinary merge changes the answer ([2] before, [] after forceMerge(1) of both indexes). Executed for TermQuery, FieldExistsQuery and a BooleanQuery on a text field and, for this doc, for FieldExistsQuery and an exact doc-values query on a numeric doc-values field ([0, 2] instead of [0], [2] instead of []). Norms, points and vectors use the same owner map (code reading, not run).
• Default vs opt-in. Opt-in: Lucene never builds a ParallelCompositeReader over two different indexes by itself (the test framework's wrapReader builds a ParallelCompositeReader over one reader only, or a ParallelLeafReader over two field-disjoint halves of the same leaf; in neither does a field occur twice). Codec-independent by code reading: the gradle runs used Asserting(Lucene104), and the fact-check runs failed the same way with 4 more seeds, under CheapBastard and a compressing stored-fields codec as well. The repro uses the default IndexWriterConfig plus NoMergePolicy only to pin the segment boundaries; the merge probe (the same t values plus a stored id field) used LogDocMergePolicy, the policy the class javadoc recommends (:39-41), with the same result.
• Scale and likelihood. Three documents. The first signal was 6 of 3,720 checks of a randomized sweep, all with a later index re-using a field. No public report of it was found since 4.0 (2012), so likelihood rare and priority Low.

TestParallelCompositeReaderFieldOwner (one method, extends LuceneTestCase, package org.apache.lucene.index of the lucene/core tests). Each index is written with new IndexWriterConfig().setMergePolicy(NoMergePolicy.INSTANCE) and a commit() after doc 1, which pins the segments to [0, 1] and [2] in both. The first index has StringField("t", "new") on doc 0 only; the second has StringField("t", "old") on every document. The reader is new ParallelCompositeReader(DirectoryReader.open(first), DirectoryReader.open(second)).

./gradlew tidy re-wraps three long lines of the file (the ParallelCompositeReader construction, the Arrays.stream chain and the comment; formatting only). The file has no Apache license header. Deterministic for its input; the recorded runs used seed DEADBEEF (codec Asserting(Lucene104)), and 4 more seeds failed the same way. As a regression test the same method fits in TestParallelCompositeReader. The merge dependence ([2] before, [] after forceMerge(1) of both indexes) is shown by the merge probe, not by this test.

cp $REPRO/TestParallelCompositeReaderFieldOwner.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestParallelCompositeReaderFieldOwner -Ptests.seed=DEADBEEF

# gradle, scratch worktree at e1faeba08bde, -Ptests.seed=DEADBEEF (same with -Ptests.asserts=false)
TestParallelCompositeReaderFieldOwner > testFirstIndexOwnsSharedField FAILED
    java.lang.AssertionError: expected:<[]> but was:<[2]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:FDB92538CC60A78]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.index.TestParallelCompositeReaderFieldOwner.testFirstIndexOwnsSharedField(TestParallelCompositeReaderFieldOwner.java:43)
:lucene:core:test (FAILURE): 1 test, 1  failure

# same test, standalone JUnit runner on the main jars: -ea and -da FAIL, expected:<[]> but was:<[2]>
# with the recommended fix: PASS, asserts on and off

# evidence probe (plain main; same t values plus a stored id field; LogDocMergePolicy), main / 10.6.0 / 9.12.3 / 9.0.0, -da:
segments=2 t:old -> [2] t:new -> [0]
segments=1 t:old -> [] t:new -> [0]      # after forceMerge(1) of both indexes

• Contract. "Each index added must have the same number of documents, and exactly the same number of leaves ... When searching, matches for a query term are from the first index added that has the field." (ParallelCompositeReader.java:26-30). In the same paragraph "index added" is the composite argument, not a leaf.
• The warning does not apply. "Undefined behavior" is promised only when the indexes are not "created and modified the same way" (:36-44). Here both have the same documents in the same order and the same segment boundaries (the constructor checks the leaf count and each leaf's maxDoc, :142-162).
• No per-index or per-document reading gives the observed answer. Per index (the javadoc): []. Per document ("the union of the fields of all documents with the same document number"): [1, 2] if the first index wins in each document, or [0, 1, 2] if both indexes' values were unioned. Only a per-segment reading gives [2], the code's answer, and it treats identical documents 1 and 2 differently.
• Overlap is supported input. Both Parallel* javadocs define the overlap case (ParallelLeafReader.java:38-40), and an upstream test relies on it: "important that parallelReader is first, so if there are field name overlaps, because changes to the schema overwrote existing field names, it wins" (TestDemoParallelLeafReader.java:178-180). LUCENE-606 (#1684) describes the 3.x overlap behaviour: a term search "only yields results if that term was contained in the first matching IndexReader which contained the field".
• History. Before 4.0, ParallelReader took owners from the merged FieldInfos of the whole index; 3.6.2 gives [] for this scenario (an API-adapted port). LUCENE-3736 (6188bc66d7d, 2012) made the new composite variant work per leaf for performance ("as it works on the atomic subreaders") and kept the 3.x sentence; its CHANGES entry offers SlowCompositeReaderWrapper + ParallelAtomicReader as the alternative (CHANGES.txt:15675-15681), which still chose owners per index (a 4.0.0 control gives []). Comments of 2012-02-21 on LUCENE-3736 by Uwe Schindler and Robert Muir discuss how the field maps are built ("The fieldInfos and fieldToReaderMap is now only used for retrieving stored fields and other global information."), but none of its 45 comments and nothing in LUCENE-6501 ("a flat structure of ParallelLeafReaders", CHANGES.txt:10577-10584) discusses a field present in more than one index, or a segment of the earlier index without the field.
• Tests. TestParallelCompositeReader uses disjoint fields; no upstream test asserts per-segment ownership.

The strongest case for "intended": field presence is a per-segment property; ParallelLeafReader's rule is defined over leaf FieldInfos; the per-leaf construction is as old as the class, and LUCENE-6501 calls the result "a flat structure of ParallelLeafReaders"; and in 2011 Shai Erera closed LUCENE-606 with "the general concept behind ParallelReader is that fields do not exist in more than 1 sub-reader". So overlapping fields could be called unsupported, with per-leaf precedence as a consequence.

Why it fails: that remark was about 3.x stored fields, in a reader that chose owners per index. The current javadocs of both classes define the overlap case and an upstream test depends on it. The class javadoc states the rule per "index added". A per-leaf reading does give [2], and it is how the code is built; but under it the answer for identical documents depends on a segment boundary and changes after a merge that keeps every document and value, which no javadoc states. The case for "intended" is not empty (an independent reviewer rated it moderate), so a maintainer may choose to document per-segment precedence instead (fix option C).

Three options; the first was built and tested, the other two were not. The choice is the maintainers'.

In prepareLeafReaders, map each field name to the first top-level reader that has it in any leaf, and pass the map to every per-leaf ParallelLeafReader through a new package-private constructor (the public constructors delegate with null). In its FieldInfos loop, a sub-reader skips a field whose owner is another index. Where the owning index has no FieldInfo for the field in a segment, the field is simply absent there, as with the 3.x reader. The full diff has 91 lines; its core is shown below.

// ParallelCompositeReader.prepareLeafReaders: owner = first index with the field in any leaf
final Map<String, Integer> fieldOwners = new HashMap<>();
for (int j = 0; j < readers.length; j++) {
  for (LeafReaderContext ctx : readers[j].leaves()) {
    for (FieldInfo fieldInfo : ctx.reader().getFieldInfos()) {
      fieldOwners.putIfAbsent(fieldInfo.name, j);
    }
  }
}
// ... new ParallelLeafReader(true, subs, storedSubs, fieldOwners) { ... }

// ParallelLeafReader, FieldInfos loop over sub-reader i (fieldOwners is null for public use):
Integer owner = fieldOwners == null ? null : fieldOwners.get(fieldInfo.name);
if (owner != null && owner != i) continue;

Trade-offs

• One extra pass over the FieldInfos of every leaf of every reader when the reader is built, O(leaves x fields); no per-query cost.
• Behaviour change for users who got the later index's values in sparse segments: they now get the per-index rule the javadoc states. It changes per-leaf metadata too, not only query routing: in such a leaf, getFieldInfos() no longer lists the later index's FieldInfo for the field (index options, doc-values type, points, vectors), so the field is absent there. Soft-deletes and parent field names still come from all sub-readers (ParallelLeafReader.java:110-124). Fields whose schema differs between the indexes were not tested; this part needs maintainer review.
• Stored fields (taken from all storedFieldsReaders), the by-document term-vector lookup (termVectors() loops over the sub-readers directly and never reads tvFieldToReader, ParallelLeafReader.java:326-375) and the index-sort metadata are untouched; ParallelLeafReader's public behaviour is unchanged.

What the checks validated

• Gradle in a scratch worktree at e1faeba08bde, JDK 25, -Ptests.seed=DEADBEEF (2026-11-02): the repro fails without the fix and passes with it, assertions on and off.
• With the fix: TestParallelCompositeReader, TestParallelLeafReader, TestParallelReaderEmptyIndex, TestParallelTermEnum, TestDemoParallelLeafReader and the repro pass (30 tests); all of org.apache.lucene.index.* (1,398 tests, 56 skipped) and org.apache.lucene.search.* (2,350 tests, 49 skipped) pass with 0 failures (these two package runs used the same diff before one comment line was shortened for formatting); checkGoogleJavaFormat is clean for the two changed files.
• The doc-values evidence probe gives [0] and [] with the fix (stock main: [0, 2] and [2]).

Decide owners as above, then wrap each later sub-leaf in a FilterLeafReader that drops the fields owned by an earlier index. This was the hunt's first fix idea.

Trade-offs

No change to ParallelLeafReader, but core has no field-filtering leaf reader (the test framework's FieldFilterLeafReader is not in core), and the wrapper must hide the field from getFieldInfos and every per-field accessor, so it is more code and easier to break when accessors are added.

Say in the javadoc that for a field in several indexes the owner is chosen per segment, so results can depend on segment boundaries and merges.

Trade-offs

Cheapest and no behaviour change, but it documents an inconsistency (identical documents answering differently, answers that change after a merge) instead of removing it, and it departs from the 3.x behaviour the javadoc sentence came from.

Method: non-formal (a seeded metamorphic differential; no model checker or proof).

• Sweep. A harness built field-split sub-indexes from random documents (60 seeds, 120 legs), opened them as a ParallelLeafReader or a ParallelCompositeReader, and compared every query result with one index built from the union of the visible fields: 3,720 checks per assertion mode on the stock main jars.
• Signal. 6 checks failed with -ea and with -da, all ParallelCompositeReader legs whose last sub-index re-uses fields, all on the text field t (TermQuery, FieldExistsQuery and a BooleanQuery, one extra hit each). ParallelLeafReader legs and composite legs without a shadowing index were clean.
• Mechanism and reduction. Reading prepareLeafReaders showed the per-segment owner choice; the case was reduced to two 3-document indexes and one JUnit test, and a merge probe showed the answer change after forceMerge(1).
• An earlier model of parallel field ownership in the hunt looked at a single segment only, so it never compared per-segment with per-index ownership.