ParallelLeafReader.termVectors() no longer follows the rule that the first reader having a field owns it

Glue two indexes together with a ParallelLeafReader and, when both have a field called body, the first one owns it: the reader reports its FieldInfo and searches its postings. Since Lucene 10.2.0, termVectors() ignores that owner. If the owner has no term vectors for body and the other index has them, you get the other index's vector (second instead of null in the repro), although the same reader says body has no term vectors. If both have vectors, the winner is decided by identity hash codes, not by the order the readers were passed in (4 of 8 reader orders wrong in the runs). Copy such a reader with addIndexes and the new index fails CheckIndex at level 2 (only at level 3 when both have vectors), while the default levels report it clean. The cause is #14373, an optimization that stopped looking up the field's owner. Opt-in API, default codec, released in 10.2.0 through 10.6.0 (10.6.0 by reading).

• ParallelLeafReader: a reader that glues together several indexes holding the same documents in the same order. Document 0 of the parallel reader is document 0 of every index, with the union of their fields. The class javadoc's use case: large fields that rarely change in one index, small fields that change often in another.
• The owner of a field: when two of the glued indexes have a field with the same name, the first index passed in that has the field owns it. The parallel reader reports the owner's FieldInfo, searches the owner's postings and reads the owner's doc values.
• Term vectors: an optional per-document copy of one field's terms (optionally with positions and offsets), stored when the field is indexed with setStoreTermVectors(true). Highlighters and MoreLikeThis read them. FieldInfo.hasTermVectors() says whether a field has them.

Two indexes with one document each, glued with new ParallelLeafReader(false, reader1, reader2):

Reader 1 owns body. Ask the parallel reader about document 0:

plr.getFieldInfos().fieldInfo("body").hasTermVectors();   // false: the owner has none
plr.terms("body");                                         // the owner's postings: "first"
plr.termVectors().get(0).terms("body");                    // ?

null: the owner has no term vectors for body, the reader's own FieldInfo says so, and TermVectors.get returns "null if term vectors were not indexed". Lucene 10.1.0 returns null.

Reader 2's vector, the term second. The same reader now says three things about body: it has no term vectors, its text is first, and here is a term vector for second. Three checks, with the program in the repro section:

In check 3, which order of a pair is wrong is decided by identity hash codes. In the runs it changed with the test seed, the -ea/-da flag and the Lucene version, while identical re-runs gave the same order.

All lines are in ParallelLeafReader.java at ea57360912ed.

1. The constructor records the owner (154-172): 'first reader having a given field "wins"' (156). For term vectors it adds the field to the map tvFieldToReader only if the owner has term vectors for it ("only add these if the reader responsible for that field name is the current", 160; put at 164). Here that map is {title: reader 1}; body is not in it.
2. termVectors() collects every reader that has any term vectors in an IdentityHashMap (330-336): here both readers.
3. get(docID) takes every field of every one of those readers (349-373): one get(docID) per reader, then each returned field goes into a TreeMap with put (363-368, 219-221), so a later reader replaces an earlier one. It never looks at tvFieldToReader, which is now written and read nowhere (that is what the static analyzer flagged). The comment above the loop, "Step 3: Aggregate only required fields" (362), is not implemented.

Why the winner depends on object identity. When both readers vector body, the last put wins, and the loop visits the readers in the IdentityHashMap's slot order: a reader's slot comes from System.identityHashCode, not from the order the readers were passed in. JDK 25's default IdentityHashMap has 32 slots; a reader whose slot is taken goes to the next one, and the readers are inserted in the order passed in. So for a pair a, b:

• Different slots (the usual case): PLR(a, b) and PLR(b, a) visit the two readers in the same order, the same reader wins in both, and exactly one order is wrong.
• The same slot (about 1 pair in 32 if hashes are uniform), unless it is the last: the reader passed second wins in both orders, so both are wrong.
• The same last slot (about 1 pair in 1,024): the second reader wraps around to slot 0 and the first wins in both orders.

Measured on main over three runs of 4,000 pairs: 117-120 shared a slot, 114-117 had both orders wrong, 2-4 had both right. So "4 of 8" is typical (about 88% of 4-pair runs), not certain.

What 10.1.0 did. get(docID) looped over tvFieldToReader and asked each field's owner for that one field. body had no entry, so no vector came back.

How it reaches disk. SlowCodecReaderWrapper hands the reader's termVectors() to addIndexes and merges (SlowCodecReaderWrapper.java:345-376). With the default codec, the term-vectors writer calls addAllDocVectors per document for such a reader (Lucene90CompressingTermVectorsWriter.java:927-929), and addAllDocVectors writes every field it is given without checking hasTermVectors() (TermVectorsWriter.java:235 and below). This happens only when the parallel reader has some field with term vectors (title here), because the merge skips term vectors otherwise (SegmentMerger.java:170-171). CheckIndex compares term vectors with the field infos only from level 2, and with the postings only at level 3 (CheckIndex.java:1114, 1130-1132, 4360); the command-line default is level 1 (:4673), and a CheckIndex without setLevel runs at level 0. Those levels check only the files' checksums.

• Who is exposed. Users of ParallelLeafReader, and of ParallelCompositeReader, which builds one ParallelLeafReader per leaf (ParallelCompositeReader.java:133), whose glued indexes share a field name where an index other than the first stores term vectors for it. Opt-in, expert API; default codec; no special settings.
• When names overlap. The javadoc's use case normally gives disjoint fields, but overlap is a defined case: Lucene's own TestDemoParallelLeafReader puts a derived index first "so if there are field name overlaps, because changes to the schema overwrote existing field names, it wins" (:178-180). A schema change that re-indexes body without term vectors in the derived index, while the original still has them, is check 1.
• Read path: another index's content. A consumer that reads term vectors without checking the field's FieldInfo first gets the other index's version of the field: MoreLikeThis and the highlighter's TokenSources call termVectors().get directly. UnifiedHighlighter uses term vectors only when FieldInfo.hasTermVectors() is true, so it would skip the leaked vector when the owner has none, but not when both indexes vector the field (by code reading; not run).
• Write path: an inconsistent index. addIndexes(SlowCodecReaderWrapper.wrap(plr)) (executed), or a merge policy whose wrapForMerge returns a ParallelLeafReader as in TestDemoParallelLeafReader (:645-651), writes a segment that CheckIndex rejects: at level 2 or 3 when the owner has no term vectors for the field, and only at level 3 when both indexes vector it (the vectors then disagree with the postings). In that merge pattern, with both indexes vectoring the field, the study saw 14 of 24 merges produce such a segment, checked at level 3, and 0 of 24 with either fix control. CheckIndex's own message says -exorcise would drop the segment with its documents (not run). The input indexes are not modified.
• Silent. No exception on read or write. The default CheckIndex levels (1 on the command line, 0 without setLevel) verify the files' checksums but skip the term-vector consistency checks.
• Depends on object identity. When several indexes vector the field, the winner is decided by identity hash codes. In the runs it changed with the test seed, the -ea/-da flag and the Lucene version (identical re-runs gave the same result), so an unrelated change can flip it.
• Scale. None: one document per index is enough.
• How common. Unknown. No report in the 18 months since 10.2.0 (tagged 2025-04-10). Lucene's randomized test wrapping never overlaps fields: LuceneTestCase.wrapReader builds a ParallelLeafReader over one reader or over two FieldFilterLeafReader halves with disjoint fields (LuceneTestCaseParent.java:2372-2405).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: term vector of body expected null, but was:<second>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotNull(Assert.java:756)
       org.junit.Assert.assertNull(Assert.java:738)
       org.apache.lucene.index.TestParallelLeafReaderTermVectorsOwner.testTermVectorsOnlyFromFirstReaderHavingTheField(TestParallelLeafReaderTermVectorsOwner.java:52)

• The class says the first index wins, for searching. ParallelLeafReader.java:38-40: "Each document contains the union of the fields of all documents with the same document number. When searching, matches for a query term are from the first index added that has the field." ParallelCompositeReader.java:30 says the same.
• The constructor applies that rule to term vectors on purpose. 'first reader having a given field "wins"' (:156) and "only add these if the reader responsible for that field name is the current" (:160), directly above tvFieldToReader.put (:164).
• What is not promised publicly. No public javadoc states the rule for term vectors. For them it comes from the constructor's comments and owner map, from every release up to 10.1.0, and from consistency with getFieldInfos(). The constructor keeps the alternative only as a TODO: "consider populating 1st leaf with vectors even if the field name has been seen on a previous leaf" (:161-162). So the non-owner vector rests on the implementation and its history; the CheckIndex failure and the identity-hash winner do not depend on that question.
• The API contracts. TermVectors.get: "Returns term vectors for this document, or null if term vectors were not indexed" (TermVectors.java:47). CheckIndex rejects "has term vectors for field=... but FieldInfo has storeTermVector=false" (CheckIndex.java:4349-4357).
• Lucene's own demo relies on it. TestDemoParallelLeafReader.java:178-180 puts the derived index first so that it wins on overlapping names.
• The change was an optimization. #14373 (DivyanshIITB; approved and merged by vigyasharma on 2025-03-24): "Replaces per-field term vector fetching with a single call per reader ... Verified via existing tests." The review covered formatting, the CHANGES entry and a null check; no test was added. It closed #7926 (LUCENE-6868), which asked only to stop loading the same vectors once per field. CHANGES.txt:1956 lists it under 10.2.0 "Optimizations". Up to 10.1.0, get() looped over tvFieldToReader.
• No test covers the case. No upstream test has a term-vector field in two glued indexes; the five parallel-reader suites pass with and without the fix.

The strongest case for "intended": (1) the javadoc rule is phrased for searching ("When searching, matches for a query term ..."), and reading term vectors is not searching (the cross-model reviewer's best argument); (2) storedFields() does not apply the owner rule either, it visits every stored-fields reader (:291-296), so the real semantics could be a union; (3) in 2017 (LUCENE-7500, #8551) dsmiley argued that a later reader with term vectors should supply them when the first has none, and mikemccand wrote that "the fields users pass to it should be orthogonal across sub-readers and we really should enforce that".

Why it fails: the constructor still builds the term-vector owner map with a comment that states the rule, 10.1.0 used it, and #14373 dropped the lookup in a change described as a pure optimization. Stored fields are a union in reader order, every value delivered; term vectors keep one value per name, picked by identity hash codes, which no documented semantics would choose. The result contradicts the same reader's getFieldInfos() and terms(), and once written out breaks a CheckIndex invariant. dsmiley's 2017 change was reverted to "functionality identical" in that same issue, and would anyway need the merged FieldInfo to say storeTermVector=true, so checks 2 and 3 stay bugs under any reading. mikemccand's "should enforce" was never implemented, and overlap is used by Lucene's own demo. Verdict: not intended, a regression. An independent reviewer from another model family (OpenAI GPT-5.6) rated the intended-behaviour argument "weak" (verdict real, confidence high); a later fact check by the same model family rated it moderate for the non-owner vector alone, weak for the CheckIndex failure and the identity-hash winner, and weak overall.

One fix was validated with gradle and the repro runner. The other options come from the study, the version-history runs and the 2017 discussion.

Patch (+7/-5 in ParallelLeafReader.java, the study's "ownerfilter" control): skip a field unless tvFieldToReader.get(fieldName) is the current reader, and create ParallelFields only when a field is added.

if (docFields != null) {
  // Step 3: Aggregate only the fields this reader owns (first reader having a field wins)
  for (String fieldName : docFields) {
    if (tvFieldToReader.get(fieldName) != entry.getKey()) {
      continue;
    }
    Terms vector = docFields.terms(fieldName);
    if (vector != null) {
      if (fields == null) {
        fields = new ParallelFields();
      }
      fields.addField(fieldName, vector);
    }
  }
}

Trade-offs

• Keeps #14373's single get(docID) per reader, the O(n) fetch #7926 asked for.
• get() returns null when no owned field has a vector, as in 10.1.0. With the owner filter, creating ParallelFields as soon as a reader returns any Fields (as the #14373 code does) would return an empty, non-null Fields for a document whose only vectors belong to non-owners.
• Still reads, and prefetch still touches, the vectors of readers whose fields are all owned by others. Option B removes that.
• Not run: the full core suite, highlighter or MoreLikeThis tests, any benchmark.

What the checks validated

• Gradle in a scratch worktree at ea57360912ed (JDK 25, removed after): the repro passes 3 of 3 (stock: 3 of 3 fail); TestParallel*, TestDemoParallelLeafReader, TestTermVectors*, TestAddIndexes and TestCheckIndex: 11 suites, 112 tests, 0 failures (stock: 112 tests, 3 failures, all in the repro); the repro with seed 1234ABCD, 10 iterations: 30 of 30 pass (meant to run with assertions off; the command line was not saved); checkGoogleJavaFormat passes.
• The repro runner with the patched class shadowed: the repro passes 3 of 3 with -ea and -da (also re-run by the fact check); the cross-model reviewer's own two tests pass; the plain program prints the 10.1.0 results.

Fill readerToTermVectors only with readers in tvFieldToReader.values() (the study's suggested variant).

Trade-offs

Avoids reading and prefetching vectors that are thrown away. The owner check in get() is still needed, because the owner of one field can also have vectors for a field another reader owns. Not run.

Loop over tvFieldToReader and ask each owner for its field, as in 10.1.0.

Trade-offs

Correct (as a control on main: the probe passes 5 of 5 with -ea, and the JUnit 3 of 3 with -ea and -da; the study's revert control passed its 6 tests and 29 upstream tests), but it brings back the per-field get(docID, field) calls that #7926 described as O(n^2) in the number of fields.

Let a later reader supply vectors when the owner has none (dsmiley, 2017), or reject overlapping field names (mikemccand, 2017).

Trade-offs

Both are behaviour changes for their own issue. The first also needs the merged FieldInfo to say storeTermVector=true; the second breaks TestDemoParallelLeafReader and users of the documented first-index rule. Neither fixes the regression alone.

Method: static analysis (CodeQL java/unused-container), the study's non-formal baseline tier, run 1, main 7cecf79523 (2026-10-21/22).

• An AI tool ran CodeQL over Lucene. The query flagged ParallelLeafReader.tvFieldToReader as a container that is written but never read. The attribution table records it as a blind find: the AI tool was not pointed at this class.
• A separate, independent AI verifier read the experiment's test only for its sequence and wrote its own 6 tests (leaf and composite reader, both orders, TestUtil.checkReader, addIndexes; 4 are deterministic and the 2 both-orders tests could pass by chance only in a rare hash-slot case) plus a probabilistic wrapForMerge test, all failing on main; a revert control and an owner-filter control made them pass, and 29 upstream tests passed in all three variants.
• On 2026-10-25 the hunt re-ran a minimal 3-test JUnit on ea57360912ed (-ea, -da, 25 iterations) and on 10.5.2, 10.2.0 and 10.1.0, built the release matrix and the commit-level A/B runs, searched for prior reports, and obtained a cross-model refute review. For this page, the fix was validated with gradle.