MultiDocValues norms view (also the merged reader passed to OneMerge#reorder): advance(target) can return a document before target

MultiDocValues.getNormValues(reader, field) (@lucene.experimental, @lucene.internal) presents the norms of a multi-segment reader as one NumericDocValues; the same view is the norms of the merged CodecReader that IndexWriter passes to MergePolicy.OneMerge#reorder (@lucene.experimental, since 9.9.0). The view keeps nextLeaf, the segment its nextDoc() reads next. advance(target) jumps to the target's segment, and when that segment has no norms for the field it returns nextDoc() before it sets nextLeaf = readerIndex + 1. nextDoc() then resumes from the stale nextLeaf (segment 0 for a fresh iterator) and, when a segment from there up to the target's holds a document with a norm, returns that document, below the target, against the DocIdSetIterator#advance contract. Three documents in three segments suffice: advance(1) returns 0 instead of 2, with the default codec and with assertions on or off. The numeric and binary views, written in the same 2016 commit, set nextLeaf first, so only the norms view is affected. Lucene's own search works per segment and its merges read these norms with nextDoc() only, so only application code that calls advance() on the view, or searches it, is hit. The fix moves one line, to the order the sibling views use; the test and 11 suites around the view pass with it.

1. The view. getNormValues returns the segment's own iterator for a single-segment reader (MultiDocValues.java:53-55) and null when no segment has norms for the field (:57-69); otherwise an anonymous view with nextLeaf = 0, currentValues, currentLeaf and docID = -1 (:71-75).
2. nextDoc() (:77-100): while currentValues is null it loads leaves.get(nextLeaf) and increments nextLeaf (:80-88); a segment without norms yields null and is skipped.
3. advance(target) (:108-138):

   int readerIndex = ReaderUtil.subIndex(targetDocID, leaves);   // :116
   if (readerIndex >= nextLeaf) {                                 // :117
     ...
     currentLeaf = leaves.get(readerIndex);                       // :123
     currentValues = currentLeaf.reader().getNormValues(field);   // :124
     if (currentValues == null) {
       return nextDoc();                                          // :126, nextLeaf not yet updated
     }
     nextLeaf = readerIndex + 1;                                  // :128
   }

A segment in which no document has the field has no FieldInfo for it, so CodecReader#getNormValues returns null (CodecReader.java:213-217), which LeafReader#getNormValues allows ("or null if no NumericDocValues were indexed", LeafReader.java:202-206). It also returns null when the field's FieldInfo has no norms (not indexed, or norms omitted); within one index IndexWriter's global field numbers keep a field's index options and omitNorms the same in every segment (FieldInfos.java:511-521), so there the trigger is a segment in which no document has the field. Direct MultiDocValues use over an arbitrary composite reader (for example leaves of separately built indexes, or wrapped leaves) has no such guarantee: there a segment's norms can also be null because the field is present but omits norms or is not indexed, and any such segment is a trigger (code reading; not run).

1. The siblings. The numeric view sets nextLeaf = readerIndex + 1; (:258) before if (currentValues == null) return nextDoc(); (:259-261), the binary view too (:391, then :392-394), and so does the norms view's own advanceExact() (:156, then :159). The sorted views substitute empty iterators for segments without the field (:467, :613, :659).
2. The merged reader. IndexWriter wraps the segments of a merge (IndexWriter.java:5557-5558) or of addIndexes(CodecReader...) (:3561-3562) in SlowCompositeCodecReaderWrapper and passes it to OneMerge#reorder; its norms producer returns MultiDocValues.getNormValues(...) (SlowCompositeCodecReaderWrapper.java:289-292).

Fresh view, nextLeaf = 0. advance(1): subIndex(1) = 1 (segment 1), and 1 >= 0, so segment 1 is loaded; its getNormValues("body") is null, so the method returns nextDoc() with nextLeaf still 0. nextDoc() loads segment 0, sets nextLeaf = 1 and returns doc 0. Expected 2.

The target's segment has null norms and lies strictly after the stale nextLeaf, and some segment from nextLeaf up to the target's holds a document with a norm; on a fresh iterator (nextLeaf = 0) that means an earlier segment has the field. When the target's segment is nextLeaf itself, or no segment from nextLeaf up to it has a norm, nextDoc() skips ahead and the result is right: [(T)(F)(T)] nextDoc() = 0 then advance(1) = 2; [(F)(F)(T)] fresh advance(1) = 2; [(T)(F)(F)(T)] nextDoc() = 0 then advance(2) = 3 (fact-check probe).

• Target in a last segment without norms: [doc 0: body] [doc 1: none], advance(1) returns 0 instead of NO_MORE_DOCS; the same for a target at or past maxDoc.
• Merged reader inside OneMerge#reorder, [doc 0: body] [docs 1-2: none] [doc 3: body]: reader.getNormValues("body") gives advance(1) = 0 and advance(2) = 0, each on a fresh iterator, expected 3 (main and 10.6.0).
• A fresh iterator can also land inside an earlier segment, not only at doc 0: [(F,T)(F)(F,T)] (docs 1 and 4 have the field), advance(2) returns 1, expected 4.
• After earlier calls: [(T)(T)(F)(T)], nextDoc() = 0 and then advance(2) returns 1, expected 3; [(F)(T)(F)], advanceExact(0) and then advance(2) returns 1, expected NO_MORE_DOCS.
• Not affected: the numeric, binary and sorted views (0 such mismatches in the bounded-exhaustive check).

• Who is exposed. Direct callers of MultiDocValues.getNormValues (its javadoc: "This is a slow way to access normalization values"); custom MergePolicy.OneMerge#reorder implementations that read the merged reader's norms with advance() or run queries over it; downstream single-leaf wrappers that delegate to it, such as Solr's SlowCompositeReaderWrapper#getNormValues (apache/solr main 9ba1a7d42e4b, lines 323-326; code reading only; no Solr request path that searches it was found; Elasticsearch and OpenSearch not checked).
• Trigger. The default codec, no special settings, and an ordinary sparse text field: on a fresh iterator, a target in a segment in which no document has the field, after one that has it (the general condition is under "When exactly" above). Deterministic.
• Effect. advance() can return a document below the target, silently. Code that searches the view inherits the step back. FieldExistsQuery uses the norms iterator as its own iterator, and with intra-segment partitions DefaultBulkScorer advances it to each partition start, so earlier documents are collected again: [0, 0, 0, 3] instead of [0, 3]. The view reports cost() = 0 and ConjunctionBulkScorer orders its iterators by cost, so the view can become the lead (it does in the executed witness); a lead that steps back can make a conjunction accept a document on which another required clause was never re-checked (with assertions off; with assertions on, Lucene's conjunction assertions fire). Some query shapes throw ArrayIndexOutOfBoundsException or IllegalArgumentException at once. No hang or index corruption was observed.
• Not affected. Lucene's own search, which works per segment (the probe's per-segment control was exact in all 3,667 layouts); Lucene's merges, which read the merged view through SortingCodecReader, which copies norms with nextDoc() only; Luke (advanceExact); the in-tree reorderers, which read no norms (BPReorderingMergePolicy: postings and vectors; the test-only MockRandomMergePolicy: the parent-field doc values); the test framework's comparisons (nextDoc()).
• Scale. Not a scale issue: three documents suffice. The defect is in the view's segment bookkeeping, which does not depend on segment size (code reading); no size-dependent behaviour was observed in the runs.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<2> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.index.TestMultiDocValuesNormsAdvance.testNormsAdvanceIntoSegmentWithoutNorms(TestMultiDocValuesNormsAdvance.java:46)

• Contract. "Advances to the first beyond the current whose document number is greater than or equal to target, and returns the document number itself. Exhausts the iterator and returns NO_MORE_DOCS if target is greater than the highest document number in the set" (DocIdSetIterator.java:137-139). NumericDocValues inherits it.
• Legal trigger. A fresh iterator, target > docID() and < maxDoc, the default codec, otherwise default settings except NoMergePolicy (which only keeps the segments apart), and a segment without the field, which LeafReader#getNormValues allows ("or null").
• Test framework. AssertingLeafReader#getNormValues wraps every norms iterator in AssertingNumericDocValues (AssertingLeafReader.java:1838-1845), whose advance() asserts docID >= target (:790-797). The repro's reader is not wrapped, so the test sees the raw value.
• History. In the introducing commit (LUCENE-7407) the numeric and binary views put nextLeaf = readerIndex+1; before the null check and the norms view after it; nothing in that issue's discussion (42 comments in the JIRA record, 41 in the migrated GitHub issue #8460), the javadoc, CHANGES.txt, MIGRATE.md or later commits calls the order a choice.
• Why tests missed it. TestMultiDocValues checks advance() on the numeric, binary and sorted views only; no upstream test calls advance() on MultiDocValues.getNormValues.
• Precedent. LUCENE-7529 (#8580, 2016) fixed another iterator-contract violation in these views (the advanceExact argument check).

Strongest case (the refute-lens reviewer, the intended-behaviour checker and the credibility audit): MultiDocValues is @lucene.internal and @lucene.experimental and calls itself "very costly"; the merged wrapper says "This view is primarily targeted at merging, not searching"; no Lucene search or merge path calls advance() on it. So this is an internal view used outside its purpose.

Why it fails: the build defines @lucene.internal as "This API is for internal purposes only and might change in incompatible ways in the next release", an API-stability note, not a licence to break DocIdSetIterator; the merged reader is a public CodecReader handed to user code by a public hook whose javadoc puts no limit on how it is read, and advance() on norms is a plain reader operation, not a search; the sibling views show the intended order; and the fix moves one line. What the argument does establish, and strongly, is narrow reach, which is why the bug is rated minor and low priority.

One fix was built and run in a scratch worktree of 645d4fe14a20. Two broader options are listed: fixing the sibling LFV-136 in the same change, and replacing the null per-segment iterators.

In MultiDocValues.getNormValues(...)'s advance(), record that the segment just loaded is consumed before falling back to nextDoc() (one line moved, MultiDocValues.java:128 to above :125). The finder and an independent verifier wrote the same change.

           currentLeaf = leaves.get(readerIndex);
           currentValues = currentLeaf.reader().getNormValues(field);
+          nextLeaf = readerIndex + 1;
           if (currentValues == null) {
             return nextDoc();
           }
-          nextLeaf = readerIndex + 1;
         }

Trade-offs

• The order of the numeric and binary views and of the norms view's own advanceExact(); the segment just loaded is consumed, so nextDoc() must continue with the next one.
• No material cost expected (a line moved); no benchmark was run.
• Does not fix the sibling LFV-136 (a NullPointerException from advance() after advanceExact()).

What the checks validated

• Stock + test: fails through gradle with assertions on and off (expected:<2> but was:<0>); + fix: passes.
• + fix: 11 suites (the test, TestMultiDocValues, the seven lucene/core test classes that call MultiDocValues.getNormValues, TestSortingCodecReader, TestAddIndexes): 97 tests, 0 failures, 0 skipped, with assertions on and off, in two separate gradle runs (the second on 2026-10-30, which also showed the test failing again once the fix was removed); :lucene:core:check -x test passes.
• In the hunt: with this fix the bounded-exhaustive check loses exactly the two norms advance-return buckets, and the verifiers', the cross-model reviewers' and the probe's tests pass.
• Not run: the full Lucene test suite, the nightly tests, any benchmark, and the other direct callers: lucene/memory's TestMemoryIndexAgainstDirectory, the norms-format tests built on BaseNormsFormatTestCase and the test framework's own comparisons (LuceneTestCaseParent), which read the view with nextDoc() only, and Luke's DocumentField, which uses advanceExact() on a fresh view.

In the numeric, binary and norms views, add an else if (currentValues == null) return nextDoc(); branch to the if (readerIndex >= nextLeaf) block, for a target in the segment that advanceExact() left without values (+3 lines per view).

Trade-offs

Fixes both iterator bugs of these views in one PR. Each fix alone removes only its own mismatches; with both, the bounded-exhaustive check finds none.

What the checks validated

• Bounded-exhaustive check with both changes in all three views: no mismatches at sequence length up to 3 on 1-3 segments and up to 4 on 2 segments.
• Not run through gradle here; LFV-136 has its own repro.

As the sorted views do, use an empty per-segment iterator instead of null.

Trade-offs

Would remove the null branch that both this bug and LFV-136 depend on, but touches nextDoc(), advance() and advanceExact() of three views. Not built or run.

Method: semi-formal (a code-reading hypothesis witnessed by bounded-exhaustive translation validation), a hunt unit (callers of ReaderUtil.subIndex), 2026-10-26. The harness compares every MultiDocValues view with the concatenation of the per-segment iterators over every legal nextDoc/advance/advanceExact sequence: length up to 3 on 1-3 segments (258 index layouts, 423,972 sequences) and length up to 4 on up to 2 segments (64,740 sequences), on the stock jar with -ea and -da. Norms advance-return mismatches: 732 (pure) and 504 (mixed) at length 3; minimal witness segments [(1)(0)], advance(1) returns 0 instead of NO_MORE_DOCS. The numeric, binary and sorted views had no advance-return mismatch. The same harness found the sibling LFV-136 (a NullPointerException bucket).