MultiDocValues numeric, binary and norms views throw NullPointerException from advance() after advanceExact() landed on a segment without the field

MultiDocValues.getNumericValues, getBinaryValues and getNormValues return one doc-values iterator over all segments of a reader. When advanceExact(target) lands in a segment that has no values for the field, it returns false (correct) and remembers that segment as a null per-segment iterator while marking the segment as loaded. A following advance(t) with t in the same segment skips its "load the segment" branch and calls currentValues.advance(...) on that null: NullPointerException at MultiDocValues.java:263 (numeric), :396 (binary) or :130 (norms). The same happens for advance(NO_MORE_DOCS) after advanceExact landed in a field-less last segment. The call sequence is legal under the DocValuesIterator and DocIdSetIterator contracts, and the sorted, sorted-set and sorted-numeric views of the same class return the right document because they substitute empty iterators for missing segments. IndexWriter hands the same views to a custom MergePolicy.OneMerge#reorder (9.9.0+), where an uncaught exception closes the writer. It came with advanceExact in 7.0.0 (LUCENE-7462, 2016) and is unchanged on main. The result is always a loud exception, never a wrong value, and no Lucene caller mixes the two calls. A three-line guard in each advance() fixes it.

• Merged view: the anonymous NumericDocValues / BinaryDocValues classes returned by MultiDocValues.getNumericValues, getBinaryValues and getNormValues when a reader has two or more segments. Each keeps currentLeaf, currentValues (the current segment's own iterator, or null) and nextLeaf (the next segment to load).
• Field-less segment: a segment in which no document has the field, so LeafReader#getNumericDocValues(field) returns null. Normal for optional or sparse fields, or a field added to the schema later.

1. Numeric advanceExact(target) (MultiDocValues.java:274-296) loads the target's segment if it is not loaded yet (readerIndex >= nextLeaf, :283): currentValues = currentLeaf.reader().getNumericDocValues(field) (:288), which is null for a field-less segment, and nextLeaf = readerIndex + 1 (:289). It sets docID = targetDocID (:291) and returns false (:292-293).
2. Numeric advance(target) (:241-271) reloads only when the target lies in a segment not loaded yet (if (readerIndex >= nextLeaf), :250); inside that branch it already handles a field-less segment (:259-261). For a target in the segment that advanceExact just loaded, readerIndex < nextLeaf, so it skips the branch and calls currentValues.advance(targetDocID - currentLeaf.docBase) (:263) on null.
3. ReaderUtil.subIndex never returns leaves.size() (ReaderUtil.java:74-94): a target >= maxDoc maps to the last segment. So after advanceExact lands in a field-less last segment, advance(NO_MORE_DOCS) throws too; the readerIndex == leaves.size() check at :251 cannot fire.
4. Binary (advance :374-404, NPE at :396; advanceExact :407-429) and norms (advance :108-138, NPE at :130; advanceExact :141-163) have the same code.
5. The sorted, sorted-set and sorted-numeric views fill an array of per-segment iterators up front with DocValues.empty*() for a field-less segment (sorted-numeric :463-473), so their currentValues is never null for a loaded segment, and they return the right document for the same calls.

Two segments, [doc 0, doc 1: no field] [doc 2: f = 42]: advanceExact(0) = false, then advance(1) throws NullPointerException at MultiDocValues$2.advance:263; expected 2. nextDoc() from the same state and the sorted-numeric view both give 2. The hunt's exhaustive harness found the NPE in 827 of the 70,662 call sequences of length up to 3 for each of the numeric, binary and norms views, and 0 mismatches for the sorted views.

• Who is hit. Code that holds a merged numeric, binary or norms view and calls advance(t) after advanceExact(e) returned false, with e and t in a segment without the field (or t >= maxDoc after landing in a field-less last segment). Two entry points: MultiDocValues.getNumericValues / getBinaryValues / getNormValues (public class, tagged @lucene.experimental and @lucene.internal), and, since 9.9.0, the merged CodecReader that IndexWriter passes to a custom MergePolicy.OneMerge#reorder (@lucene.experimental), whose norms, numeric and binary doc values are these views (SlowCompositeCodecReaderWrapper.java:291, :412, :417).
• No Lucene caller is affected. DocumentDictionary uses advance only; the infix suggesters open a fresh view per hit; Luke uses advanceExact only; the in-tree reorderers and merge consumers iterate.
• Ordinary data. Default codec and config; 2 segments and 3 documents are enough. A field-less segment is normal for optional or sparse fields.
• What happens. A NullPointerException at the first such advance(), with assertions on and off. No wrong value, no hang, nothing written.
• Inside reorder it closes the writer (executed on main and 10.5.2). If the exception escapes a reorder override, IndexWriter treats it as a tragic merge failure and closes: forceMerge(1) throws, a reopened writer retries the same merge and closes again, and with merge-on-commit (on by default) commit() fails and the documents added since the last commit are rolled back. The committed index stays intact (CheckIndex clean). With the fix the merge completes.
• How likely. Rare: code that mixes the two calls fails on ordinary sparse data, so it would most likely surface during development. No prior public report since 7.0.0 (2017).

The test sits in org.apache.lucene.index next to the upstream TestMultiDocValues but uses no package-private access. Its result does not depend on the seed (it uses no randomness the outcome depends on; the test framework's random codec does not matter).

Not shown by this one test, all from the same null currentValues and executed elsewhere: the binary and norms views, the public OneMerge#reorder path (9.9.0+), and advance(NO_MORE_DOCS) after advanceExact lands in a field-less last segment.

As the regression test: the same test unchanged, for example as a new method of TestMultiDocValues. It fails on stock main and passes with fix A or B, with assertions on and off (gradle). :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs, licenses) passes with fix A and this test in place.

cp TestMultiDocValuesAdvanceAfterAdvanceExact.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestMultiDocValuesAdvanceAfterAdvanceExact -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false instead (same result)

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (the same with =false, and on 10.6.0)
TestMultiDocValuesAdvanceAfterAdvanceExact > testAdvanceAfterAdvanceExactOnSegmentWithoutField FAILED
    java.lang.AssertionError: expected advance(1) == 2, got java.lang.NullPointerException: Cannot invoke "org.apache.lucene.index.NumericDocValues.advance(int)" because "this.currentValues" is null
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:75B1A4024E2D61B9]:0)
        at org.apache.lucene.index.TestMultiDocValuesAdvanceAfterAdvanceExact.testAdvanceAfterAdvanceExactOnSegmentWithoutField(TestMultiDocValuesAdvanceAfterAdvanceExact.java:45)

        Caused by:
        java.lang.NullPointerException: Cannot invoke "org.apache.lucene.index.NumericDocValues.advance(int)" because "this.currentValues" is null
            at org.apache.lucene.index.MultiDocValues$2.advance(MultiDocValues.java:263)
            at org.apache.lucene.index.TestMultiDocValuesAdvanceAfterAdvanceExact.testAdvanceAfterAdvanceExactOnSegmentWithoutField(TestMultiDocValuesAdvanceAfterAdvanceExact.java:42)

1 test completed, 1 failed

• Contract. DocValuesIterator#advanceExact (DocValuesIterator.java:24-33): "After this method returns, docID() returns target"; the only calls it declares illegal after a false return are intoBitSet and docIDRunEnd (:30-31). DocIdSetIterator#advance (DocIdSetIterator.java:137-163) is undefined only for target <= current or after exhaustion; for target > current it "behaves as if written" as a nextDoc() loop (:144-153), and it "may be called with NO_MORE_DOCS" (:157-159). The repro obeys both. Nothing in MultiDocValues' javadoc narrows this.
• Same class, right answer. The sorted, sorted-set and sorted-numeric views return 2 for the same calls, and so do single-segment codec iterators. The numeric view's own reload branch handles a field-less segment (:259-261), so field-less segments are meant to work.
• Tests treat the mix as legal. The test framework's AssertingNumericDocValues.advance asserts only target > in.docID() (AssertingLeafReader.java:791-801). Upstream TestIndexedDISI calls nextDoc() right after a false advanceExact() (:595-600). Upstream TestMultiDocValues never mixes the two on one iterator (testRandomAdvance :424-437, testRandomAdvanceExact :439-448 run on separate iterators), and its numeric and binary tests put the field in every document, which is why no test caught it.
• History. advance() came from LUCENE-7407 (f7aa200d406, 2016-09-21), when a loaded segment always had a non-null iterator. A month later LUCENE-7462 (9aca4c9d560) added advanceExact(), which can leave currentValues null for a loaded segment. git log -G 'currentValues == null' lists only these two commits; no javadoc, CHANGES.txt or MIGRATE.md entry, test or discussion restricts the mix.
• A committer has seen the null state, in a different, illegal call: in the 2020 review of apache/lucene-solr#1733 (LUCENE-9450), mikemccand traced an NPE from binaryValue() to currentValues being null "since .advanceExact was not yet called" (a value accessor on an unpositioned iterator, misuse) and suggested an assert. Not a report of this bug.

The strongest case for "intended" (the refute-lens cross-model reviewer and the devil's advocate): MultiDocValues is @lucene.experimental and @lucene.internal, its javadoc says "This is very costly" and steers users to per-leaf access, and no Lucene caller mixes the two calls, so the sequence is out of contract or not worth fixing.

Why it fails: @lucene.internal is an API-stability label ("might change in incompatible ways in the next release", RenderJavadocPlugin.java:506), not permission to throw on a sequence the returned type's own contract allows. The same iterators also reach the public experimental OneMerge#reorder hook as ordinary NumericDocValues / BinaryDocValues of a CodecReader, with nothing that warns the hook's author. The tag lowers priority, not validity. Verdict: not intended; a low-priority robustness bug. Residual risk: a maintainer may answer "internal API, use per-leaf access"; the reorder path answers that.

Fix A and fix B were run through gradle in a scratch worktree of 645d4fe14a20. A is recommended: the smallest, most direct change, and the one every checker ran. With the fix for the sibling bug LFV-142 as well, the hunt's exhaustive harness reports no mismatches at all.

In advance() of the numeric (:250-263), binary (:383-396) and norms (:117-130) views. Numeric shown; the other two are identical.

         if (readerIndex >= nextLeaf) {
           ...
           if (currentValues == null) {
             return nextDoc();
           }
+        } else if (currentValues == null) {
+          // positioned by advanceExact on a leaf that has no values for this field
+          return nextDoc();
         }
         int newDocID = currentValues.advance(targetDocID - currentLeaf.docBase);

Trade-offs

• Correct by construction: currentValues == null with a target in the current segment means advanceExact found no values there, so the next document with a value is in a later segment, where nextDoc() starts (nextLeaf).
• Covers advance(NO_MORE_DOCS) on a field-less last segment too (nextDoc() returns NO_MORE_DOCS).
• No change for any call sequence that works today; one null check per advance() call.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (2026-11-01): the repro fails on stock main and passes with A, each with -Ptests.asserts=true and =false.
• With A: 18 core test classes, 273 tests, 0 failures, 5 skipped (all @Nightly): the repro, TestMultiDocValues, TestNorms, TestCustomNorms, TestDocValuesIndexing, TestSegmentReader, TestDirectoryReader, TestIndexSorting, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestDemoParallelLeafReader, TestMaxTermFrequency, TestUniqueTermCount, TestIndexingSequenceNumbers, TestSimilarityProvider (the core tests that read these views), TestSortingCodecReader, TestMergePolicy and TestAddIndexes.
• With A: the whole suggest module (the in-tree production caller of these views), 31 classes, 236 tests, 0 failures, 7 skipped; :lucene:core:check -x test passes with A and the test in place. The full core suite was not run.
• Earlier, the same fix: the hunt's 5-test class 5/0 and the public OneMerge#reorder test 1/0; an independent 7-test class 7/0; upstream TestMultiDocValues 7/7; in the reorder probe the merge completes and the writer stays open.

An independent verifier's alternative: if (readerIndex >= nextLeaf || currentValues == null) { in each advance().

-        if (readerIndex >= nextLeaf) {
+        if (readerIndex >= nextLeaf || currentValues == null) {

Trade-offs

One condition per view, but it asks the segment for its doc values again (one extra getNumericDocValues / getBinaryDocValues / getNormValues call), which again returns null and falls into the existing return nextDoc(). Reads less directly than A.

What the checks validated

Gradle, same worktree: the repro passes with B, with assertions on and off; the independent verifier's 7-test class passes 7/0.

Use DocValues.emptyNumeric() / DocValues.emptyBinary() for a field-less segment, like the sorted views (:463-473).

Trade-offs

Removes the null state from all methods at once, but touches nextDoc(), advance() and advanceExact() of three views: a larger diff. Not tested.

Method: semi-formal (bounded-exhaustive translation validation). A hunt unit auditing the callers of ReaderUtil.subIndex wrote a harness that builds every small index layout (which documents of which segment have a value) and checks each MultiDocValues view, on the stock jar, against the concatenation of the segments' own iterators, over every legal nextDoc / advance / advanceExact sequence: length up to 3 on 1-3 segments (258 layouts, 70,662 sequences per view, 423,972 sequence checks over the six views), and length up to 4 on up to 2 segments (42 layouts, 10,790 sequences per view, 64,740 checks).

It reported a NullPointerException bucket for the numeric, binary and norms views (827 sequences each in the run of length up to 3) with the minimal witness: segments [(00)(1)], calls [advanceExact(0), advance(1)]. The sorted, sorted-set and sorted-numeric views had 0 mismatches. A run past maxDoc added the advance(NO_MORE_DOCS) case. The same harness found the sibling norms bug LFV-142; with both fixes it reports no mismatches.

The finding was turned into JUnit tests (direct and through OneMerge#reorder), reviewed by a reviewer from another model family (OpenAI GPT-5.6) in three sessions, reproduced independently, judged, and cut to the one-test repro on 645d4fe14a20.