SortedDocValues#ordinalRangeIntoBitSet, called right after advanceExact returned false, marks a document that has no value

SortedDocValues#ordinalRangeIntoBitSet(fromDoc, toDoc, minOrd, maxOrd, bitSet, offset), added in 10.6.0 by #16180, fills bitSet with "the doc IDs in [fromDoc, toDoc) whose ordinals are in [minOrd, maxOrd]", and its javadoc says the default "falls back to per-doc evaluation via advanceExact and ordValue". The default instead starts at the current document whenever docID() >= fromDoc and calls ordValue() there (SortedDocValues.java:117-118 on main 645d4fe14a20). After advanceExact(d) returned false the iterator sits on d, which has no value, and ordValue() is documented as illegal in that state. The default codec's sparse SORTED iterator (two or more distinct values, some documents without one) has no override, and its ordValue() reads with the stale index that advanceExact left: in the tested layouts doc d is judged by the previous valued document's ordinal (in a DENSE block the next one's, or the padding after the last ordinal when none follows), or the call throws when no earlier document has a value (RuntimeException(EOFException), or ArrayIndexOutOfBoundsException on a plain ByteBuffersDirectory). Wrappers that inherit the default (FilterSortedDocValues, MultiDocValues) are affected even with one distinct value. With docs {0: "a", 1: none, 2: "b"}, advanceExact(1) then ordinalRangeIntoBitSet(1, 3, ord("a"), ord("a"), bits, 0) sets bit 1 instead of nothing, with assertions on and off. In this call pair only the start document can be wrong. No Lucene query, sort, facet or merge path makes this call order, and the method is new in 10.6.0 (released 2026-10-29), so the priority is low. Skipping a start document without a value (8 lines) fixes it; the test and 14 doc-values suites pass with the fix.

1. The cursor after a false advanceExact. "After this method returns, docID() returns target" (DocValuesIterator.java:24-28); the note below names only intoBitSet and docIDRunEnd() as illegal after a false return (:30-31).
2. The default (SortedDocValues.java:117-122):

   for (int doc = docID() >= fromDoc ? docID() : advance(fromDoc); doc < toDoc; doc = nextDoc()) {
     long ord = ordValue();
     if (ord >= minOrd && ord <= maxOrd) {
       bitSet.set(doc - offset);
     }
   }

With docID() == fromDoc it starts on that document and calls ordValue() (:118), which "is illegal to call ... after advanceExact(int) returned false" (:38-39).

1. Which iterator runs it. For ordinals stored as one packed block with more than one value (Lucene90DocValuesProducer.java:1473-1477), the dense branch overrides the method with random access (:1548-1555) and is correct; the sparse branch (:1557) has no override, and its ordValue() is (int) values.get(disi.index()) (:1569-1571). The generic iterator (one distinct value, or ords not in one block) delegates to the numeric rangeIntoBitSet (:1654-1661) and answers correctly when called directly. FilterSortedDocValues (so also ExitableDirectoryReader's wrapper), MultiDocValues.MultiSortedDocValues and AssertingSortedDocValues do not override the method, so wrapped iterators run the faulty default, over the generic iterator too: there ordValue() reaches ords.longValue(), which for one distinct value is the constant entry.minValue (:1090-1096), so a wrapper marks the value-less document whenever that ordinal is in range (AssertingSortedDocValues with -ea fails assert exists instead).
2. What the stale index holds (the tested layouts). After a miss in a SPARSE block, IndexedDISI steps the index back to the last valued document before the target, or leaves -1 if there is none (IndexedDISI.java:607-616); in a DENSE block the index is the rank of the next valued document (:712-713), which is one past the last stored ordinal when no valued document follows. Hence doc 0's ordinal in the example, the next document's ordinal in a DENSE block, the padding after the last ordinal (read as 0) when nothing follows, and values.get(-1), an exception, when nothing precedes. These are the executed layouts, not every IndexedDISI state: a target inside a 65,536-document block with no values (advanceBlock, :495-516), for example, was not run.
3. Only the start document. After it the loop moves with nextDoc(), which lands only on documents with a value, so later bits are right and no later valued document is dropped. Separate, pre-existing behaviour, not this bug: when docID() > fromDoc the default starts at docID() and never evaluates [fromDoc, docID()), valued documents included; the fixes below keep that (a forward-only iterator cannot go back).

• No in-tree caller reaches it. The only caller, DocValuesRangeIterator.BulkSortedRangeIterator#intoMaybeBlock, passes from = Math.max(blockStart, docID()) (DocValuesRangeIterator.java:513-519) on an iterator it moves with advance() only (advanceDisi, :296-301), so docID() always has a value there. An executed probe (8,400 query executions per run over 14 random skip-indexed indexes, raw, asserting and exitable readers, assertions on and off) found 0 mismatches while the default ran about 13,000 times per reader; the direct call pair on the same indexes misbehaved in 250 of 327 trials. Open PR #16897 adds a second caller, which also positions with advance().
• Who can hit it. Application code that probes a document with advanceExact and then bulk-evaluates from it; the test of open PR #16835 makes the analogous call pair on the numeric sibling. Default codec and configuration; any index size (three documents suffice).
• Effect. Silent wrong results: one extra document per call, judged by a neighbouring document's ordinal from the same field and segment, or an exception (RuntimeException(EOFException), or ArrayIndexOutOfBoundsException on a plain ByteBuffersDirectory). No hang, no resource use, no write.
• Not affected. Unpositioned iterators, iterators moved by nextDoc()/advance() or by an advanceExact that returned true; dense SORTED fields; the unwrapped generic iterator (one distinct value); the default implementations of NumericDocValues#rangeIntoBitSet and SortedNumericDocValues#rangeIntoBitSet, which are per-doc advanceExact loops. Not every codec override of those siblings is safe: the sparse multi-valued Lucene90 SortedNumericDocValues#rangeIntoBitSet override (Lucene90DocValuesProducer.java:2202-2255, start check :2210-2213) has the same flaw; that is LFV-415, classed known because open PR #16835 guards it for table-encoded entries.
• Releases. 10.6.0 (where #16180 is listed), released 2026-10-29 from the RC1 bits, which contain the bug. Priority low.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: doc 1 has no value but was marked
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertFalse(Assert.java:65)
       org.apache.lucene.index.TestSortedOrdinalRangeAfterFalseAdvanceExact.testDocWithoutValueIsNotMarked(TestSortedOrdinalRangeAfterFalseAdvanceExact.java:33)

• Contract. SortedDocValues.java:99-105 (the method's javadoc); :38-39 (ordValue()); DocValuesIterator.java:30-31 lists only intoBitSet and docIDRunEnd() as illegal after a false advanceExact. The method has no @lucene.internal or @lucene.experimental tag, while DocIdSetIterator#intoBitSet is @lucene.internal (DocIdSetIterator.java:203).
• Encoding-dependent answers. An executed consistency probe on one codec: a SORTED field with one distinct value (unwrapped) and a NumericDocValues field give the per-doc answer from the same state; a sparse SORTED field with two values marks the value-less document.
• The test framework flags it. AssertingSortedDocValues.ordValue asserts exists (AssertingLeafReader.java:1084), so the default trips Lucene's own assertion.
• History. The first revision of #16180 (f5a83a8d11a) was the per-doc advanceExact loop. A reviewer (jimczi) asked "Why advanceExact per-doc here instead of iterating with advance/nextDoc? This is really just a range scan over the docs that have a value, which is the same pattern as the default DocIdSetIterator.intoBitSet and the old BulkOrdinalRangeIterator." They gave a performance reason (per-doc advanceExact walks every gap document of a sparse field) and proposed today's loop; revision 03ccba9729b adopted it and left the javadoc unchanged. Nobody discussed a start state reached by a false advanceExact.
• Why tests missed it. Three tests reach the method, all through DocValuesRangeIterator, which moves the iterator with advance() and never leaves it on a false advanceExact. TestDocValuesRangeIteratorIntoBitSet#testOrdinalRangeIntoBitSetMatchesPerDocEvaluation and #testOrdinalIntoBitSetMatchesLinearScan index real documents that all have a value; TestDocValuesOrdinalRangeIterator#testSortedDocValuesIntoBitSetMatchesLinearScan runs the stock default on a mock iterator in which some documents have no value, but never from a false advanceExact.

Strongest case (the intended-behaviour check, the adjudication, the prior-report search): the default was modelled on DocIdSetIterator#intoBitSet on purpose: jimczi's #16180 review chose the scan over present documents so that sparse fields do not walk every gap document. In #14529 a maintainer (jpountz) wrote "It shouldn't be legal to call intoBitSet after advanceExact returns false", which became the note on advanceExact; that settled the rule for DocIdSetIterator#intoBitSet, not for every bulk method. So this call order could be called misuse, fixed by adding the method to that note; the method is also new in 10.6.0, no in-tree caller reaches the state, and the default already skips [fromDoc, docID()) when docID() > fromDoc, an implied precondition (the hunt's credibility audit).

Why it fails: intoBitSet is relative to docID(), while this method takes an explicit absolute fromDoc and its javadoc still describes the per-doc loop; nothing documents a precondition; the answer changes with the encoding; one layout throws a codec exception (EOFException, or ArrayIndexOutOfBoundsException on a plain ByteBuffersDirectory) rather than a clean argument check. Open PR #16835 (kkewwei, approved by sgup432, not merged) accommodates this call order in sibling code: for table-encoded entries it adds exactly this guard to the Lucene90 numeric and sorted-numeric overrides, and its test calls advanceExact(fromDoc) before the bulk call; the PR is framed as a speed-up and does not call the sequence a public API contract. Residual risk: maintainers may prefer to document a precondition; the stale javadoc and the codec exception would still need fixing then.

Two options were built and run in a scratch worktree of 645d4fe14a20; a documentation-only change is not recommended alone.

The finder's fix, with the guard written as doc < toDoc, the guard shape kkewwei uses in open PR #16835 (table-encoded numeric overrides). It keeps the scan over documents with a value that jimczi asked for in the #16180 review, and the existing acceptance of docID() > fromDoc; like the stock code, it then starts at docID() and does not evaluate [fromDoc, docID()).

-    for (int doc = docID() >= fromDoc ? docID() : advance(fromDoc); doc < toDoc; doc = nextDoc()) {
+    int doc = docID();
+    if (doc < fromDoc) {
+      doc = advance(fromDoc);
+    } else if (doc < toDoc && advanceExact(doc) == false) {
+      // positioned by advanceExact() on a doc without a value: skip it
+      doc = nextDoc();
+    }
+    for (; doc < toDoc; doc = nextDoc()) {
       long ord = ordValue();

Trade-offs

• One advanceExact(docID()) per call when the iterator is already at or past fromDoc (legal: target >= docID()); for the in-tree caller one cheap check per MAYBE block. No benchmark was run.
• Caveat to state (the adjudication, from the cross-model refute review): flush-time BufferedSortedDocValues and some merge views throw UnsupportedOperationException from advanceExact, but they also throw from advance(), so the stock default already fails on them when docID() < fromDoc; none of them is passed to this method in-tree.

What the checks validated

• Stock + test: fails with assertions on and off; + fix: passes with assertions on and off.
• + fix: 14 suites, 728 tests (65 skipped), 0 failures, assertions on and off: TestDocValuesRangeIteratorIntoBitSet, TestDocValuesRangeIterator, TestDocValuesOrdinalRangeIterator, TestDocValuesOrdinalSetIterator, TestDocValuesQueries, TestSkipBlockRangeIteratorBitSetOperations, TestBulkScorerConsistency, TestSortedSkipperScorerSupplier, TestSortedSetDocValuesSetQuery, the three TestLucene90DocValuesFormat* suites, TestMultiDocValues and the new test; :lucene:core:check -x test passes.
• With the patched file shadowed, the finder's 2 tests (incl. the EOFException layout) and the independent reproduction's 7 tests (random sparse layouts, the DENSE block, MMapDirectory) pass. Not run: the full test suite, nightly tests.
• The finder's own form of the fix, shadowed together with the numeric sibling's fix, took the harness class LEAD-B from 1,770 mismatches (1,350 on SORTED views, this bug; 420 on SORTED_NUMERIC views, LFV-415) to 0.

for (int doc = Math.max(fromDoc, docID()); doc < toDoc; doc++) { if (advanceExact(doc)) { ... } }, as in the first revision of #16180 (plus Math.max, so docID() > fromDoc stays legal; like the recommended fix, it then starts at docID()). Built and run: the test and the same 14 suites pass with assertions on.

Trade-offs

Matches the javadoc word for word and the numeric siblings, but visits every document in the range instead of only those with a value, the cost the #16180 review asked to avoid on sparse fields.

Add ordinalRangeIntoBitSet to the "illegal after advanceExact returned false" note on DocValuesIterator#advanceExact, as #14529 did for intoBitSet.

Trade-offs

Also needs the javadoc's "per-doc evaluation via advanceExact" sentence fixed, and leaves a silent extra hit or a codec exception for callers who miss the note, while the dense override and the unwrapped single-value path keep answering correctly.

Method: bounded-exhaustive translation validation (semi-formal), a hunt unit. The harness drives the stock jar through every legal move sequence of depth up to 3 over nextDoc, advance, advanceExact, intoBitSet, docIDRunEnd, intoArray, longValues, rangeIntoBitSet and ordinalRangeIntoBitSet on segments of up to 5 documents (skip index on and off), plus random sequences on 70,000-document segments covering SPARSE, DENSE and ALL blocks and two-cursor interleavings, for 26 doc-values and norms views, and compares each step with a fresh nextDoc() enumeration of the same view: about 990M sequences over two batches. Mismatch class LEAD-B (advanceExact(t) == false, then a bulk range call starting at t) had 1,770 mismatches in a small-scope run: 1,350 on SORTED views (this bug) and 420 on SORTED_NUMERIC views (the numeric sibling LFV-415); 0 with this fix and the sibling's fix shadowed. The SORTED mismatches came from the segment's own iterator, its merge instance, the ParallelLeafReader, FilterSortedDocValues, ExitableDirectoryReader and AssertingLeafReader views (that run recorded wrong bits there, not an assertion failure), and the multi-segment views. The finder then wrote a public-API JUnit on the stock main jars; this doc's test is a one-method version of it.