Flush-time doc-values iterators move only their doc cursor in intoBitSet(), so values read afterwards belong to earlier documents

When a segment is flushed, IndexWriter hands the codec's DocValuesConsumer one iterator per doc-values field. Each of the five flush-time iterators (Buffered{Numeric,Binary,Sorted,SortedNumeric,SortedSet}DocValues) has a doc cursor and a sequential value cursor, and only nextDoc() moves the value cursor from one document to the next. Since #16126 (10.5.0) their intoBitSet() delegates to the doc cursor alone, so after nextDoc() and intoBitSet(2, bits, 0) over values 10, 11, 12 the iterator reports doc 2 with doc 0's value 10, and every later value stays shifted by the same two documents. That breaks DocIdSetIterator#intoBitSet's rule that an override "should behave exactly as" the nextDoc() loop. It is latent: Lucene's own consumer calls intoBitSet() only on fresh iterators whose values it never reads, so stock indexes are correct, and only an application-installed custom doc-values codec that mixes the two calls on one iterator could write values against the wrong documents, usually silently. 10.5.0 through 10.6.0 and main are affected; removing the five overrides fixes it.

• Flush iterators. For each doc-values field of an unsorted segment, the writer's flush passes a producer whose getNumeric (getBinary, getSorted, getSortedNumeric, getSortedSet) returns a new Buffered*DocValues iterator over the in-memory buffers (NumericDocValuesWriter.java:134-172 at 645d4fe14a20). Two routes add a singleton wrapper: a SORTED_NUMERIC field whose documents all have one value is handed over as DocValues.singleton(BufferedNumericDocValues) (SortedNumericDocValuesWriter.java:215-225) and shows the same bug; a single-valued SORTED_SET field as DocValues.singleton(BufferedSortedDocValues) (SortedSetDocValuesWriter.java:274-285), where the wrapper adds its own stale ordinal (LFV-091). With an index sort and a flush that must re-sort, the Sorting* views are used instead; they read values by doc ID and are not affected.
• Two cursors. docsWithField is DocIdSetIterator.all(n) while every document so far has the field (as in the example), otherwise a BitSetIterator; the values come from a PackedLongValues.Iterator (or a lengths iterator plus a bytes input). Only nextDoc() moves the value side from one document to the next (NumericDocValuesWriter.java:191-198: value = iter.next()); in a multi-valued document nextValue() and nextOrd() also step through that document's values.
• intoBitSet contract. "This should behave exactly as if implemented as below, which is the default implementation: for (int doc = docID(); doc < upTo; doc = nextDoc()) { bitSet.set(doc - offset); }" (DocIdSetIterator.java:186-210). NumericDocValues#longValue "Returns the numeric value for the current document ID".

1. #16126 (10.5.0) added the same override to all five iterators: intoBitSet(...) { docsWithField.intoBitSet(upTo, bitSet, offset); } (NumericDocValuesWriter.java:215-218, BinaryDocValuesWriter.java:213-216, SortedDocValuesWriter.java:347-350, SortedNumericDocValuesWriter.java:318-321, SortedSetDocValuesWriter.java:385-388).
2. docsWithField.intoBitSet (DocIdSetIterator.all(n)'s range iterator in the example, a BitSetIterator for a sparse field) sets the range of bits and advances the doc cursor to upTo; nothing touches the value cursor.
3. So the doc cursor is ahead of the value cursor by the number of nextDoc() steps the reference loop would have taken: the documents with the field that intoBitSet moved over, counting the one it lands on. Every value read afterwards belongs to that many documents earlier. In this call order (nextDoc(), intoBitSet(), then reads) the value cursor only lags, nothing throws, and the values of the last documents are simply never returned.
4. Not always silent: intoBitSet() also leaves the per-document count and position alone. If a SORTED_SET consumer has already read the current document's ords, the next nextOrd() reads past BufferedSortedSetDocValues's per-document array: with 2 values per document, nextDoc(), two nextOrd(), intoBitSet(1, ...), nextOrd() throws ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2 (executed on 645d4fe14a20, assertions on and off; returns doc 1's ords with the fix).

The hunt's five-type test (6 documents): field values are 10 + d (or "t" + d) on doc d; after nextDoc() and intoBitSet(2, bits, 0) the bits (0 and 1) and docID() (2) are right in every case. Norms are not affected (NormValuesWriter.BufferedNorms has no override).

• Who is hit. Authors of a custom DocValuesFormat (the codec extension point; DocValuesConsumer is @lucene.experimental) whose consumer calls the @lucene.internal intoBitSet() on a flush-time iterator and then reads values from the same iterator, for example a single pass that writes the docs-with-field bit set and the values together. A custom IndexSorter reads these iterators at flush too and could hit it the same way (code reading only, not executed). Beyond installing the custom codec (IndexWriterConfig.setCodec), no setting is needed.
• Not hit: stock Lucene. Lucene90DocValuesConsumer calls intoBitSet() only through IndexedDISI.writeBitSet, always on a fresh iterator used only for doc IDs (Lucene90DocValuesConsumer.java:166-174, :472-475, :658-661), and reads values from other fresh iterators with nextDoc(). Soft-delete counting uses nextDoc() only. Norms have no override.
• Usually silent. Bits and doc IDs are right; only values (and, for SORTED_NUMERIC and SORTED_SET, value counts) are wrong. The one exception found: a SORTED_SET consumer that read the current document's ords before intoBitSet() gets an ArrayIndexOutOfBoundsException from its next nextOrd(). The format test for intoBitSet (BaseDocValuesFormatTestCase#doTestRandomIntoBitSet) runs on a merged segment's reader and checks only docID() and the bits, so no test reads flush-iterator values after intoBitSet().
• Scale. Every flushed segment with the field; deterministic; 3 documents are enough.
• How likely. Rare: no in-tree consumer has the pattern and nobody has reported it since 10.5.0. The pattern does exist downstream: one columnar codec folds a value pass into IndexedDISI.writeBitSet and keeps the default intoBitSet() loop on purpose, documenting that delegating it "would advance the underlying cursor behind this iterator's back".

Public Java API only (in the visibility sense; FixedBitSet and intoBitSet are @lucene.internal): IndexWriter, IndexWriterConfig, the codec SPI (DocValuesFormat, DocValuesConsumer, DocValuesProducer, FieldInfo, SegmentWriteState, SegmentReadState), NumericDocValues and FixedBitSet; from the test framework LuceneTestCase and TestUtil.getDefaultDocValuesFormat() / TestUtil.alwaysDocValuesFormat(...). The package org.apache.lucene.index is only a convention. The observing format delegates to the default one, so the segment it writes is an ordinary default-format segment (the test does not reopen it or run CheckIndex), and the assertion runs after the writer is closed, so a failure is a plain ComparisonFailure. It passes when doc 2 is current with value 12, which is what the fix gives.

It covers NUMERIC only; the other four iterators have the same one-line override. The hunt's five-type test and an independent reproducer's dense/sparse suite showed all five fail the same way, but they are not part of the repro.

The test is lint-clean as is: with it in lucene/core/src/test, checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass.

# gradle, scratch worktree of main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (the same with asserts=false)
TestFlushNumericDocValuesIntoBitSet > testValueAfterIntoBitSet FAILED
    org.junit.ComparisonFailure: expected:<doc=2 value=1[2]> but was:<doc=2 value=1[0]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1AC0B0C3115D08D5]:0)
        at app//org.junit.Assert.assertEquals(Assert.java:117)
        at app//org.junit.Assert.assertEquals(Assert.java:146)
        at app//org.apache.lucene.index.TestFlushNumericDocValuesIntoBitSet.testValueAfterIntoBitSet(TestFlushNumericDocValuesIntoBitSet.java:101)
1 test completed, 1 failed

# standalone, same seed: the same message on main (-ea, -da), 10.6.0 (-ea, -da), 10.5.2, 10.5.0; 10.4.0 (-ea, -da) passes

# the hunt's five-type test (6 documents; main ea57360912ed, the same on 645d4fe14a20), NUMERIC case: the shift persists to doc 3
expected:<bits=2 doc=2 value=1[2 next doc=3 value=13]> but was:<bits=2 doc=2 value=1[0 next doc=3 value=11]>

• The contract is explicit. DocIdSetIterator.java:186-204: an override "should behave exactly as if implemented as" the docID()/nextDoc() loop. These overrides reproduce the loop's doc IDs and bits but not its effect on the values. NumericDocValues#longValue "Returns the numeric value for the current document ID" (NumericDocValues.java:31-37); the documented restrictions on intoBitSet are that offset is at most the current doc ID and the iterator is positioned (DocIdSetIterator.java:196-197) and, for doc-values iterators, not after advanceExact returned false (DocValuesIterator.java:30-31); the repro meets all three, and none says values are undefined afterwards. The argument is an inference: the javadoc gives the reference loop and does not separately say that a subclass's value state must follow it.
• A regression. #16126 ("Implement #intoBitSet and #docIDRunEnd in DocValuesWriter implementations", iverase, merged 2026-05-26, milestone 10.5.0) says only: "This are mostly iterators wrapping other iterators so we just delegate the method to the underlaying iterators." One approving review, no comments, no tests for these iterators; CHANGES lists it under Optimizations. From 10.2.0 (#14069) to 10.4.0 the inherited loop kept the values right.
• Precedent. #16714 (dweiss, merged 2026-09-25, 10.6.0) added the missing setOrd() after in.intoBitSet in the four SortedSetSelector wrappers that #16126 added in the same commit: "it also did find a bug in there so it's piggybacked". It did not touch the writers. Analogous, not identical: there a wrapper's cached ordinal was not refreshed; here a direct iterator's value cursor is not advanced.
• Lucene's other iterators keep their value state coherent. The reader side resets its per-document state in intoBitSet (Lucene90DocValuesProducer.java:2258-2260, set = false), and the test framework's AssertingNumericDocValues leaves longValue() legal after intoBitSet (AssertingLeafReader.java:815-823, :846-850).
• Nothing permits it. No javadoc, CHANGES or MIGRATE entry, test or PR discussion says values are undefined after intoBitSet.

The strongest case (cross-model reviews and the devil's-advocate verifier): intoBitSet is @lucene.internal, the codec API @lucene.experimental; the Buffered* iterators are package-private and sequential-only (advance and advanceExact throw); DocValuesConsumer's javadoc lets an implementation "iterate over the values multiple times" (DocValuesConsumer.java:60-64), and the in-tree consumer does exactly that, one fresh iterator per pass. So mixing intoBitSet() and value reads on one iterator is unsupported use of internal API, and the overrides are a fair optimisation for every real caller. A maintainer may also read @lucene.internal as limiting behavioural guarantees, not only compatibility; the cross-model fact check rated this case moderate.

Why it fails: @lucene.internal is about API stability, not a licence for an iterator to report doc 2 with doc 0's value; the method's own javadoc states what an override must preserve. The project fixed an analogous omission in the SortedSetSelector wrappers from the same PR as a bug (#16714). And the error is usually silent: the doc IDs are right, so misplaced values surface nowhere. What survives is the severity: latent and custom-codec only. Verdict: not intended, a minor latent defect.

One fix was built and validated; the others are sketches with their trade-offs.

Delete intoBitSet from BufferedNumericDocValues, BufferedBinaryDocValues, BufferedSortedDocValues, BufferedSortedNumericDocValues and BufferedSortedSetDocValues, so they inherit the default nextDoc() loop, which moves both cursors; drop the four FixedBitSet imports that become unused (5 files, +5/-24 lines with a one-line comment in each class). Keep the docIDRunEnd overrides: they do not move any cursor.

// NumericDocValuesWriter.BufferedNumericDocValues; the same in the other four Buffered* classes
-    @Override
-    public void intoBitSet(int upTo, FixedBitSet bitSet, int offset) throws IOException {
-      docsWithField.intoBitSet(upTo, bitSet, offset);
-    }
+    // No intoBitSet override: the default nextDoc() loop also advances the value iterator.

Trade-offs

• Gives back #16126's speed-up where the default codec calls IndexedDISI.writeBitSet on these iterators at flush (fields that some but not all documents have): the loop now also decodes each value it passes. Not measured.
• Restores the 10.2.0-10.4.0 intoBitSet behaviour (the default loop), which was correct; the docIDRunEnd overrides, which 10.4.0 did not have, stay.
• A single-valued SORTED_SET field flushes as DocValues.singleton(BufferedSortedDocValues); there SingletonSortedSetDocValues.intoBitSet still leaves a one-document stale ordinal (LFV-091, a separate one-line fix).

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (--offline, seed DEADBEEF, JDK 25.0.3). Stock: the repro fails with assertions on and off.
• With the fix: the repro plus TestLucene90DocValuesFormat, TestLucene90DocValuesFormatMergeInstance, TestLucene90DocValuesFormatVariableSkipInterval, TestIndexedDISI, TestPerFieldDocValuesFormat, TestDocValuesIndexing, TestDocValues, TestDocsWithFieldSet, TestIndexSorting, TestMultiDocValues and the four doc-values update suites pass with assertions on and off (15 suites, 884 tests, 0 failures, 69 skipped).
• Every core test in org.apache.lucene.index.* and org.apache.lucene.codecs.* with the fix (266 suites, 3032 tests, 0 failures, 178 skipped), and TestSimpleTextDocValuesFormat in :lucene:codecs (148 tests, 0 failures, 10 skipped).
• Lint with the fix: checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass.
• On 645d4fe14a20 with the same diff shadowed in: the hunt's five-type test, 5 of 5 pass (assertions on); the single-valued SORTED_NUMERIC route and the SORTED_SET read-ords-then-intoBitSet() order, 2 of 2 pass (assertions on and off). Limits: the gradle regression test is NUMERIC only, and single-valued SORTED_SET stays broken until LFV-091 is fixed.
• Earlier, the hunt's equivalent fix passed the finder's five-type test, the independent reproducer's 10 tests and the adjudication's re-run.

After docsWithField.intoBitSet(...), advance the value cursor by the number of documents with the field that were passed (for SORTED_NUMERIC and SORTED_SET also the per-document counts), then load the current document's value.

Trade-offs

The iterator does not know that count today (it would need the DocsWithFieldSet bits or a running counter), and PackedLongValues.Iterator has only hasNext() and next() (no skip), so the value side still steps entry by entry: more code in five classes for a gain that was never measured.

A javadoc note on the flush producers.

Trade-offs

Contradicts DocIdSetIterator#intoBitSet's "behave exactly as" rule, and a consumer has no way to tell a flush iterator from any other NumericDocValues.

Method: bounded-exhaustive translation validation (the hunt's "semi-formal" tier), hunt unit (doc-values and norms cursor coherence), 2026-10-26, main ea57360912ed.

• The harness (Coherence.java) enumerated every legal move sequence of depth <= 3 over nextDoc, advance, advanceExact, intoBitSet, docIDRunEnd, intoArray and the bulk value calls, on segments of <= 5 documents, for 26 doc-values and norms views, including the flush-time iterators handed to a pass-through DocValuesConsumer. After every step it compared the iterator state with a fresh nextDoc() enumeration of the same view.
• The sequence nextDoc > intoBitSet > value read on the flush-time view mismatched for all five doc-values types. The finder reduced it to a public-API JUnit test on the stock main jars (5 of 5 fail, assertions on and off), and the hunt later minimized it to one NUMERIC test.
• Earlier units had noted the same pattern by code reading (one as a possible latent defect, one as benign), without a repro.