Index-sorted flush: the numeric doc-values and norms iterator ends intoBitSet(maxDoc) on maxDoc instead of NO_MORE_DOCS

When an index sort is set and the flush actually reorders the documents, IndexWriter re-orders each numeric doc-values field and the norms at flush and hands the codec's consumer a NumericDocValuesWriter.SortingNumericDocValues iterator (already-sorted input gets an unsorted iterator). If every document of the segment has the field, that iterator's intoBitSet(upTo, bitSet, offset) with upTo >= maxDoc sets the right bits but leaves docID() at maxDoc (2 for a 2-document segment) instead of NO_MORE_DOCS. The intoBitSet javadoc says the method "should behave exactly as" the docID()/nextDoc() loop, which ends on NO_MORE_DOCS here, and the sparse branch of the same method does end there. A consumer that re-reads docID() until NO_MORE_DOCS never ends, as IndexedDISI.writeBitSet does. Lucene's own consumers skip the bulk load for such fields, so default indexes are unaffected; a custom codec that bulk-loads the flush-time iterator is. Introduced by #16126 in 10.5.0; one line fixes it.

1. The dense flag. At flush with an index sort, NumericDocValuesWriter.getDocValuesProducer calls sortDocValues(..., sortMap.size() == docsWithField.cardinality()) (NumericDocValuesWriter.java:150-155); norms do the same (NormValuesWriter.java:74-80). The flag is true exactly when every document has a value, and then sortDocValues keeps docsWithField = null (:113-116), the dense representation from #12381 (9.8.0).
2. The iterator. The consumer gets new SortingNumericDocValues(sorted) (NumericDocValuesWriter.java:169; norms NormValuesWriter.java:95). A single-valued SortedNumericDocValuesField gets the same producer wrapped in DocValues.singleton (SortedNumericDocValuesWriter.java:215-226).
3. Its nextDoc() returns NO_MORE_DOCS once docID + 1 == maxDoc (:247-254).
4. Its intoBitSet (:281-293, added by #16126) clamps upTo to maxDoc (:283). The dense branch sets the bits and then docID = upTo; (:287), which is maxDoc whenever the caller asked for maxDoc or more. The sparse branch of the same method maps that case to NO_MORE_DOCS: docID = upTo < dvs.maxDoc ? dvs.advance(upTo) : NO_MORE_DOCS; // set the current doc (:290).
5. The contract. DocIdSetIterator#intoBitSet "should behave exactly as if implemented as" for (int doc = docID(); doc < upTo; doc = nextDoc()) bitSet.set(doc - offset); (DocIdSetIterator.java:187-195), which ends on NO_MORE_DOCS here; docID() may only be -1, the current document or NO_MORE_DOCS (:113-124). The method is @lucene.internal (:203).

The bits are right in every run; only the position is wrong. IndexedDISI.writeBitSet loops for (doc = it.nextDoc(); doc != NO_MORE_DOCS; doc = it.docID()) with one intoBitSet per 65,536-document block (IndexedDISI.java:210-213). After the first block the position is maxDoc, so the next intoBitSet sets nothing, and the empty block trips assert cardinality > 0 (IndexedDISI.java:118); with assertions off it writes a 4-byte block header (:117, :119) and loops again without end (the hunt's test stopped it at 1 MiB). That spin comes from re-reading docID(); by reading only, a consumer that calls nextDoc() and longValue() instead would get maxDoc + 1 and index past the values array (NumericDocValuesWriter.java:247-254, :269-270). Other consumer shapes were not tested.

Lucene90DocValuesConsumer writes no IndexedDISI when numDocsWithValue == maxDoc (Lucene90DocValuesConsumer.java:467, before writeDISI at :474), Lucene90NormsConsumer likewise (Lucene90NormsConsumer.java:107, before IndexedDISI.writeBitSet at :117), and SortingCodecReader always builds a non-null docsWithField (SortingCodecReader.java:793-804), so merges never take the dense branch. An executed probe saw 0 intoBitSet calls on dense fields across index-sorted flushes of 1, 2, 3, 100 and 65,543 documents, a sorted forceMerge and addIndexes of a SortingCodecReader, with a clean CheckIndex.

• Who is hit. An application or library whose custom doc-values or norms consumer (plugged in through FilterCodec, PerFieldDocValuesFormat or its own codec; DocValuesFormat, DocValuesConsumer and NormsConsumer are @lucene.experimental) calls intoBitSet up to maxDoc or NO_MORE_DOCS on the flush-time iterator, directly or through the @lucene.internal IndexedDISI.writeBitSet, and then reads docID(), for a numeric doc-values, single-valued sorted-numeric or norms field that every document of an index-sorted segment has.
• Default vs opt-in. Opt-in: in the tested stock paths nothing observable changes, and the index, values, norms and sort order are correct; by reading, the current Lucene90 doc-values and norms consumers (and the backward-codecs Lucene80DocValuesConsumer) skip the docs-with-field encoding for a field every document has.
• Scale. None needed: 2 documents are enough, and a field present on every document is the ordinary case, so a consumer shaped like IndexedDISI.writeBitSet fails at the first index-sorted flush that reorders such a field.
• How it shows. It depends on the consumer. In the executed case, IndexedDISI.writeBitSet (which re-reads docID() until NO_MORE_DOCS) fails the flush with an AssertionError or writes until the output fails: no wrong match and no committed segment. Other consumer shapes were not tested; by reading, one that calls nextDoc() and longValue() would fail out of bounds instead.
• Why fix it anyway. #16126 added these overrides as an optimization (CHANGES "Optimizations"), for callers that bulk-load; the dense case is the common one for such a caller, and the sparse branch next to it already ends correctly. The fix is one line and adds one comparison per bulk call.

One test class, one method, in org.apache.lucene.index (package for locality only). Public classes, but not only supported API: IndexWriterConfig.setIndexSort, FilterCodec, NormsFormat/NormsConsumer (the codec classes are @lucene.experimental), ByteBuffersDirectory, and a direct call of DocIdSetIterator#intoBitSet, which is publicly callable but tagged @lucene.internal (DocIdSetIterator.java:203); that is the main reason the cross-model reviews rated it out of contract. The probe sits in a norms consumer because NormsConsumer has only two methods to delegate; norms flush through the same SortingNumericDocValues as numeric doc values. The sort values are 0 and -1 so that the flush really sorts (already-sorted input takes the unsorted path). The bits are not asserted: they are correct, and only the end position is claimed.

Not shown by this test, which covers the norms path only; executed elsewhere: the numeric doc-values and single-valued sorted-numeric paths, upTo = NO_MORE_DOCS, and the IndexedDISI.writeBitSet runaway (the hunt's 5-test class on a 100-document segment: but was:<100> for doc values, for upTo = NO_MORE_DOCS and for norms, and writeBitSet fails with AssertionError, or with assertions off passes 1 MiB of output; the independent reproducer's 12-test class: docID() = 100 for the single-valued sorted-numeric path). The one test is the repro and, unchanged, the suggested regression test.

# gradle on main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (=false gives the same)
TestSortedFlushIntoBitSetEndPosition > testIntoBitSetUpToMaxDocEndsOnNoMoreDocs FAILED
    java.lang.AssertionError: docID() after intoBitSet(maxDoc) expected:<2147483647> but was:<2>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:9CB77348D7A246]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.index.TestSortedFlushIntoBitSetEndPosition.testIntoBitSetUpToMaxDocEndsOnNoMoreDocs(TestSortedFlushIntoBitSetEndPosition.java:91)
1 test completed, 1 failed

# the same test on the 10.6.0 release jars (assertions on and off): expected:<2147483647> but was:<2>
# the same test on the 10.4.0 release jars: passes

# the hunt's 5-test class on a 100-document segment (main ea57360912ed; not part of the repro)
testSortedDenseDocValuesUpToMaxDoc     => dv:sort: docID() after intoBitSet(100) over every doc expected:<2147483647> but was:<100>
testSortedDenseDocValuesUpToNoMoreDocs => dv:sort: docID() after intoBitSet(2147483647) over every doc expected:<2147483647> but was:<100>
testSortedDenseNorms                   => norms:body: docID() after intoBitSet(100) over every doc expected:<2147483647> but was:<100>
testWriteBitSetOnSortedDenseField      -ea: writeBitSet failed: java.lang.AssertionError
                                       -da: writeBitSet failed: java.lang.IllegalStateException: runaway output: more than 1048576 bytes written
(testUnsortedDenseControl passes)

• The contract. DocIdSetIterator#intoBitSet "should behave exactly as if implemented as" the docID()/nextDoc() loop (DocIdSetIterator.java:187-195); docID() returns -1, NO_MORE_DOCS or "the doc ID it is currently on" (:113-124). The call meets both stated preconditions: the iterator is positioned, and offset (0) is not greater than docID() (:196-197).
• The same method. The sparse branch, written in the same commit, ends on NO_MORE_DOCS (NumericDocValuesWriter.java:290, "set the current doc"), and the class's nextDoc() ends there too (:247-254).
• Tests. BaseDocIdSetTestCase checks "if docID() / nextDoc() return the same value after #intoBitSet has been called" (BaseDocIdSetTestCase.java:222), and BaseDocValuesFormatTestCase asserts expected.docID() == values.docID() after intoBitSet (BaseDocValuesFormatTestCase.java:1085-1089). No test covers the writer-side iterators: #16126 added none.
• History. From 10.2.0 to 10.4.0 the class inherited the default loop and ended on NO_MORE_DOCS (10.4.0 passes the repro). The #16126 PR body describes the change as "mostly iterators wrapping other iterators so we just delegate the method"; this method is not a delegation. The PR had one approval, no comments and no follow-up.
• Precedent (it shows maintainers fix such edge cases; it does not settle this one). Lucene fixed other intoBitSet edge cases as bugs: #14922 (IndexedDISI, CHANGES 10.3.0 "Bug Fixes"), #16250/#16252 (RoaringDocIdSet, CHANGES 10.5.0 "Bug Fixes") and #16105 (BatchDocValuesRangeIterator, a public class, an end state at the maxDoc boundary). Those were reachable from Lucene's own code or from search; this one is a flush-time state no in-tree caller reaches, so a maintainer may weigh it differently.

The strongest case for "works as designed" came from the cross-model reviews (OpenAI GPT-5.6): intoBitSet is @lucene.internal ("This API is for internal purposes only and might change in incompatible ways in the next release"), IndexedDISI is @lucene.internal, the consumer APIs are @lucene.experimental, and the stock consumers deliberately encode a field present on every document without a docs-with-field bitset, so the wrong end state is unobservable in supported use; a codec that relies on an internal bulk hook is outside the contract.

Why it does not make the position correct (though it may well decide how a maintainer triages the report): @lucene.internal limits API stability, not the behaviour the method's own javadoc states, and #16126 added this override as an optimization for callers that bulk-load. The same reviewer wrote that its argument "does not establish that maxDoc itself was a deliberate semantic choice" and that the assignment "is an oversight suitable for low-risk internal hardening". The scope argument correctly lowers the severity (latent, minor); it does not make the position right. Verdict: a real, latent defect; no in-tree consumer reaches the branch.

Fixes A and B were run through gradle in a scratch worktree of 645d4fe14a20. A is recommended: the smallest change, the one the finder, the independent reproducer, the adjudication and the cross-model reviews ran, and it mirrors the sparse branch next to it.

One line at NumericDocValuesWriter.java:287. After the clamp, upTo == maxDoc means the last document was consumed, which is where nextDoc() returns NO_MORE_DOCS; for upTo < maxDoc the dense position stays upTo.

       if (upTo > docID) {
         if (dvs.docsWithField == null) {
           bitSet.set(docID - offset, upTo - offset);
-          docID = upTo;
+          docID = upTo < dvs.maxDoc ? upTo : NO_MORE_DOCS;
         } else {

Trade-offs

• Fixes all three paths at once (numeric doc values, norms, single-valued sorted-numeric), since they share the class.
• Cost: one comparison per bulk call (not benchmarked); no change for upTo < maxDoc or for the sparse branch.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (2026-11-01): the repro fails on stock main and passes with A, each with -Ptests.asserts=true and =false.
• With A: 10 core test classes, 470 tests, 0 failures, 35 skipped (34 @Nightly, 1 assumption): the repro, TestIndexSorting, TestSortingCodecReader, TestNorms, TestCustomNorms, TestDocValuesIndexing, TestDocValues, TestLucene90DocValuesFormat, TestLucene90DocValuesFormatVariableSkipInterval and TestLucene90NormsFormat.
• With A and the test in place, :lucene:core:check -x test passes (google-java-format, ecj lint, forbidden APIs). The full core suite was not run.
• Earlier, the same change: the hunt's 5 tests pass (assertions on) and the independent 12 tests pass (assertions on and off); a reviewer's runs of 10 TestLucene90DocValuesFormat intoBitSet tests and 8 TestLucene90NormsFormat sparse tests pass.

Drop the assignment from both branches and end with docID = upTo < dvs.maxDoc ? dvs.advance(upTo) : NO_MORE_DOCS;, which also works for the dense case because NumericDVs.advance(target) returns target when docsWithField == null (:318-325).

Trade-offs

• Same behaviour as A, with one rule for both branches so they cannot drift apart again.
• Touches the sparse branch too (a slightly larger diff to review).

What the checks validated

• Gradle, same worktree: the repro passes with B with -Ptests.asserts=true and =false; the repro plus TestIndexSorting: 2 classes, 63 tests, 0 failures.
• Not run with the other suites or the static checks.

For example let IndexedDISI.writeBitSet treat docID() >= maxDoc as the end.

Trade-offs

Keeps this iterator out of line with the intoBitSet and docID() contracts that the default implementation and the sparse branch follow; hardening one consumer without fixing the iterator leaves other consumers exposed. A maintainer could still reasonably harden the in-tree consumer as well, or decline the report because no in-tree caller reaches the branch.

Method: non-formal (code reading). A hunt unit on IndexedDISI and doc-values bulk iteration wrote a Z3 bit-vector model of IndexedDISI's arithmetic (38 obligations, 0 unexpected). One obligation records what happens when a source iterator's intoBitSet does not move past upTo: writeBitSet then flushes a block of cardinality 0. Reading the iterators that can feed writeBitSet, the finder found that SortingNumericDocValues does exactly that in its dense branch. No tool output gave the first signal, so the find is classed as non-formal.

The finder then wrote a JUnit test with a real index-sorted IndexWriter flush through a FilterCodec on the stock main jars (5 tests, 4 failing with assertions on and off, the unsorted control passing). It was reviewed by a reviewer from another model family (OpenAI GPT-5.6), reproduced independently (which added the single-valued sorted-numeric path), judged, and cut to the one-test repro on 645d4fe14a20.