IndexWriter#addBatch does not range-check the tuple doc ids of a doc-values-only DictionaryColumn: doc id numDocs() lands on the next document added

IndexWriter#addBatch(ColumnBatch) (experimental, since 10.5.0) indexes a batch column by column; the application writes the tuple cursors that say which batch document gets which value, and ColumnBatch#numDocs says "All column doc-ids must be in the range [0, numDocs())". The column pass enforces that rule with ColumnValidation.checkDocID on every LongColumn, BinaryColumn and vector tuple, and the row pass enforces it for stored and indexed columns, but IndexingChain.processDictionaryColumn hands a doc-values-only DictionaryColumn's tuples straight to the SORTED / SORTED_SET writers, which check only that ids increase (SORTED) or only assert it (SORTED_SET). So in a 2-document batch a tuple for doc 2 is accepted, stored for segment doc baseDocID + 2, and the next document added with plain addDocument silently reads that value; doc id -1 can land on the document added just before the batch, if there is one. No error with assertions on or off. Every trigger is a caller cursor that breaks the documented contract of an experimental API, so the priority is low and maintainers will likely call the fix hardening; but the same mistake is rejected on every other column kind, and here a document outside the batch changes. A related gap, fixed by the same patch: on SORTED_NUMERIC and SORTED_SET columns the only check that tuple doc ids never decrease is a Java assert. It is in 10.5.0 to 10.6.0 and main; a cursor wrapper that calls checkDocID fixes it and passes all 8,898 core tests.

// IndexingChain.java:1223-1257 (processDictionaryColumn), added by #16101, unchanged since
case SORTED -> {
  SortedDocValuesWriter writer = (SortedDocValuesWriter) pf.docValuesWriter;
  if (column.density() == Column.Density.DENSE) {
    OrdinalsCursor cursor = column.values();
    ColumnValidation.checkDenseCount(column, cursor.size(), numDocs);   // :1238
    writer.addDenseOrdinalValues(baseDocID, dict, cursor);
  } else {
    writer.addOrdinalTuples(baseDocID, dict, column.tuples());          // :1241  no checkDocID
  }
}
case SORTED_SET -> {
  SortedSetDocValuesWriter writer = (SortedSetDocValuesWriter) pf.docValuesWriter;
  OrdinalsTupleCursor cursor = column.tuples();                         // dense or sparse
  writer.addOrdinalTuples(baseDocID, dict, cursor);                     // :1247  no checkDocID
}

// SortedDocValuesWriter.java:131-144 (addOrdinalTuples)
while ((batchDocID = cursor.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
  int docID = baseDocID + batchDocID;
  if (docID <= lastDocID) { throw new IllegalArgumentException("... appears more than once ..."); } // :133
  int ord = cursor.ordValue();
  int id = lookupOrTranslate(ord, dictionary, ordToHash);   // the ordinal IS range-checked (:186)
  pending.add(id);
  docsWithField.add(docID);
  lastDocID = docID;
}

1. Every other tuple loop checks the batch doc id first: ColumnValidation.checkDocID(column, batchDocID, numDocs) (ColumnValidation.java:296-307, "Throws if batchDocID is outside [0, numDocs)") at IndexingChain.java:982, :990, :1017, :1026, :1038 (LongColumn), :1125, :1136, :1149, :1162 (BinaryColumn), :1280, :1293, :1306 (vectors). The row pass rejects out-of-order (:903-909) and out-of-range (:948-958) ids of stored and indexed columns.
2. processDictionaryColumn does not, for sparse SORTED and for every SORTED_SET column. The writers do not check the range either: the SORTED writer checks only that ids increase (SortedDocValuesWriter.java:133), the SORTED_SET writer only asserts it (SortedSetDocValuesWriter.java:125).
3. So segment doc baseDocID + numDocs gets a value although that document does not exist yet. The next document added to the in-memory segment takes that number; if it has no value for the field, nothing notices, and the flush writes the value as its own. Doc id -1 gives segment doc baseDocID - 1, the document added just before the batch, if the in-memory segment has one. For SORTED_SET the value is added to that document's set, because the writer keeps its last document open until the next one or the flush; SORTED accepts it only if that document has no value for the field (otherwise the order check at :133 rejects it, as it does when the batch starts the segment). The executed -1 runs (a SORTED_SET test of the study, a SORTED case of the diagram probe) each had one earlier document without a value.
4. No error with the default codec, assertions on or off, when a later document fills the slot. With no later document, one exploration run (assertions off, default codec; a single observation, not a general result) committed a 2-document segment that read back 0=a 1=b (batch doc 1 had no value) and CheckIndex reported it clean; with the test framework's asserting codec the flush can instead fail an assertion.
5. Not affected: DENSE SORTED dictionary columns (the dense path counts values, :1236-1239) and stored or indexed dictionary columns (the row pass rejects the id first).

SortedNumericDocValuesWriter.addValue (:63), SortedSetDocValuesWriter.addValue (:85) and SortedSetDocValuesWriter.addOrdinalTuples (:125) check that doc ids never decrease only with assert docID >= currentDoc;. The two addValue asserts were written in 2016 (LUCENE-7474) for the per-document chain, which generates the ids itself, and #15990 routed caller tuple ids into them; #16101 wrote addOrdinalTuples with a copy of the same assert; the cursor javadoc says the ids are "emitted in non-decreasing order". With tuples (1, v) then (0, v') on a SORTED_NUMERIC LongColumn or SORTED_SET column: with assertions on, an AssertionError escapes addBatch and the writer closes as a tragic event (IndexWriter.java:1581-1589); with assertions off (production) the batch is accepted and a later valid addDocument with that field is rejected with Out of order doc ids: last=1, next=0; the study also saw a later document read a batch value ([20] instead of [40]) and, for SORTED_SET, the next commit fail and close the writer. NUMERIC, BINARY/SORTED and SORTED dictionary columns already reject backwards ids, through the single-valued writers' "appears more than once in this document" check. The one-test repro covers the range gap only.

• Who is hit. Callers of the experimental IndexWriter#addBatch (also updateDocuments(Term|Query, ColumnBatch)), new in 10.5.0, with a doc-values-only DictionaryColumn (SORTED sparse; SORTED_SET sparse or dense) and a tuple cursor of their own that returns numDocs or -1. Lucene ships only abstract columns and cursors, so every caller writes its own cursor.
• No special setup. Default IndexWriterConfig, codec and directory; a 2-document batch with one boundary tuple; deterministic; assertions on or off. Far-out ids were not run.
• What goes wrong. A document outside the batch, which the caller added with plain addDocument, silently gets a SORTED or SORTED_SET value it never had (for SORTED_SET, an extra value in its set), and anything that reads that field's doc values sees it. With no later document, a document inside the batch can get the value instead, and CheckIndex called the result clean (one exploration run, default codec).
• What does not go wrong. Correct cursors are unaffected; DENSE SORTED, stored and indexed dictionary columns are checked; nothing outside doc values is touched. The one downstream user found (Elasticsearch, by code reading) builds no DictionaryColumn, and its tuple cursors take doc ids from a bounded row counter.
• Impact class: data corruption, likelihood contrived (it needs a buggy caller cursor on an experimental API); priority low.

The test subclasses DictionaryColumn and ColumnBatch through their public and protected constructors, sets the default codec explicitly (TestUtil.getDefaultCodec()) and uses no randomness; a second seed (5EED0098, gradle, assertions on) gives the same failure. It accepts any IllegalArgumentException and does not check its message, the live-doc count, or the deleted documents a rejected batch leaves behind. Use this one test as the regression test too.

The same failure, by the hunt repro harness: main 645d4fe14a20, 10.6.0 and 10.5.0, assertions on and off. It does not compile against 10.4.0, which has no column API. The SORTED_SET, doc id -1 and order cases come from the study's other tests and are supporting evidence, not part of the repro.

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on
TestDictionaryColumnDocIdOutOfRange > testDocIdEqualToNumDocsDoesNotReachNextDocument FAILED
    java.lang.AssertionError: tag of the document added after the batch expected null, but was:<b>
        at org.apache.lucene.document.column.TestDictionaryColumnDocIdOutOfRange.testDocIdEqualToNumDocsDoesNotReachNextDocument(TestDictionaryColumnDocIdOutOfRange.java:90)
1 test completed, 1 failed

# gradle, same, -Ptests.asserts=false: the same failure and message
# repro harness, the same message: main 645d4fe14a20, 10.6.0 and 10.5.0, -ea and -da (1 run, 1 failed each)
# with the fix: the test passes (gradle, assertions on and off)

• The same boundary rejects the same mistake everywhere else. ColumnValidation.checkDocID runs on every Long, Binary and vector tuple, and the row pass rejects bad ids for stored and indexed columns. Upstream tests assert it: TestColumnBatchIndexing#testRowColumnOutOfRangeDocIdThrows and #testRowColumnOutOfOrderDocIdThrows, TestColumnBatchVectorColumn#testVectorOutOfRangeDocIDFails and #testDecreasingDocIDFails. The study's parity probe: out-of-range ids on BinaryColumn SORTED, LongColumn NUMERIC and a stored DictionaryColumn are rejected with IllegalArgumentException and the writer stays open.
• The other clause of the same javadoc is enforced. OrdinalsTupleCursor says ids are "batch-local (0 to numDocs - 1)" and each ordinal "must be in [0, dictionary.length)"; the ordinal rule is checked on this path and tested (TestColumnBatchDictionaryColumn#testOutOfRangeOrdSparseThrows, #testOutOfRangeOrdDenseThrows), the doc-id rule is not.
• History shows an omission, not a decision. checkDocID came with the API (#15990, 2026-05-14). #16101 (2026-05-26) added DictionaryColumn without it, with one approving review and no comments; its tests cover out-of-range ordinals only. The reviewed history (both PRs and their review threads, the 9 mails of the dev@ DISCUSS thread) and the code comments give no sign that the omission was deliberate.
• Precedents. #16708 (merged 2026-09-28; CHANGES.txt lists it under 10.6.0 "Improvements", not "Bug Fixes") closed another addBatch-vs-addDocument validation gap, vector values; the API author's open #16116 adds doc-id order and upper-bound checks for a new column-pass consumer. Both support consistency; #16708's label also points toward a "hardening" label for this one.
• The damage leaves the batch: the value lands on a document the caller added separately, through addDocument.

Strongest case for "intended" (caller misuse, so the fix is hardening): the rule is a caller obligation. ColumnBatch#numDocs says "All column doc-ids must be in the range [0, numDocs())", the cursors say ids are batch-local and non-decreasing, the API is @lucene.experimental, and addBatch declares only IOException, so nothing promises an IllegalArgumentException. Breaking the precondition is undefined behaviour. (Built by the study's intended-behaviour checker and an independent reviewer from another model family, OpenAI GPT-5.6, which rated it moderate.)

Why it fails: (1) Lucene treats the range part of that precondition as something to enforce: every other column-pass tuple loop range-checks its doc ids, since the API's first commit, with tests, and the neighbouring ordinal clause of the same javadoc is enforced on this very path (order is enforced less uniformly: three sibling paths only assert it). (2) The range check is two int comparisons per tuple, which every sibling path already pays; fix A adds one order comparison and one wrapper call per dictionary tuple. (3) The failure is silent and lands outside the caller's batch. (4) #16708 and #16116 show the project fixing exactly this kind of gap. What remains is the label, bug or hardening, which sets the low priority, not the verdict. A maintainer may also cite the dev@ proposal's goal of "less redundant field validation"; the study's intended-behaviour checker read that as schema checks once per column, not per-tuple doc-id checks. Verdict: not intended.

A is validated and recommended; its checkedTuples wrapper is what fixes this bug, and the writer changes close the related order gap. B is the range part alone. A separate finding about a rejected out-of-range ordinal on a SORTED_SET DictionaryColumn has a different root cause, but its fix edits the same addOrdinalTuples loop. Workaround for users today: in your own cursor, check 0 <= doc && doc < numDocs, and that ids never decrease, before returning a doc id from nextDoc().

The study's fix, formatted with the build's google-java-format, +61/-6 in IndexingChain, SortedNumericDocValuesWriter and SortedSetDocValuesWriter. Shown: the wrapper that both dictionary branches now use (writer.addOrdinalTuples(baseDocID, dict, checkedTuples(column, numDocs))). The writer changes replace assert docID >= currentDoc; with if (docID < currentDoc) throw new IllegalArgumentException("field \"...\": doc ids must be added in non-decreasing order; got " + docID + " after " + currentDoc).

private static OrdinalsTupleCursor checkedTuples(DictionaryColumn column, int numDocs) {
  final OrdinalsTupleCursor in = column.tuples();
  return new OrdinalsTupleCursor() {
    int last = -1;

    @Override
    public int nextDoc() {
      int doc = in.nextDoc();
      if (doc != DocIdSetIterator.NO_MORE_DOCS) {
        ColumnValidation.checkDocID(column, doc, numDocs);
        if (doc < last) {
          throw new IllegalArgumentException(
              "DictionaryColumn \""
                  + column.name()
                  + "\" returned out-of-order batch doc-id "
                  + doc);
        }
        last = doc;
      }
      return doc;
    }

    @Override
    public int ordValue() {
      return in.ordValue();
    }
  };
}

Trade-offs

Callers with a buggy cursor now get an IllegalArgumentException from addBatch, as on the other column kinds. Cost: three int comparisons (two in checkDocID, one for order) and one wrapper call per dictionary tuple (no benchmark run). The writer checks also sit on the per-document path (addValue runs for every SORTED_NUMERIC / SORTED_SET value of addDocument, where ids are always in order); an if replaces an assert there. Not a rollback: the exception comes after updateBatch has reserved the batch's doc ids, and earlier tuples may already be buffered; the batch's documents are then marked deleted (DocumentsWriterPerThread.java:339-354), not erased. In the diagram probe's fix run the rejected batch left 2 deleted documents in RAM and tuple (0, a) buffered; the readers then showed one live document with no tag, and CheckIndex was clean. What was tested is a usable writer and no spill onto the next document.

What the checks validated

• Scratch worktree of main 645d4fe14a20 (removed afterwards), seed DEADBEEF: stock, the test fails through gradle with assertions on and off; with A it passes.
• Closest suites with A, -Ptests.asserts=true and false: the repro, org.apache.lucene.document.column.* (TestColumnBatchBinaryColumn, TestColumnBatchDictionaryColumn, TestColumnBatchLongColumn, TestColumnBatchMultiColumnField, TestColumnBatchTokenStreamColumn, TestColumnBatchVectorColumn), TestColumnBatchIndexing, and the doc-values writer users TestDocValuesIndexing, TestDocValues, TestMultiDocValues, TestIndexSorting, TestLucene90DocValuesFormat*: 784 tests in 15 classes, 55 skipped, 0 failures each time.
• All of :lucene:core:test with A (assertions on): 8,898 tests in 749 classes, 322 skipped, 0 failures; :lucene:core:checkGoogleJavaFormat passes on the formatted patch. The same results before and after formatting.
• The study's 6-test class (range and order cases) fails 6 of 6 on stock 645d4fe14a20 and passes 6 of 6 with A, assertions on and off.

Keep only the checkDocID call in the wrapper. A check inside the two addOrdinalTuples loops would need more: both take only (int baseDocID, List<BytesRef>, OrdinalsTupleCursor), so they would have to gain numDocs and the column for checkDocID(Column, int, int), or carry their own range check.

Trade-offs

Fixes this bug; leaves the order gap assert-only. Use it if the order gap is fixed separately.

Document that out-of-range ids are undefined behaviour.

Trade-offs

Leaves a silent write onto another document, and keeps the dictionary path inconsistent with every other column kind.

• Method (non-formal): fuzzing in a hunt unit on the columnar addBatch API. A Jazzer target built random column batches and checked a writer-consistency property after every addBatch. Its smoke run found a dense SORTED_SET DictionaryColumn for numDocs = 7 whose 8 tuples had doc ids 0 to 7 (the target numbers dense tuples 0, 1, 2, ...), so the last was doc id 7 = numDocs, one past the end; addBatch accepted it and it changed the next document's values ("recovery doc lost its SORTED_SET value for field a (batch threw=false)"). The tuple count alone is not the fault: a dense SORTED_SET column needs at least one value per document, and this path counts nothing.
• Diagnosis. A read-only check showed that every column-pass tuple loop calls checkDocID except the dictionary path. The order gap was found separately, by code reading while modelling the column pass: the 2016 per-document asserts now receive caller tuple ids.
• Reduction. The finder wrote public-API JUnit tests (6 tests, range and order); a later minimization kept only the 2-document SORTED case as the one-test repro.