A SORTED_SET DictionaryColumn tuple that addBatch rejects leaves an empty document behind, which can make the next commit fail and close the IndexWriter

IndexWriter#addBatch (experimental, since 10.5.0) indexes documents column by column. For a SORTED_SET DictionaryColumn, Lucene range-checks the ordinal of every (docID, ordinal) tuple. It rejects an ordinal outside the dictionary with IllegalArgumentException, treats that as a non-aborting exception (the batch's documents are marked deleted) and leaves the writer open.

But SortedSetDocValuesWriter.addOrdinalTuples switches to the new document before it reads and checks the ordinal. When the rejected tuple is the first one of a document, that document stays open with no values, and the next value for the field, from a later document, records it with a value count of 0.

With assertions on, the next commit always fails (AssertionError). With them off it can fail only when the rejected document is the segment's first with the field, as in the repro, where it fails with ArrayIndexOutOfBoundsException. A failed flush is a tragic event, so the writer closes and every uncommitted document is lost: 1 of 7 survives in the repro. Otherwise, with assertions off, the commit succeeds but keeps the empty document: the field then throws NullPointerException on open or fails CheckIndex, or, when no document has more than one value, shows nothing. A cursor whose ordValue() throws on a document's first tuple, with every ordinal valid, does the same.

The single-valued SORTED path validates first. The fix is to switch documents after the check, as addValue already does.

• addBatch: IndexWriter#addBatch(ColumnBatch) (experimental, since 10.5.0) adds N documents given as columns, one field at a time.
• DictionaryColumn: a column of string values given as a fixed dictionary plus, per document, ordinals into it. For SORTED_SET doc values its OrdinalsTupleCursor yields (docID, ordinal) pairs; a document with 3 values yields 3 pairs with the same docID.
• The doc-values writer (SortedSetDocValuesWriter) buffers one in-memory segment's values. It keeps one open document (currentDoc) and its values. When a value for another document arrives, finishCurrentDoc() records the open one: its value count and its doc ID in docsWithField.
• Non-aborting exception: an exception that rejects the document(s) being added but keeps the writer and its buffered documents. Lucene marks the rejected documents deleted. An aborting exception, such as any failure during a flush, is tragic: the writer rolls back to the last commit and closes.

1. Every SORTED_SET DictionaryColumn, dense or sparse, goes to SortedSetDocValuesWriter.addOrdinalTuples (IndexingChain.java:1244-1248). Only sparse columns were executed. A column that is also stored or indexed reads every ordinal in the row pass first (ColumnFieldAdapter.java:79-81), so an out-of-range ordinal fails there and never reaches this loop: doc-values-only columns are the exposed ones.
2. For each tuple the loop first switches documents: if (docID != currentDoc) { finishCurrentDoc(); currentDoc = docID; } (SortedSetDocValuesWriter.java:126-129). Only then does it call cursor.ordValue() (:130) and range-check the ordinal (:131-140).
3. When the first tuple of a document is rejected, or ordValue() throws, currentDoc already names the new document, which has 0 values.
4. The exception is non-aborting. DocumentsWriterPerThread.updateBatch only marks the batch's documents deleted (:349-354), and the writer's currentDoc stays where it is.
5. The next value for the field, from any later document, calls finishCurrentDoc(). That records the rejected document with a count of 0 (:162-193).
6. At flush, BufferedSortedSetDocValues.nextDoc asserts ordCount > 0 (:360), so with assertions on every layout fails. With assertions off, the Lucene90 consumer reads a value that is not there (Lucene90DocValuesConsumer.java:403, 1005), but only when the empty document is the first with the field in the segment; otherwise the flush succeeds (see the numbers below).
7. Any exception during a flush is aborting (DocumentsWriterPerThread.java:585-587, 594-604), so a failed flush closes the writer.

addValue, the per-document and BinaryColumn path into the same writer, validates first and switches after (:84-105). The single-valued SortedDocValuesWriter.addOrdinalTuples also records the document only after it translates the ordinal (SortedDocValuesWriter.java:139-142).

Default IndexWriterConfig and codec. 1 committed document, 5 buffered, a rejected 1-document batch (dictionary [x, y], ordinal 5), then a document with tags p, q:

• commit() throws AssertionError (-ea) or ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0 (-da);
• isOpen() is false;
• the reopened index has 1 of 7 documents.

With -da and NoMergePolicy, take a 1-value document, the rejected batch, then a 2-value document. The commit succeeds and keeps the 0-value document. The writer has written the field as multi-valued, but omitted the ord addresses, because 3 values = 3 documents (Lucene90DocValuesConsumer.java:924-933). Opening the field then throws NullPointerException: Cannot read field "numBlocks" because "meta" is null (Lucene90DocValuesProducer.java:2303). With 3 values in the last document, CheckIndex fails instead ("docValueCount=0 for docID=1"); an integrity-level CheckIndex also fails for [2] | rejected | [2] and [1] | rejected | [2, 2, 2]. With 1 value in every document nothing is visible: the field is written as single-valued, and the deleted document gets a stale copy of the previous document's ordinal (SortedSetSelector.java:165-168). So -da does not always fail. The hunt's sweep read back the live data of 589 of 605 -da layouts exactly, and the other 16, its cells where the rejected document came first, failed at commit; its CheckIndex ran without a level set and skipped the doc-values check, so the 589 do not show a clean index.

• Who is hit.
  • Applications that call IndexWriter#addBatch with a SORTED_SET DictionaryColumn that only feeds doc values, when the first tuple of some document is rejected (an ordinal outside the dictionary) or its ordValue() throws, and that catch the exception and keep indexing.
  • Default IndexWriterConfig and codec; dense and sparse columns take the same path (only sparse ones were executed). A column that is also stored or indexed reads every ordinal in the row pass first, so an out-of-range ordinal fails there instead.
  • SORTED DictionaryColumns and BinaryColumns that feed SORTED_SET are not affected, nor is a bad tuple after a valid tuple of the same document.
• Why it bites.
  • The rejection looks clean: isOpen() is true and getTragicException() is null.
  • The damage shows up at the next flush or commit. Its exception, an AssertionError or ArrayIndexOutOfBoundsException inside the codec, does not point back at the batch.
  • The rollback discards every document since the last commit, from every thread that shares the writer.
• Persisted damage with -da, the production default. When an earlier document in the segment already has the field, the commit succeeds and keeps the 0-value document.
  • Some layouts then commit a segment whose SORTED_SET field cannot be opened, sorted on or merged. An independent probe saw a later writer's forceMerge(1) fail on it and report an unrecoverable error.
  • Other layouts fail CheckIndex (integrity level); when no document has more than one value, nothing is visible.
• How likely. Rare: an opt-in experimental API, an error path, and no known downstream user. A read-only code search of Elasticsearch, OpenSearch and Solr found no SORTED_SET DictionaryColumn builder; Elasticsearch uses the column package through BinaryColumn, which is not affected.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected commit to succeed after the rejection, got java.lang.AssertionError
       org.apache.lucene.index.TestRejectedSortedSetOrdinalBreaksCommit.testCommitAfterRejectedOrdinal(TestRejectedSortedSetOrdinalBreaksCommit.java:84)
     Caused by: java.lang.AssertionError
       org.apache.lucene.index.SortedSetDocValuesWriter$BufferedSortedSetDocValues.nextDoc(SortedSetDocValuesWriter.java:360)
       org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.isSingleValued(Lucene90DocValuesConsumer.java:961)
       org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:977)
       org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedSetField(PerFieldDocValuesFormat.java:127)
       org.apache.lucene.index.SortedSetDocValuesWriter.flush(SortedSetDocValuesWriter.java:301)
       [...]

• Lucene validates this input on purpose.
  • The range check is in the code (SortedSetDocValuesWriter.java:131-140).
  • The dense cursor documents it: "The indexing path validates this on every call and throws IllegalArgumentException on violation" (OrdinalsCursor.java:50-52).
  • TestColumnBatchDictionaryColumn#testOutOfRangeOrdSparseThrows asserts the rejection on the sparse tuple path, but only for SORTED (:391-407). Re-typed to SORTED_SET, its close() fails.
• The design note. Non-aborting exceptions "can still partially update the index structures. These updates are consistent, ... we immediately mark the document as deleted so that the document is always atomically ("all or none") added to the index" (DocumentsWriter.java:76-79). addDocument promises that after an Exception "the index will be consistent" (IndexWriter.java:1465); addBatch's short javadoc (:1528-1536) does not restate it.
• Recovery tests. The four TestColumnBatchIndexing#*ThrowMidBatchRecovers tests (:682, 740, 794, 841) commit and keep indexing after a cursor throws mid-batch.
• Siblings validate first.
  • SortedSetDocValuesWriter.addValue (:84-105).
  • SortedDocValuesWriter.addOrdinalTuples (:123-146), added by the same commit.
  • That commit (#16101) also gave SortedDocValuesWriter.addDenseOrdinalValues a try/finally that keeps the writer consistent when the cursor throws (:169-182).
• History.
  • #16101 has one approving review and no discussion of the order.
  • #16553 (2026-09-01) fixed the same class of defect in stored fields: a non-aborting exception left per-document state, and the next flush failed.
  • In #16734, a reviewer asked that the dense vector path record doc IDs only after validation.

The strongest case for "intended". OrdinalsTupleCursor says only that each ordinal "must be in" range (OrdinalsTupleCursor.java:26, 46-47), without the dense cursor's "validates and throws" sentence. addBatch and the cursor are @lucene.experimental. So, the argument goes, the caller broke a precondition, and what follows is undefined. An independent reviewer from another model family (OpenAI GPT-5.6) rated this "moderate" and still judged the bug real (high).

Why it fails.

• Lucene does not treat the ordinal as undefined behaviour: it checks it, throws a non-aborting exception and keeps the writer open.
• A cursor whose ordValue() throws with every ordinal valid breaks no precondition, and hits the same defect.
• The SORTED version of the same test passes.
• No javadoc, CHANGES or MIGRATE entry says that a failed addBatch needs rollback().

One honest limit: the upstream recovery tests throw from nextDoc() or the dense fill, not from ordValue(), and none uses a DictionaryColumn. So the ordValue() case rests on the general non-aborting contract, not on a word-for-word test.

Option A is the recommended fix and was run through gradle. B is a design alternative that was not run. C is a safety net, run against the new test only.

Move if (docID != currentDoc) { finishCurrentDoc(); currentDoc = docID; } below the range check in SortedSetDocValuesWriter.addOrdinalTuples. A rejected or throwing first tuple then leaves currentDoc on the previous, complete document. A rejected later tuple leaves the open document with at least one value: a consistent, deleted document.

      assert docID >= currentDoc;
      // Read and validate the ordinal before switching documents, like addValue: if ordValue()
      // throws or the ordinal is rejected, no document may be left open without values.
      int ord = cursor.ordValue();
      if (ord < 0 || ord >= dictionary.size()) {
        throw new IllegalArgumentException(/* ... is out of range [0, n) */);
      }
      if (docID != currentDoc) {
        finishCurrentDoc();
        currentDoc = docID;
      }

Trade-offs

• No extra work per tuple; the same statements in a different order.
• It covers both triggers, the rejected ordinal and a throwing ordValue(). After the switch the loop still calls dictionary.get(ord), hash.add and ArrayUtil.grow. For a dictionary that the DictionaryColumn constructor checked and that is not changed later, these fail only at hard size limits (ArithmeticException at the byte pool's 2 GB limit, IllegalArgumentException beyond the maximum array length). Those are non-aborting too and could still leave a document open; option C covers that case.
• It does not add the missing tuple doc-ID checks of a separate bug in the same loop, which needs its own change.

What the checks validated

• Scratch worktree of main a9779ef3c193, gradle, seed DEADBEEF, -Ptests.asserts=true and =false.
• The new test passes 6 of 6.
• 11 classes, 254 tests, 0 failures, 3 skipped: org.apache.lucene.document.column.* (7 classes, including TestColumnBatchDictionaryColumn and the new test), TestColumnBatchIndexing, TestDocValuesIndexing, TestIndexWriterExceptions and TestDocValues.
• Static checks pass: ecjLintMain, ecjLintTest, forbiddenApis and checkGoogleJavaFormat.
• Earlier, the same reordering made the finder's, the independent reproducers', the cross-model reviewer's and the security probe's tests pass, with -ea and -da.
• Not run: the full ./gradlew check, other modules, x86 and benchmarks.

Wrap the loop body in try/finally, like SortedDocValuesWriter.addDenseOrdinalValues (SortedDocValuesWriter.java:169-182). When a tuple fails while the open document has no values, leave the writer with no open document.

Trade-offs

Broader: it covers any exception after the switch. The "no open document" state already exists: currentDoc == -1, which finishCurrentDoc() skips (SortedSetDocValuesWriter.java:163-165). Since the previous document has already been recorded, B sets currentDoc back to -1; setting it back to the previous document would record that document twice. More code than A. Not run.

In finishCurrentDoc(), return early when currentUpto == 0. Then docsWithField and the value counts never list an empty document, whatever left it open (+5 lines).

Trade-offs

One more branch per document. It hides the wrong order instead of fixing it: currentDoc still names the rejected document. Good as defence in depth next to A, weaker on its own.

What the checks validated

run on its own, without A: the new test 6 of 6 and TestColumnBatchDictionaryColumn 25 of 25 pass with assertions on and off; the wider suites of A were not run for C

• The hunt unit. It covered the column-oriented addBatch pass. Its AI tool modelled the commit state of the column-pass doc-values writers in Z3 (dense_commit.py). Check W4 asked whether a document can be registered with 0 ordinals after a rejected ordinal: reachable (sat).
• The sweep. A failure-injection sweep on the real jar, "dict SORTED_SET sparse bad ordinal", varied the layout around one bad ordinal over 605 grid points. With -ea all 605 failed. With -da, 16 failed at commit with ArrayIndexOutOfBoundsException, and in 589 the live data read back exactly. Its CheckIndex ran without a level set and skipped the doc-values check, so the 589 do not show a clean index.
• The candidate. The AI tool then wrote the JUnit repro and the fix sketch. An independent reproducer found the -da unreadable-field layouts and the throwing-cursor trigger.