With an index sort and a parent field, a rejected document left as the last document of a segment makes the next commit throw and close IndexWriter, losing every uncommitted document

With an index sort and IndexWriterConfig#setParentField, the setup for nested documents (and mandatory for document blocks under an index sort on indexes created by 10.0.0 or later), the flush-time sort compares every document by its next parent: in.compare(parents.nextSetBit(docID1), parents.nextSetBit(docID2)) (IndexingChain.java:261-274 on main 645d4fe14a20). That assumes every doc ID has a parent marker at or after it. A document that IndexWriter rejects with a non-aborting IllegalArgumentException (the per-field schema rule is documented in MIGRATE.md, and upstream's own test expects the exception) has already used a doc ID, which is only marked deleted, and a schema conflict throws before the parent marker is written. When that doc ID is the last one in the in-memory segment and the segment also holds an accepted multi-document block, nextSetBit returns NO_MORE_DOCS (Integer.MAX_VALUE) and the sorter uses it as an array index. The next commit(), NRT reopen or close() throws ArrayIndexOutOfBoundsException: Index 2147483647 out of bounds for length 3; an exception during flush is aborting, so IndexWriter closes itself and every document added since the last commit is lost. The committed index stays intact and a new writer works (executed with one earlier commit). Every release from 9.10.0 to 10.6.0 and main are affected; giving the last rejected doc ID a parent marker fixes it.

All lines are on main 645d4fe14a20, under lucene/core/src/java/org/apache/lucene/.

1. Doc IDs are used before validation. DocumentsWriterPerThread.updateDocuments calls indexingChain.processDocument(numDocsInRAM++, doc, isLastDoc) (index/DocumentsWriterPerThread.java:289-291); the comment above it says "Even on exception, the document is still added (but marked deleted)" (:283-288). A block of more than one document sets hasBlocks on the segment (:296-299). After a non-aborting exception the used doc IDs are only marked deleted (deleteLastDocs, :303-307).
2. The parent marker is written in pass 2. IndexingChain.processDocument checks the schema in pass 1 (initAndValidateFields, index/IndexingChain.java:640-642), where a doc-values type, point dimension, vector or index-options conflict throws. Only then, in pass 2, does it write the parent marker, and only for the last document of the call (:644-652). A document rejected in pass 1 never gets one (a pass-2 rejection of the call's last document does, see below); non-last documents of a block never get one either.
3. The flush-time sort compares parents. maybeSortSegment (index/IndexingChain.java:251-299): when the segment has blocks and a parent field, it loads the markers into a BitSet and wraps every sort comparator as in.compare(parents.nextSetBit(docID1), parents.nextSetBit(docID2)) (:261-274).
4. NO_MORE_DOCS becomes a doc ID. BitSet.nextSetBit returns NO_MORE_DOCS when no bit is set at or after the index (util/BitSet.java:89-93; NO_MORE_DOCS = Integer.MAX_VALUE, search/DocIdSetIterator.java:111). In the repro the markers are {1}, so nextSetBit(2) is 2147483647, and the LONG sorter reads values[docID] from an array of length maxDoc = 3 (index/IndexSorter.java:308-322). All 13 sort types the hunt tested fail, but not all at the same place: BinarySortField fails inside FixedBitSet.get (an AssertionError with -ea, "Index 33554431 out of bounds for length 1" with -da), while the 7 other types of the fact check's probe throw "Index 2147483647 out of bounds for length 3" in their comparator; the writer closes itself in all of them.
5. The failure is tragic. An exception from IndexingChain.flush is aborting (index/IndexingChain.java:301-305): DocumentsWriterPerThread.flush records it and fires a tragic event (index/DocumentsWriterPerThread.java:585-589, :594-604), so IndexWriter closes itself and rolls back to the last commit.

A bounded-exhaustive check on the real jar compared an exact model, "the flush fails if and only if the in-memory segment holds an accepted multi-document block and its last doc ID has no parent marker", against every script of 1 to 5 operations over 8 operations (an accepted single document or block; a single document, a block child or a block's last document rejected in pass 1 or pass 2): 37,448 scripts, 7,520 predicted and 7,520 observed failures, 0 mismatches (-ea); with -da, the 4,680 scripts of length 1 to 4, 740 = 740. The "never fails" model gives 740 mismatches on the same jar, so the check is not vacuous.

In words: the last operation on that segment was a pass-1 rejection of a single document or of a block's last document (doc-values type, point or vector dimension or similarity, index options, the reserved parent field name), or any rejection of a non-last block child (also an immense term or an analyzer IOException). A pass-2 rejection of a single document or a block's last document does not trigger it, because the marker is written first in pass 2. The experimental IndexWriter#addBatch path fails the same way.

• Who is hit. Applications that index nested documents as blocks (addDocuments / updateDocuments) under an index sort. For indexes created by 10.0.0 or later the parent field is mandatory for that (DocumentsWriterPerThread.java:272-281; MIGRATE: "IndexWriter requires a parent document field in order to use index sorting with document blocks"), so every such application has the configuration. On 9.x indexes it applies only if the parent field is set. addDocuments and updateDocuments are tagged @lucene.experimental (IndexWriter.java:1521, :1603, :1616); in the repro only the accepted block uses addDocuments, the rejected document goes through the non-experimental addDocument.
• What it takes. One accepted multi-document block in the in-memory segment (one per indexing thread), and a rejected document as the last operation on that segment before it is flushed. The rejections come from ordinary input: a field whose doc-values type or point dimensions conflict with an earlier document, an immense term in a nested child, an analyzer exception in a child. Any accepted document after the rejection on the same segment avoids it.
• What breaks. Loudly: the next commit(), NRT reopen or close() (with the default commitOnClose) throws, IndexWriter closes itself and every document buffered since the last commit is lost (by code reading also those in other threads' in-memory segments; the single-thread loss is executed). The committed index stays intact and a new IndexWriter opens on it (executed: with one earlier commit, that document survives the crash and a new writer adds a document and commits).
• Default vs opt-in. Index sorting and blocks are opt-in. Within that: default codec and directory, assertions on or off, one thread, three documents.
• How likely. Plausible. Rejected documents followed by a flush under an index sort do occur: the related #15125 crash was hit downstream, in a failed upgrade (OpenSearch#19660) and in what elastic/elasticsearch#153950 calls "both production failures"; #15125's own PR body shows a stack trace from a downstream engine. No public report of this crash was found in the recorded searches.

TestIndexSortRejectedLastDocument (49 lines with the license header, one test method, public API only). expectThrows checks the rejection; the test fails at w.commit() (line 42). It fails with every seed tried (DEADBEEF, 1234ABCD; -ea and -da); the test framework still randomizes the directory and codec through newDirectory(). Compared with the hunt's copy, its one comment is reworded and wrapped onto two lines so that checkGoogleJavaFormat passes with the file in the tree; the code is identical. The block is added with addDocuments, which is tagged @lucene.experimental; the rejected document goes through the non-experimental addDocument. As a regression test the method can go into TestIndexSorting unchanged.

cp $REPRO/TestIndexSortRejectedLastDocument.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestIndexSortRejectedLastDocument -Ptests.seed=DEADBEEF

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF
TestIndexSortRejectedLastDocument > testCommitAfterRejectedDocument FAILED
    java.lang.ArrayIndexOutOfBoundsException: Index 2147483647 out of bounds for length 3
        at org.apache.lucene.index.IndexSorter$LongSorter.lambda$getDocComparator$0(IndexSorter.java:322)
        at org.apache.lucene.index.IndexingChain.lambda$maybeSortSegment$1(IndexingChain.java:273)
        at org.apache.lucene.index.Sorter.lambda$sortAndLeaveUnpacked$0(Sorter.java:230)
        at org.apache.lucene.index.Sorter.sort(Sorter.java:136)
        at org.apache.lucene.index.Sorter.sortAndLeaveUnpacked(Sorter.java:238)
        at org.apache.lucene.index.IndexingChain.maybeSortSegment(IndexingChain.java:297)
        at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:305)
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:492)
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:541)
        at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:496)
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:688)
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3809)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4253)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4215)
        at org.apache.lucene.index.TestIndexSortRejectedLastDocument.testCommitAfterRejectedDocument(TestIndexSortRejectedLastDocument.java:42)

# standalone runner, unpatched jars
main 645d4fe14a20, -ea, seed DEADBEEF   tests_run=1 failures=1  ArrayIndexOutOfBoundsException: Index 2147483647 out of bounds for length 3
main 645d4fe14a20, -da, seed DEADBEEF   tests_run=1 failures=1  (same)
main 645d4fe14a20, -da, seed 1234ABCD   tests_run=1 failures=1  (same)
10.6.0 release jar, -da                 tests_run=1 failures=1  (same)

# writer state after the failure (the same steps as a plain main(), ByteBuffersDirectory; diagram probe, -ea and -da)
PROBE A commit(): threw java.lang.ArrayIndexOutOfBoundsException: Index 2147483647 out of bounds for length 3 | isOpen=false tragic=ArrayIndexOutOfBoundsException sameAsThrown=true
PROBE A after: indexExists=false
# run A2: one committed document first, then the same steps
PROBE A2 after: indexExists=true numDocs=1
PROBE A2 new writer: addDocument + commit OK, numDocs=2

# close() instead of commit(): a variant of the same shape (sort values, the conflicting field on the child, FSDirectory)
close -> java.lang.ArrayIndexOutOfBoundsException: Index 2147483647 out of bounds for length 3 ; writer open=false ; tragic=java.lang.ArrayIndexOutOfBoundsException: Index 2147483647 out of bounds for length 3
index has a commit point: false

• Contracts. When IndexWriter closes itself: the class javadoc, "if you hit an Error, or disaster strikes during a checkpoint then IndexWriter will close itself" (IndexWriter.java:162-165), and getTragicException, "closed as a side-effect of a tragic exception, e.g. disk full while flushing a new segment" (:6108-6112); every documented case is an environmental failure, none a document IndexWriter rejected itself. addDocument: "if an Exception is hit ... the index will be consistent, but this document may not have been added" (:1465-1466), and a term over MAX_TERM_LENGTH gives an IllegalArgumentException (:1480-1481). The schema rule a doc-values type conflict breaks is documented in lucene/MIGRATE.md:1557-1567 (LUCENE-9334, "Require consistency between data-structures on a per-field basis"), and upstream's test expects an IllegalArgumentException for it (TestIndexSorting.java:3537). The DocumentsWriterPerThread comment on this path: a rejected document "won't cause the DWPT to be stale nor the entire IW to abort and shutdown. In such a case we only mark these docs as deleted" (DocumentsWriterPerThread.java:392-398). setParentField: the marker "is required" to keep blocks in order under index sorting (IndexWriterConfig.java:579-589), with no @lucene.experimental tag.
• Upstream precedent. #15125 ("Handle inconsistent schema on flush with index sorts", merged 2025-09-29, first released in 10.5.1) fixed a NullPointerException in the same maybeSortSegment after a rejected document. Its test testFlushAfterInconsistentSchema (TestIndexSorting.java:3497-3556) asserts that a rejected single document followed by commit() works with an index sort (set in half of its runs), without a parent field.
• History. The parent-aware comparator and setParentField came together in #12829 (2024); its comments and reviews never discuss rejected documents.
• Tests. No upstream test combines an index sort, a parent field, an accepted block and a trailing rejection; testBlockContainsParentField rejects blocks but never has an accepted block in the segment.

The strongest case for "intended or misuse": the addDocument javadoc promises only that the index stays consistent, and the last commit is intact; addDocuments is @lucene.experimental; feeding schema-conflicting documents is an application error; and blocks that lose their parent are a known unsupported state (s1monw on #12829, about deleting only a parent).

Why it fails: every self-close IndexWriter documents is an environmental failure (an Error, a checkpoint disaster, disk full while flushing), never a document it rejected itself; Lucene's answer to such input is a non-aborting IllegalArgumentException (the schema rule in MIGRATE.md:1557-1567, the exception upstream's test expects at TestIndexSorting.java:3537, and the DocumentsWriterPerThread comment that such a rejection must not "abort and shutdown" the writer); the rejected document goes through the non-experimental addDocument; the parentless doc ID is created by Lucene's own rejection handling, the user deletes nothing; and upstream fixed #15125, a rejected document then a flush under an index sort, as a bug. The intended-lens reviewer from another model family (OpenAI GPT-5.6) rated the argument weak, and the devil's-advocate verifier refuted each part of it.

One fix was validated with gradle in a scratch worktree at 645d4fe14a20 (JDK 25, -Ptests.seed=DEADBEEF); the other options were not built here.

DocumentsWriterPerThread (both updateDocuments and updateBatch) calls a new IndexingChain.markLastRejectedDocAsParent after deleteLastDocs; IndexingChain remembers the last doc ID that has a marker and, if the last used doc ID has none, writes one (registering the parent field first if this segment has none yet). +31/-0 in two files.

// DocumentsWriterPerThread, non-aborting failure path (updateDocuments and updateBatch):
deleteLastDocs(numDocsInRAM - docsInRamBefore);
if (numDocsInRAM > docsInRamBefore) {
  indexingChain.markLastRejectedDocAsParent(numDocsInRAM - 1);
}

// IndexingChain (lastParentDocID is set wherever a parent marker is written):
void markLastRejectedDocAsParent(int lastDocID) throws IOException {
  if (parentPf == null || lastDocID <= lastParentDocID) {
    return;
  }
  if (parentPf.fieldInfo == null) {
    initializeFieldInfo(parentPf);
    parentPf.trySetValidatedFrozenFieldType();
  }
  ((NumericDocValuesWriter) parentPf.docValuesWriter)
      .addValue(lastDocID, parentField.numericValue().longValue());
  lastParentDocID = lastDocID;
}

Trade-offs

• Restores the invariant every block-aware consumer relies on (every doc ID has a parent at or after it) at the one place that breaks it. The rejected documents are deleted anyway, so the extra marker changes nothing visible; a block cut short by a rejected child becomes a deleted block.
• Cost: one int store per parent on the indexing path; the rest runs only after a rejection.
• It also registers the parent field in a segment whose only documents were rejected, which makes the minimal repros of two related hunt findings pass (a BP-reordering merge and a merge that expects the parent field in every segment). It does not touch the sorters, so the related bug where a user deletes only a block's parent still needs its own fix.

What the checks validated

• Gradle, scratch worktree at 645d4fe14a20 + the diff + the minimal test: :lucene:core:checkGoogleJavaFormat passes; :lucene:core:test with TestIndexSortRejectedLastDocument 1, TestIndexSorting 62, TestIndexWriterExceptions 40 (3 skipped), TestIndexWriterExceptions2 1, TestColumnBatchIndexing 37 and TestAddIndexes 39: 180 tests, 0 failures, BUILD SUCCESSFUL. Without the diff the minimal test fails. Not run: the whole core test suite, so treat the diff as a starting point for review, not a finished patch.
• Standalone runner: the minimal test passes with the fix with -ea and -da; the bounded-exhaustive check gives 0 failures and 0 mismatches over the 4,680 scripts of length 1 to 4 (-da), including a merge, a second commit and CheckIndex; the twin finders' repros pass; the refute-lens reviewer's port of the fix passes its own test, including forceMerge(1).

Treat documents with no following parent as their own block, in the flush-time comparator, in the merge-time MultiSorter and in Sorter.sort(LeafReader) (Sorter.java:185-199, the same parents.nextSetBit wrapper, used by SortingCodecReader.wrap).

Trade-offs

The comparator alone is not enough: a twin finder's comparator-only version makes the JUnit pass, but its model-based test still reported 15 failures in 400 runs (9 to 13 runs failed: the harness counts failure events and prints only the first 10, from 8 runs), because the flushed segment then ends with deleted parentless documents and the next merge throws in MultiSorter, which makes the same parents.nextSetBit(docID) assumption (MultiSorter.java:166-168). Changing these sites would also cover the delete-only-the-parent case.

Move the marker write ahead of initAndValidateFields.

Trade-offs

Incomplete by code reading: it covers a rejected single document or block's last document, but not a rejected non-last child, where the block's last document is never processed.

Method: code reading, with a small Alloy model. The hunt's index-sorting hunt (2026-10-25) noticed that an earlier study had judged this flush site unreachable ("a flushed segment always ends with a parent"), and that a rejected document breaks that premise. Its Alloy 6.2 model of the flush-time and merge-time doc maps with blocks produced a 3-document counterexample (an accepted block, then a failed block whose document has no parent marker), and a 4-test JUnit on main confirmed it (2 tests fail).

• An indexing-chain schema hunt (2026-10-25) found the same root by code reading; its fault-injecting model-based test reported 22 failures in 100 random runs on the stock jar (at least 11 runs failed; the 10 failures it printed are all an ArrayIndexOutOfBoundsException at flush or commit).
• An IndexWriter API fault hunt (2026-10-26) reached the same root by code reading and characterised it with a bounded-exhaustive script enumerator: 31,314 single-threaded runs, and the only failing class was this one (78 runs, all with an index sort and a parent field).
• An adjudication unit reduced the trigger to one addDocument, wrote the minimal test, checked an exact trigger model against the real jar over 37,448 scripts, and extended the fix to the addBatch path.