A document block rejected at a child can leave a segment without the parent field; if the merge policy keeps it, later merges close IndexWriter

With IndexWriterConfig#setParentField and no index sort, IndexWriter's row-by-row indexing path creates the parent field in a segment only when it indexes the last document of a block. When it rejects a block at a child with a documented, non-aborting exception, such as a term longer than MAX_TERM_LENGTH, the documents it already indexed stay in the segment as deleted documents. If that block was the only content of its in-memory segment, for example the first thing indexed after a commit, the segment is flushed with no live documents and no parent field. Merge policies normally drop it; SoftDeletesRetentionMergePolicy with a matching retention query, or any policy whose keepFullyDeletedSegment returns true, keeps it. From then on every merge that combines it only with segments holding no multi-document block throws IllegalStateException ("expected parent doc field ... across all segments") and closes IndexWriter, losing uncommitted documents, also with each new writer on the index, until forceMergeDeletes() removes the segment. Readers that hold it next to other segments fail FieldInfos.getMergedFieldInfos, so UnifiedHighlighter fails, also with an index sort. The data stays intact and CheckIndex reports the index clean. Introduced by #12829 in 9.10.0; every release since and main are affected.

• Document block: IndexWriter.addDocuments adds several documents together, children first and the parent last; the javadoc promises that a reader sees "all or none of the documents" (IndexWriter.java:1496-1497).
• Parent field: with IndexWriterConfig.setParentField("_parent"), "IndexWriter will add an internal field to every root document" (IndexWriterConfig.java:580-581): the last document of each block, and every single document. It lets index sorting and merge-time reordering keep blocks together.
• Fully deleted segment: a segment whose documents are all deleted. IndexWriter drops it unless the merge policy's keepFullyDeletedSegment returns true (IndexWriter.java:6429-6432, MergePolicy.java:761-767). SoftDeletesRetentionMergePolicy returns true when its retention query matches any document of the segment, evaluated with hard deletes ignored (SoftDeletesRetentionMergePolicy.java:80-93).
• Merged view: without an index sort, a merge of two or more segments first wraps them in one SlowCompositeCodecReaderWrapper (for optional reordering), whose field infos come from FieldInfos.getMergedFieldInfos. That method requires every segment to name the same parent field.

1. Registered at the last document only. On the row-by-row path that addDocument(s) and updateDocument(s) use, IndexingChain.processDocument queues and indexes the parent field only if (parentPf != null && lastDocInBlock) (IndexingChain.java:604-608, :645-652). (The column-batch path, addBatch, creates it eagerly, :824-833.)
2. A rejected block stays behind, deleted. DocumentsWriterPerThread.updateDocuments indexes a block's documents one by one and, on a non-aborting exception at a child, marks those it already processed deleted (deleteLastDocs, DocumentsWriterPerThread.java:303-306); hasBlocks is set only when the whole block succeeded (:297-298). The writer stays open.
3. An empty segment without the field. If the block was the only content of its in-memory segment, the segment is flushed with 0 live documents, no parent FieldInfo and hasBlocks=false. Kept by the merge policy, it is committed.
4. The merged view checks it. Unless some input has a multi-document block and some input lacks the parent field (independent checks, IndexWriter.java:5548-5555), mergeMiddle builds the merged view even when no reorderer is configured (:5557). For two or more inputs, its constructor calls getMergedFieldInfos (SlowCompositeCodecReaderWrapper.java:96), and getAndValidateParentField throws when one segment says _parent and another says null, also for a segment with no live documents (FieldInfos.java:209, :220-238). The soft-deletes field in the same method tolerates null (:203-208).
5. Tragic. The merge exception closes IndexWriter (tragicEvent(t, "merge"), IndexWriter.java:5112), and uncommitted documents are lost. The bad segment is still in the last commit, so the next writer fails the same way. UnifiedHighlighter (UnifiedHighlighter.java:715-730) and MatchRegionRetriever (:479) call getMergedFieldInfos on the top-level reader, so they fail on readers that hold the segment next to a segment with the parent field (a one-segment reader is returned unchecked, FieldInfos.java:200-201).
6. Escape hatches. A singleton merge never builds the view (SlowCompositeCodecReaderWrapper.java:49-57), so forceMergeDeletes() or a delete-reclaiming merge of the empty segment alone removes it. From 9.11.0 on (#13128), a merge that includes a multi-document-block segment skips the view and succeeds.

A second trigger, found while writing this doc, comes from caller misuse: a single document that contains the reserved field name ("_parent" is a reserved field, IndexingChain.java:617-620, rejected on purpose) is rejected before the parent field is initialized (:641) and leaves a 1-document segment with no fields at all; forceMerge(1) then fails the same way on main and 10.5.2. It carries less weight than the oversized term.

• Who. Opt-in on two counts: a parent field set without an index sort, and a merge policy that keeps fully deleted segments, typically SoftDeletesRetentionMergePolicy whose retention query matches the rejected documents (match-all, or a range that covers them), or any policy whose keepFullyDeletedSegment returns true. The default configuration has neither.
• Trigger. A block rejected at a child by a documented non-aborting exception: an oversized term, or a doc-values type conflict in the child (both executed); also, by caller misuse, a single document carrying the reserved parent field name (executed). It must be the only content of a fresh in-memory segment, for example the first thing indexed after a commit or refresh; a rejected block that shares its segment with an accepted document does no harm. A randomized stress run hit it without targeting in 12 of 3,200 rounds.
• Writes. Each merge that pairs the empty segment only with segments built from single documents throws and closes IndexWriter: in a probe, another client's 3 acknowledged documents were lost on each of two failures. Natural merges hit it (with TieredMergePolicy defaults the first natural merge after the rejection failed), and so does the merge-on-refresh of an NRT reopen. It repeats with each new writer, even with a plain TieredMergePolicy, until forceMergeDeletes(), or a merge with a multi-document-block segment, removes it. TieredMergePolicy may also reclaim the segment on its own when deletes are a large share of the index.
• Reads. On a reader with two or more segments that holds this one (a one-segment reader is not checked), FieldInfos.getMergedFieldInfos throws, so every UnifiedHighlighter request with the default offset source fails; search itself works. This also happens with an index sort. A SoftDeletesDirectoryReaderWrapper view hides the empty segment and highlighting works.
• Not affected. Committed data stays intact and CheckIndex reports no problems. With an index sort, merges skip the merged view and succeed, which removes the segment.
• Scale and likelihood. No scale needed (the repro submits 5 documents and writes 3 segments); rare in practice because all the preconditions must line up.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: forceMerge(1) after a block rejected at a child threw java.lang.IllegalStateException: expected parent doc field to be "_parent " across all segments but found a segment with different field "null"
       org.apache.lucene.index.TestRejectedBlockSegmentWithoutParentField.testForceMergeAfterBlockRejectedAtChild(TestRejectedBlockSegmentWithoutParentField.java:52)
     Caused by: java.lang.IllegalStateException: expected parent doc field to be "_parent " across all segments but found a segment with different field "null"
       org.apache.lucene.index.FieldInfos.getAndValidateParentField(FieldInfos.java:226)
       org.apache.lucene.index.FieldInfos.getMergedFieldInfos(FieldInfos.java:209)
       org.apache.lucene.index.SlowCompositeCodecReaderWrapper.<init>(SlowCompositeCodecReaderWrapper.java:96)
       org.apache.lucene.index.SlowCompositeCodecReaderWrapper.wrap(SlowCompositeCodecReaderWrapper.java:56)
       org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5557)
       [...]

• Contracts. IndexWriter#addDocument: "if an Exception is hit (for example disk full) then the index will be consistent, but this document may not have been added" and "each term in the document can be no longer than MAX_TERM_LENGTH in bytes, otherwise an IllegalArgumentException will be thrown" (IndexWriter.java:1465-1466, :1480-1481). addDocuments: "Atomically adds a block of documents ... such that an external reader will see all or none of the documents" (:1496-1497); that promise itself is kept, since the block's indexed documents are deleted and no reader sees them; what breaks is "the index will be consistent", because the leftover segment later makes merges fail. MergePolicy#keepFullyDeletedSegment: a segment for which it returns true "should be kept even if it's fully deleted", useful "if the merge policy implements retention policies for soft deletes" (MergePolicy.java:761-763). Nothing documents that a rejected document can make merges fail.
• Tests. TestIndexWriterExceptions keeps fully deleted segments on purpose: "we also need to keep fully deleted segments since otherwise we clean up fully deleted ones and if we flush the one that has only the failed document the docFreq checks will be off below" (TestIndexWriterExceptions.java:806-811). TestSoftDeletesRetentionMergePolicy asserts that the policy keeps fully soft-deleted segments whose documents match (:80-97). No upstream test combines a parent field, a rejected document and a kept fully deleted segment.
• Intent of the check. In the #12829 review, a reviewer (@mikemccand) asked of getAndValidateParentField: "As far as we know, users should never be able to do anything that would trigger this? This means we have some bug that allowed broken schema into the index?", and the author (@s1monw) answered "yeah I think so too". #13016 requires the parent field to be configured before any document is added, "to prevent documents without the parent field from being indexed".
• Precedent. CHANGES 9.11.0: "GITHUB#13369: Fix NRT opening failure when soft deletes are enabled and the document fails to index before a point field is written" (CHANGES.txt:2911-2912): the same class (a non-aborting failure plus soft deletes leaving field infos that do not match the segment), fixed as a bug.

Strongest case for "not a Lucene bug": the retention policy, or the custom policy, chose to keep a segment whose documents are all hard-deleted, so the policy should be fixed (the sibling registry row LFV-228 shows the same "retention ignores hard deletes" pattern in another method of SoftDeletesRetentionMergePolicy).

Why it fails: a plain keepFullyDeletedSegment() == true policy fails the same way with no document soft-deleted, and the independent reproducer and the devil's-advocate verifier reproduce it with no soft-deletes field at all; once the segment is committed, a writer with the default TieredMergePolicy fails too; multi-segment readers fail regardless of the merge policy; and the inconsistent segment is written by IndexWriter itself, from a rejection it documents as leaving the index consistent. The other readings fail as well: keepFullyDeletedSegment "is for testing" (the stock SoftDeletesRetentionMergePolicy implements it and core tests rely on it); "a parent field without an index sort is optional" (it is legal, and the reader-side failure happens with an index sort too); "huge tokens are the application's job" (that is the security threat model, and the writer rejects the term as documented). The devil's-advocate verifier and the cross-model refute review rated the intended-behaviour case weak. A later cross-model fact check of this doc rated it per path: weak for the retention policy, moderate for a custom keepFullyDeletedSegment() == true policy and for the reader-side check (a deliberate schema guard), and strong for the reserved-name trigger, which is misuse. Verdict: a defect for the oversized-term trigger.

Two fixes were applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF) with the paste-ready test and the 21 closest upstream classes in core, misc, highlighter and backward-codecs; both were also shadow-run with the standalone runner, with the single-document probe. Either makes every repro pass. Which one, or both, is the maintainers' call: a maintainer may well prefer D, or A + D (prevention plus tolerance), which was not applied or tested as one patch.

The finder's sketch, +5 lines in FieldInfos.java. Such a segment contributes no document to a merge or a reader, so it cannot break block structure.

String field = ctx.reader().getFieldInfos().getParentField();
if (field == null && ctx.reader().numDocs() == 0) {
  // a segment without live documents (for example one that holds only documents of a
  // rejected block, which never got the parent field) cannot contribute any document
  continue;
}

Trade-offs

• Fixes merges, the reader-side getMergedFieldInfos / UnifiedHighlighter failure and the single-document trigger. Because the check runs at merge and read time, it should also let a fixed release open and merge indexes that already hold such a segment: the patched writer still writes the empty segment, and a new writer then merges it fine (testNewWriterOnCommittedIndex). No index written by an old release (9.10.0 to 10.5.2) was opened under the patch, and A does not rewrite the index.
• A segment with live documents and no parent field still fails, so two related paths that end in the same check from other causes (addIndexes(Directory) of an index without a parent field into a writer with one; reopening a field-less index with a parent field) are not covered; they were seen once by the devil's-advocate verifier and are not part of this report.
• It adds a numDocs() call per segment: SegmentReader returns a stored count (SegmentReader.java:234-237), but IndexReader.numDocs() "may run in O(maxDoc)" for other implementations (IndexReader.java:295-296), and getMergedFieldInfos accepts any reader. The check then depends on live-document counts while the empty segment's other field metadata is still merged; no targeted test covers custom leaf readers or unusual OneMerge.wrapForMerge wrappers. The empty segment stays until some merge drops it.

What the checks validated

• Stock: the paste-ready test fails 4 of 4; with the independent reproducer's classes 13 tests, 9 failed (assertions on and off).
• With A: 13 of 13 pass with assertions on and off. With assertions on, the 21 closest upstream classes (TestFieldInfos, TestSoftDeletesRetentionMergePolicy, TestIndexWriterExceptions, TestIndexWriterExceptions2, TestAddIndexes, TestIndexSorting, TestIndexWriter, TestIndexWriterMergePolicy, TestIndexWriterForceMerge, TestSoftDeletesDirectoryReaderWrapper, TestColumnBatchIndexing, TestIndexOptions, TestIndexWriterDelete, TestFilterMergePolicy, TestMergePolicy, TestTieredMergePolicy, TestLogMergePolicy; misc TestBPReorderingMergePolicy, TestBPIndexReorderer; highlighter TestUnifiedHighlighter; backward-codecs TestIndexSortBackwardsCompatibility) pass: 660 tests, 18 skipped, 0 failures (673 tests in 24 classes counting the 3 repro classes). checkGoogleJavaFormat passes.
• With assertions off, a later fact check ran the paste-ready test with TestFieldInfos, TestSoftDeletesRetentionMergePolicy and TestIndexWriterExceptions under A: 64 tests, 3 skipped, 0 failures. The single-document probe passes with A shadowed. The full :lucene:core:test suite was not run.

+10/-5 in IndexingChain.processDocument: initialize the parent field's FieldInfo with the segment's first document, before its own fields, as the column-batch path already does (IndexingChain.java:824-833), instead of queueing it only for a block's last document.

if (parentPf != null) {
  if (parentPf.fieldInfo == null) {
    parentPf.schema.resetJustDocId(docID);
    initializeFieldInfo(parentPf);
    parentPf.trySetValidatedFrozenFieldType();
  }
  if (lastDocInBlock) {
    parentPf.schema.resetJustDocId(docID);
  }
}

Trade-offs

• Prevents new bad segments for both triggers and follows #13016's intent that no segment lacks the parent field; it keeps the per-segment schema invariant and does not relax FieldInfos, which a maintainer may prefer.
• Does not help an index that already holds such a segment: the check is unchanged (code reading).
• It creates the parent FieldInfo, with numeric doc values and a NumericDocValuesWriter (IndexingChain.java:1372, :1423), in a segment that may never get a parent value (all its documents rejected). The suites pass, but no focused CheckIndex or assertion test covers such a segment.
• With an index sort, block-aware sorting is used only when the segment has blocks (IndexingChain.java:261), so an all-rejected segment's empty parent field is not read there (code reading); the index-sort classes run were TestIndexSorting and TestIndexSortBackwardsCompatibility.

What the checks validated

• The paste-ready test passes 4 of 4 with assertions on and off; with assertions on, the same 21 upstream classes pass: 660 tests, 18 skipped, 0 failures (664 tests in 22 classes with the repro class); checkGoogleJavaFormat passes; the single-document probe passes with D shadowed.
• The independent reproducer's classes were left out: they assert the bug's precondition (a segment without the parent field), so they fail by design once that segment is no longer written.

Also covers the two related paths above.

Trade-offs

It would let segments whose live documents carry no parent marker merge into a segment that has the field, which #13016 set out to prevent ("documents without the parent field ... later to be treated as child documents upon merge").

Fixes merges for writers without a reorderer.

Trade-offs

Leaves the reader-side failure, and OneMerge.reorder (MergePolicy.java:325) has no way to say "not needed" without an API change.

Method: randomized stress testing with a replay oracle, then code reading (non-formal tier).

• The hunt unit drove IndexWriter's public API with random operations and one injected fault per round, and checked every acknowledged operation against a replay by sequence number. Stage 2 added soft deletes with SoftDeletesRetentionMergePolicy, parent-field document blocks, rejected blocks, doc-values updates and merges on full flush (8 seeds x 400 rounds).
• 12 of 3,200 rounds ended with an unexpected tragic close carrying this IllegalStateException; in 9 of them the injected fault never fired, so the failure came from ordinary API calls.
• The finder traced it by reading the parent-field registration and the merge path, and wrote a deterministic 3-test public-API JUnit; this doc's 4-test version narrows the claims and adds the reader-side test.
• Found 2026-10-26 on main a9779ef3c193; then checked by a cross-model review (reading only), an independent reproducer, a refute review, a devil's-advocate verifier, an adjudication, a follow-up (history, prior reports, security) and an audit.