IndexWriter accepts doc-values updates of the reserved parent field, which breaks index-sorted document blocks

With an index sort and IndexWriterConfig#setParentField, IndexWriter writes a reserved numeric doc-values mark on the last document of every block, and CheckIndex and merges use those marks to find block boundaries. A document that contains the field is rejected as "a reserved field", but IndexWriter's doc-values update methods check only index-sort fields, so an application can overwrite or erase a mark. Giving a child a mark splits its block inside an already sorted segment: one updateNumericDocValue call on a committed 2-document block leaves a segment that CheckIndex rejects (segment has indexSort=<long: "sort"> but docID=0 sorts after docID=1), with no merge. Removing a parent's mark leaves its block unmarked: if that block was the last one of its segment and every document there has a sort value, the next merge throws EOFException and closes the writer, and once the update is committed later writers fail the same way; otherwise the orphans silently join the next block. Every release from 9.10.0 to 10.6.0 and main is affected. Only an application that writes the field it configured as its parent field can trigger it, and cross-model reviews call that out of contract, so it reads best as missing input validation. Low priority; a 16-line check fixes it.

• Index sort: IndexWriterConfig#setIndexSort. Every segment keeps its documents in that order, and CheckIndex checks it.
• Document block: documents added together with addDocuments, children first and the parent last. IndexWriter guarantees that sorting and merging never reorder or split a block.
• Parent field: IndexWriterConfig#setParentField("_parent"). IndexWriter writes _parent = -1 on the last document of every block (IndexingChain.java:166-169, :604-609, :645-651). A block sorts by its parent's value. From 10.0.0 a parent field is required to combine blocks with an index sort (MIGRATE.md:875-881).

1. A document that names the parent field is rejected: "_parent" is a reserved field and should not be added to any document (IndexingChain.java:616-619, :748-750).
2. The doc-values update methods have no such check. updateNumericDocValue (IndexWriter.java:1962), updateBinaryDocValue (:1996), updateSortedNumericDocValue (:2037) and buildDocValuesUpdate (:2081, which serves updateDocValues, softUpdateDocument(s) and tryUpdateDocValue) reject only index-sort fields (:1965, :2003, :2041, :2098). Once a block has been indexed, _parent is an existing doc-values-only NUMERIC field, which FieldInfos.verifyOrCreateDvOnlyField (FieldInfos.java:562-647) accepts.
3. CheckIndex.testSort walks only the documents that carry the parent field and compares each block's parent with the next (CheckIndex.java:1255-1282). Merges build block boundaries from the field's current values and give each document the sort value of the next marked document (MultiSorter.java:52-63, :166).

Before the update the segment is doc 0 = child (sort=9, no mark) and doc 1 = parent (sort=1, _parent=-1): one block, trivially sorted. updateNumericDocValue(id:child, "_parent", -1) gives doc 0 a mark, and closing the writer commits it as a doc-values generation of the same segment. testSort now sees two blocks, compares 9 with 1 and reports segment has indexSort=<long: "sort"> but docID=0 sorts after docID=1. Nothing re-sorts the segment. If the marked child had its parent's sort value, CheckIndex would stay clean.

updateDocValues(term, new NumericDocValuesField("_parent", (Long) null)) on a parent is accepted too. Its block's documents then have no mark of their own, and the next merge gives them the sort value of the next marked document of their segment (MultiSorter.java:166). What follows depends on the layout:

• The block was the last one of its segment, and every document there has a sort value (the layout of every earlier test that threw): nextSetBit returns NO_MORE_DOCS, and the first forceMerge(1) reads past the end of the sort values and throws RuntimeException(EOFException) (from DirectReader via IndexSorter$LongSorter; a plain ByteBuffersDirectory throws IndexOutOfBoundsException). The writer closes with a tragic exception. If the update was committed before the merge, writers opened later fail their first forceMerge(1) the same way (three tried), while readers still open and search; if the merge ran before any commit, the tragic close drops the update and a new writer merges fine.
• Another block follows in the same segment, or some documents have no sort value: nothing throws and CheckIndex stays clean, but the orphans silently become members of the next block.

After a mark is added and a merge runs, a block join can attribute a child to another root.

Without an index sort, a verifier's control stayed clean through later merges; merge-time reorderers such as BPIndexReorderer, which also read the mark, were not tested. Changing a parent's mark to another non-null value is harmless. Once a block exists, updateBinaryDocValue and updateSortedNumericDocValue already fail with a type-mismatch message, because the field is NUMERIC. On an index with no documents yet, updateDocValues even creates _parent with the type it is given; after a binary one, the writer's next block fails with cannot change field "_parent" from doc values type=BINARY to inconsistent doc values type=NUMERIC.

• Who is exposed. Only applications that (a) use index sorting with document blocks and setParentField, and (b) send a doc-values update (updateNumericDocValue, updateDocValues, softUpdateDocument(s) or tryUpdateDocValue) that names the field they configured as the parent field. The default configuration has no parent field. Untrusted documents cannot trigger it (the document path rejects the field), but an application that passes user-chosen field names to these update methods would pass the trigger on; that was not ruled out.
• Silent at the call. The update returns a sequence number like any other. The damage shows later: CheckIndex rejects the committed segment, or the next merge fails and closes the writer.
• Lasting, once committed. In the tested remove-a-mark layouts (the unmarked block last in its segment, a sort value on every document, forceMerge(1)), once the update had been committed each writer opened later failed its first forceMerge and closed, while reads kept working (three reopened writers tried). When the merge ran before a commit, the tragic close dropped the update and a new writer merged fine. A natural merge chosen by the merge policy was not tested.
• Wrong block membership. After a mark is added and a merge runs, ToParentBlockJoinQuery mapped a child to another block's root; a removed mark that does not make the merge fail silently regroups documents into the next block.
• Scale. Two documents and one call; deterministic.
• How likely. Low: no legitimate workflow needs to write the parent field. A code read of Elasticsearch, OpenSearch and Solr found no default-configuration path that sends such an update (Solr only if an administrator declares its internal _is_root_ field as an in-place-updatable docValues-only field; not executed).
• Not claimed. Wrong top hits from sort-based early termination were never verified.

Public API only (IndexWriter, IndexWriterConfig, CheckIndex.testSort, ByteBuffersDirectory; the test framework's LuceneTestCase and TestUtil.getDefaultCodec()). The test sits in org.apache.lucene.index next to the core tests but uses no package-private API. It passes if the update is rejected with IllegalArgumentException (the fixed behaviour) or if the segment stays in index-sort order. The update call itself does not throw on stock code; only the sort check fails.

The same test is the suggested regression test. With two lint-only edits (the unused catch parameter e renamed to _, the 133-column comment wrapped) it passes the core lint and still fails on stock main and passes with the fix.

# gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true (environment lines cut)
TestParentFieldDocValuesUpdate > testUpdateParentFieldOfChildKeepsIndexSorted FAILED
    java.lang.AssertionError: expected null, but was:<org.apache.lucene.index.CheckIndex$CheckIndexException: segment has indexSort=<long: "sort"> but docID=0 sorts after docID=1>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:20EFBA983838F9D8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotNull(Assert.java:756)
        at org.junit.Assert.assertNull(Assert.java:738)
        at org.junit.Assert.assertNull(Assert.java:748)
        at org.apache.lucene.index.TestParentFieldDocValuesUpdate.testUpdateParentFieldOfChildKeepsIndexSorted(TestParentFieldDocValuesUpdate.java:57)

1 test completed, 1 failed

# -Ptests.asserts=false: the same failure and message
# with the fix: the update is rejected with IllegalArgumentException and the test passes (1 test, 0 failures)

# the remove-a-mark case (the hunt's second test, main ea57360912ed, -ea; its unmarked block is the last of its segment): the first forceMerge(1) after the update
java.lang.RuntimeException: java.io.EOFException
    org.apache.lucene.util.packed.DirectReader$DirectPackedReader1.get(DirectReader.java:204)
    org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$5.longValue(Lucene90DocValuesProducer.java:1040)
    org.apache.lucene.index.IndexSorter$LongSorter.lambda$getComparableProviders$0(IndexSorter.java:298)
    org.apache.lucene.index.IndexSorter$ComparableValues$1.setTopValue(IndexSorter.java:103)
    [...] MultiSorter

• Reserved, enforced by validation. IndexingChain.java:616-619 throws for a document that names the field; TestIndexSorting.testBlockContainsParentField asserts the message. The update API is a second public write path to the same field without that check.
• The javadoc states the purpose (IndexWriterConfig.java:579-589): IndexWriter "will add an internal field to every root document", "In order to maintain the API guarantee that the document order of a block is not altered by the IndexWriter a marker for parent documents is required."
• The same class is guarded, as a bug. LUCENE-7581 (2016) made IndexWriter reject doc-values updates of index-sort fields; its CHANGES entry says "since this would lead to corruption". #16565 (2026-08) added the missing guard to updateBinaryDocValue. TestIndexSorting.testBadDVUpdate asserts all four entry points.
• The design discussion treats the field as IndexWriter's (#12829 review): its author called it "a field name that IW fully controls it's type and content" and "a purely internal field", and wrote "you can't use the parent field I think that is reserved for the top level"; a reviewer asked "are there any other restrictions on the use of the field? Could we clear up the javadoc to be more explicit?" and another answered "+1 to tighten up the language". Doc-values updates were not discussed. The #15960 commit message says the reserved mechanism existed "to guard the field name from external use".
• Nothing permits it. No javadoc, MIGRATE entry, CHANGES entry or test covers a doc-values update of the parent field. The update javadoc (IndexWriter.java:1950-1961) asks only for an existing doc-values-only field, which _parent is.

The cross-model reviews (OpenAI GPT-5.6, refute and works-as-designed lenses) made the strongest case: the parent field is documented as internal (IndexWriterConfig.java:579-589; MIGRATE.md:879, "The internally used parent field"), IndexWriter owns its values, and the application chose the name itself. Writing it through a generic API is mutation of writer-owned metadata, garbage in, garbage out; a missing check makes this hardening, not a correctness bug. Both rated it "not real", medium, while reproducing it and agreeing with the root cause and the fix. The cross-model fact check of this doc rated this case strong: a maintainer could reasonably close it as out of contract while still accepting the check as hardening.

Why it does not settle it: Lucene answers "callers might write the reserved field" with validation, already on the document path, and it fixed the closest analogue (updates of index-sort fields) as a bug; those fields belong to the application, so the precedent shows how Lucene treats updates that break the sort invariant, not that writer-owned fields are in contract. The update methods document their preconditions and _parent meets them; nothing in their javadoc excludes fields named in the writer config, and the other such field, the soft-deletes field, is updated through these very methods (a designed use), so the javadoc gives callers no hint. And the failure is silent at the call. What survives is the scope: it is missing input validation, with low likelihood, and only an application writing its own parent field triggers it. Verdict (adjudication): confirmed, medium.

Option A was run through gradle in a scratch worktree of 645d4fe14a20. B and C are alternatives that were not run. Hardening MultiSorter is not a fix: the add-a-mark case breaks the segment at commit, with no merge.

A private ensureNotParentField(field) helper in IndexWriter, called first in updateNumericDocValue, updateBinaryDocValue and updateSortedNumericDocValue, and for every field in buildDocValuesUpdate (which serves updateDocValues, softUpdateDocument(s) and tryUpdateDocValue). The message mirrors the document path's "is a reserved field".

private void ensureNotParentField(String field) {
  String parentField = config.getParentField();
  if (parentField != null && parentField.equals(field)) {
    throw new IllegalArgumentException(
        "\"" + field + "\" is a reserved field and its doc values cannot be updated");
  }
}
// called first in updateNumericDocValue, updateBinaryDocValue and
// updateSortedNumericDocValue, and for each field in buildDocValuesUpdate

Trade-offs

• Runs before verifyOrCreateDvOnlyField, so the binary and sorted-numeric methods now say "reserved field" instead of the type-mismatch message, and updateDocValues cannot create the field with another type on an index without documents (executed).
• Changes which error comes first: updateBinaryDocValue(parentField, null) now gets the reserved-field error instead of today's "cannot update a field to a null value" (executed), and an updateDocValues field named after the parent field is rejected before its doc-values type is checked.
• The null-safe comparison keeps today's behaviour for a writer without a parent field and for a null field name.
• Rejects whether or not an index sort is set, like the document path. Without a sort a verifier's control stayed clean (merge-time reorderers such as BPIndexReorderer, which also read the mark, were not tested), so a maintainer could limit the check to sorted indexes; rejecting always is simpler, keeps the field writer-owned and also protects code that reads the mark without an index sort.
• Behaviour change: an application that writes its parent field through these methods now gets an exception; no upstream test or javadoc relies on that call.
• Open PR #16810 edits the same four guard blocks (it merges the index-sort checks into one helper).
• Worth adding: one sentence in the setParentField and update javadocs that the parent field cannot be updated (a reviewer asked for more explicit javadoc in #12829).

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (--offline, seed DEADBEEF). Stock: the repro fails with assertions on and off.
• With the fix: the repro plus TestIndexSorting, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestSortedNumericDocValuesUpdates, TestMixedDocValuesUpdates, TestIndexWriter, TestAddIndexes, TestColumnBatchIndexing, TestIndexOptions, TestSoftDeletesRetentionMergePolicy and TestSoftDeletesDirectoryReaderWrapper pass with assertions on and off (12 suites, 383 tests, 0 failures, 3 skipped).
• The whole org.apache.lucene.index test package with the fix (215 suites, 1396 tests, 0 failures, 56 skipped), and two parent-field suites in other modules: TestBPIndexReorderer (8 tests) and TestIndexSortBackwardsCompatibility (68 tests).
• Lint with the fix: checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass once the repro gets two lint-only edits (the unused catch parameter renamed to _, the long comment wrapped); with the edits the repro still fails on stock code and passes with the fix.
• Earlier, the hunt's equivalent fix sketch on main ea57360912ed passed the finder's 2 tests, the first verifier's 5 (two seeds), 4 more probe tests and 242 existing tests.

All four update paths call it (IndexWriter.java:1964, :2010, :2048, :2097) and it already knows parentFieldName (FieldInfos.java:384-390), so one check covers them.

Trade-offs

Puts an IndexWriter policy into a schema-verification method; for the binary and sorted-numeric methods the index-sort message still comes first.

#16810 replaces the four copies of the index-sort check with one ensureNotIndexSortField helper; generalising it to "fields IndexWriter owns" (sort fields of the config and of existing segments, plus the parent field) keeps one guard.

Trade-offs

Ties this fix to an open, unreviewed PR.

It would change only the remove-a-mark case (the merge-time end state shared with LFV-011).

Trade-offs

The add-a-mark case breaks the segment at commit, with no merge involved, so the update must be rejected anyway.

Method: an Alloy 6.2 model plus code reading, then JUnit (the hunt's "semi-formal" tier).

• A hunt unit (index sorting with document blocks) modelled the flush sort and the merge of blocks in Alloy (SortBlocks.als). While modelling which inputs can change parent marks, the finder read IndexWriter's update guards (IndexWriter.java:1962-2110 at the time) and wrote the predicate goodInputsAfterParentDVUpdate: marks may differ from the block structure after a doc-values update.
• The check MergeAfterParentDVUpdate (blocks preserved by the merge) was satisfiable: a 2-document counterexample in which a parent mark no longer matches its block.
• The finder's JUnit tests (add a mark; remove a mark) then failed on the stock jars of main, 10.5.2, 9.12.3 and 9.10.0, with assertions on and off, and passed with a fix sketch.
• It counts as semi-formal because the model encoded a hypothesis formed while reading the code and the tool then showed its consequence.