IndexWriter accepts doc-values updates of a field that a custom index sort reads, so a segment can fall out of its declared order

IndexWriter refuses in-place doc-values updates of a field "involved in the index sort", because the segment would then no longer be in its declared order. It decides which fields are involved from SortField#getField alone (IndexWriterConfig.java:512-513). Since LUCENE-9330 (8.6.0) a custom SortField#getIndexSorter can read any field, and since 10.5.0 BinarySortField's javadoc explicitly lets a subclass derive its key "from the stored bytes and/or other doc-values fields of the segment". An update of such a field is accepted. In the repro the key is (price, sku), declared on sku; a (5), b (7), c (8) are flushed in that order and updateNumericDocValue(id:c, price, 1) is accepted. The segment keeps the order a, b, c while still declaring the sort: CheckIndex at level 2 or higher reports segment has indexSort=<binary: "sku"> but docID=1 sorts after docID=2, a top-1 search sorted by the index sort returns a instead of c because it compares no document after the first non-competitive one, and after adding d (6) and forceMerge(1) the merged segment stores a, d, b, c instead of c, a, d, b. It is deterministic and needs 3 documents. This BinarySortField form fails on 10.5.0-10.5.2 and on main; the generic form (a custom IndexSorter that reads another field) fails on every release from 8.6.0 to 10.5.2 and on main. 10.6.0 has the same code (read). It needs an opt-in index sort with a custom sort key that reads another doc-values field, plus an in-place update of that field; default configurations are not affected.

• Index sort: with IndexWriterConfig#setIndexSort, every flushed and merged segment stores its documents in that order and records the sort in its segment info. Searches and merges rely on the recorded order.
• Custom sort key: since 8.6.0 a SortField supplies its own IndexSorter, which gets a LeafReader and so can read any field. BinarySortField (10.5.0) sorts by bytes; a subclass may override getSortKeyDocValues to derive the key from other doc-values fields, and registers a SortFieldProvider so the sort can be read back on merge.
• In-place doc-values update: updateNumericDocValue, updateBinaryDocValue, updateSortedNumericDocValue, updateDocValues, tryUpdateDocValue and softUpdateDocument(s) change a document's doc values without reindexing it. The document keeps its position in the segment.

1. setIndexSort accepts any sort field with an IndexSorter (IndexWriterConfig.java:506-510) and records the protected fields as SortField::getField only (:512-513). getIndexSortFields is documented as "Returns the field names involved in the index sort" (LiveIndexWriterConfig.java:417-420).
2. Every update path rejects only those names, with "cannot update docvalues field involved in the index sort": IndexWriter.java:1965 (updateNumericDocValue), :2003 (updateBinaryDocValue), :2041 (updateSortedNumericDocValue) and :2098 (buildDocValuesUpdate, shared by updateDocValues, tryUpdateDocValue and softUpdateDocument(s)).
3. In the repro the key reads price but is declared on sku (BinarySortField#getIndexSorter passes getSortKeyDocValues to the sorter, BinarySortField.java:178-182). The update of price is accepted, the documents do not move, and the segment still declares the sort.
4. Code that trusts the declared order goes wrong. TopFieldCollector checks per segment whether the search sort is a prefix of the index sort and then, once the queue is full, compares no document after the first non-competitive one (TopFieldCollector.java:59, :93-104, :151-172; below the hit-count threshold, 1000 by default, it keeps counting hits without ranking them). MultiSorter merges segments on the rule "The documents for each incoming leaf reader must already be sorted by the same sort!" (MultiSorter.java:35-40). CheckIndex reports the first inversion (CheckIndex.java:1274-1282), at level 2 or higher only (:1114, :1146-1149).
5. The writer can find the fields a sorter reads, and does so elsewhere: when a doc-values field is first indexed into a segment, IndexingChain#initializeFieldInfo calls validateIndexSortDVType for it (IndexingChain.java:1381-1385), which probes each sorter with an empty reader and type-checks whatever field it asks for (:1635-1727). The update guard does not use that.

Key = (price, sku), declared on sku. Add c (8), a (5), b (7): the segment stores a, b, c. updateNumericDocValue(id:c, price, 1) is accepted, so docs 0, 1, 2 are a (5), b (7), c (1). CheckIndex level 1: clean, the sort check does not run; level 2: segment has indexSort=<binary: "sku"> but docID=1 sorts after docID=2. Top-1 sorted by the index sort: a, after reading the keys of docs 0 and 1 only; the same sort with FIELD_DOC appended, which is not a prefix of the index sort and compares every document: c. Add d (6) and forceMerge(1): the merged segment stores a, d, b, c; the declared order is c, a, d, b.

• Who. Applications that configure an index sort (opt-in) with their own sort key that reads a doc-values field other than the declared one, and that update that field in place: a BinarySortField subclass overriding getSortKeyDocValues (10.5.0+, @lucene.experimental), or any custom SortField#getIndexSorter (8.6.0+). The built-in sort fields read only their declared field, so this gap does not apply to them (on 10.5.x a BinarySortField's declared field was itself updatable, a separate gap fixed by #16565). Lucene's own tests derive keys only from the sort's own field (TestBinarySortField's MinValueSortField javadoc: "no companion field is needed").
• Trigger. Deterministic once both conditions hold: an accepted update of such a field breaks the segment whenever it changes the key order of documents within that segment. Three documents show all three symptoms; no concurrency; assertions on or off. A SORTED_NUMERIC companion (for example the minimum of a multi-valued numeric field) is updatable too, through updateSortedNumericDocValue, which is on main and the 10.x branches and was released in 10.6.0 (#16581).
• Effect. Silent when written. Sorted top-k searches that early-terminate on the index sort return wrong top hits, searchAfter pages are wrong (bounded-exhaustive harness), and merges and later addIndexes write the disorder into new segments. CheckIndex at level 2 or higher reports the segment as broken; the default level 1 does not run the sort check. Documents are not lost, and filtered searches return only matching documents. A code fix does not repair segments that were already written, and a merge does not either (it carries the disorder forward); they have to be rewritten, for example by reindexing.
• How rare. Registry likelihood "rare" (a judgement): an experimental extension point plus in-place updates of the field it reads; one public downstream subclass reads a companion field but shows no in-place updates of it, and nobody has reported this since 8.6.0.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected null, but was:<org.apache.lucene.index.CheckIndex$CheckIndexException: segment has indexSort=<binary: "sku"> but docID=0 sorts after docID=1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotNull(Assert.java:756)
       org.junit.Assert.assertNull(Assert.java:738)
       org.junit.Assert.assertNull(Assert.java:748)
       org.apache.lucene.index.TestDerivedSortKeyDocValuesUpdate.testUpdateOfFieldReadBySortKey(TestDerivedSortKeyDocValuesUpdate.java:116)

• Contracts. setIndexSort: "Set the Sort order to use for all (flushed and merged) segments" (IndexWriterConfig.java:504). getIndexSortFields: "Returns the field names involved in the index sort" (LiveIndexWriterConfig.java:417); a field the key is computed from is involved. The guard's own message says "involved in the index sort".
• The guard's intent. LUCENE-7581 (#8632, 6.4.0) added it; its CHANGES entry reads "Lucene now prevents updating a doc values field that is used in the index sort, since this would lead to corruption" (lucene/CHANGES.txt:8683-8685), and in that issue a committer asked to "prevent changing any field involved in the index sort". In 2016 a sort could read only its own field, so a name check was complete. #16565 (10.6.0) extended it to updateBinaryDocValue: "Updating a sort field's doc values would make the persisted sort order inconsistent with the values, so it should be rejected up front".
• The documented pattern. BinarySortField's class javadoc (BinarySortField.java:41-45) lets a subclass derive the key from "other doc-values fields of the segment"; getSortKeyDocValues (:149-158) restricts only the read pattern and the value lifetime. Neither BinarySortField, SortField#getIndexSorter, IndexSorter, SortFieldProvider, setIndexSort nor the update methods say that such fields must not be updated.
• The inconsistency. IndexingChain#validateIndexSortDVType (IndexingChain.java:1635-1727) treats a field the sorter reads as part of the index sort: a devil's-advocate probe saw a BINARY price rejected under a sort that reads NUMERIC price ("expected field [price] to be NUMERIC"). The update guard does not. CheckIndex, Lucene's own checker, calls the result an error.
• Tests. TestIndexSorting#testBadDVUpdate asserts the rejection for the declared sort field; no upstream test updates a field that a sorter reads, and none asserts that such an update is allowed.

The devil's-advocate verifier and the cross-model reviews built the strongest case for "intended": BinarySortField and SortField#getIndexSorter are expert @lucene.experimental hooks (the IndexSorter interface and SortFieldProvider carry no such tag), a SortField exposes one field name, and the writer cannot know what an opaque sorter reads, so keeping companion fields stable is the extension author's job.

Why it fails: no javadoc states that rule; the writer can and does discover the fields a sorter reads (validateIndexSortDVType); the guard's stated intent covers "any field involved in the index sort"; and the outcome is a segment that Lucene's own CheckIndex rejects and that merges propagate, not a documented limitation. Every verifier rated the case at most moderate. Verdict: a defect. What remains is a point about the fix: upstream may prefer a documented rule, or an API for a sort field to declare the fields it reads, over probing; so state the expectation as "reject the update, or document that such fields must not be updated".

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF). The other options were not run. Which one, or another design, is the maintainers' call; option D could cover the sibling gaps in one change.

setIndexSort builds indexSortFields from each sort field's name plus every doc-values field its IndexSorter requests from an empty recording DocValuesLeafReader, a probe modelled on the one in validateIndexSortDVType: both call getDocComparator(reader, 0) on an empty doc-values reader, but that one type-checks a single field and throws from getFieldInfos(), while this one records every name requested and returns FieldInfos.EMPTY. Patch:, +61/-4 in IndexWriterConfig.java; updateNumericDocValue and updateDocValues of price are rejected, and the test checks the guard's message for updateNumericDocValue; an unrelated field stays updatable). The finder's earlier sketch (in the hunt folder) is the same logic.

for (SortField sortField : sort.getSort()) {
  fields.add(sortField.getField());
  try {
    sortField.getIndexSorter().getDocComparator(probe, 0);
  } catch (IOException | RuntimeException _) {
    // best effort: if a sorter fails here, fields it would read later are not recorded
  }
}
return Collections.unmodifiableSet(fields);

Trade-offs

• Best effort: a sorter that chooses its fields from segment state (FieldInfos, maxDoc, values) escapes the probe, and so does any field a sorter would request after it throws on the empty reader (fields requested before the throw stay recorded); a point the cross-model reviewer raised.
• Cost: each sorter runs once per setIndexSort, with maxDoc 0.
• The probe calls getIndexSorter() a second time and swallows any RuntimeException it throws, which matters only for a stateful or side-effecting sorter.
• getIndexSortFields() now returns an unmodifiable set instead of the Collectors.toSet() result (its mutability was never documented).
• getIndexSortFields() returns more names, which matches its javadoc; a downstream caller of it (one exists in Solr's atomic-update code) would then protect those fields too.
• The probe could be shared with validateIndexSortDVType instead of duplicated.

What the checks validated

• Stock: the repro class fails 3 of 4 with assertions on and off; the control passes. The fifth (rejection) test fails on stock too: expected:<[price, sku]> but was:<[sku]>.
• With the fix: the 5 tests pass with assertions on and with assertions off.
• With the fix: TestIndexSorting, TestBinarySortField, TestIndexWriterConfig, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestMixedDocValuesUpdates and TestSortedNumericDocValuesUpdates pass (188 tests, 2 skipped, 0 failures).
• With the fix: the full lucene/core test suite passes with assertions on (8,894 tests, 322 skipped, BUILD SUCCESSFUL; the JUnit XML of the fact check's re-run counts 749 classes, 0 failures and 0 errors, including the 5-test class); :lucene:core:ecjLint and :lucene:core:checkGoogleJavaFormat pass.
• Earlier, with the finder's sketch shadowed: the finder's, the independent reproducer's and the rootcause step's tests and the generic probe pass; TestIndexSorting and TestBinarySortField pass (87 tests).
• After the fact checks the catch block's comment was reworded (no code change): on a fresh scratch worktree both diffs apply, the 5 tests pass with assertions on and off, and ECJ lint and the format check pass. The full suite was not re-run for that change.

For example a SortField method that returns Set.of(getField()) by default and that BinarySortField subclasses override; setIndexSort takes the union.

Trade-offs

Exact and cheap, with no probing. But a subclass that forgets to declare its companion field silently keeps today's behaviour, and it adds API to experimental classes.

State in BinarySortField, getSortKeyDocValues and SortField#getIndexSorter that fields a custom sort key reads must not be updated in place.

Trade-offs

No code change, but the silent misordering stays for anyone who misses the sentence, and CheckIndex at the default level does not catch it.

Combine the first or second option with open PR #16810's ensureNotIndexSortField (the sort taken from existing segments when the writer config has none) and with the reserved parent field (hunt row LFV-095), so that one check covers all three.

Trade-offs

Larger change across IndexWriter and IndexWriterConfig; #16810 is open and stale.

Method: bounded-exhaustive model-based testing on the real jar (semi-formal tier).

• Operations: add, delete first or last, update of the sort field, update of the field the derived key reads, flush, forceMerge(1), addIndexes of a sorted reader, addIndexes(SortingCodecReader.wrap(unsorted)), and commit plus reopen. Every script of length 4 or less, in 26 configurations (default and derived keys, missing first or last, reverse, two multi-field sorts, search only).
• Oracle after every step: an independent reference order, checked against each segment's physical order, CheckIndex's sort check, top-k with and without the index-sort shortcut, searchAfter pages and the comparator axioms.
• Result: 1,975,300 scripts and 7,769,228 steps with assertions off; every failure, and only failures, in scripts that update the companion field. The shortest counterexample was minimised into a JUnit.
• The hypothesis was a planning lead (an unexecuted lead from an earlier unit), so the harness confirmed and pinned down a suspected gap; it was not a blind find. Found 2026-10-27 on main a9779ef3c193; then checked by an independent reproducer, a devil's-advocate verifier, reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a follow-up (history, prior reports, security), the registration check and an audit.