ParallelLeafReader claims an index sort that its served values break when an unsorted sidecar listed first holds the sort field: wrong sorted hits, and addIndexes writes a segment that violates its declared sort

A ParallelLeafReader takes its index sort from the first sub-reader that declares one (ParallelLeafReader.java:126-142 on main 645d4fe14a20) but takes each field from the first sub-reader that has it (:156-159), and publishes the sort (:193) without asking which reader serves the sort fields. When an unsorted sidecar listed first re-indexes a field of the sort, the reader claims an order that its served values break. In the repro (10 documents, a main index sorted on s = 0-9, a sidecar that gives docs 0-3 the values 100-103) it serves 100, 101, 102, 103, 4, ..., 9 under <long: "s">: a top-3 search sorted by s returns docs 0, 1, 2 instead of 4, 5, 6, IndexSortSortedNumericDocValuesRangeQuery s:[0 TO 9] counts 10 instead of 6, a skip-indexed slow range query 0 instead of 6, and ParallelCompositeReader behaves the same. IndexWriter.addIndexes(SlowCodecReaderWrapper.wrap(reader)) into an index sorted on s checks only the claimed sort and writes a segment whose documents break it; CheckIndex level 2 rejects it, the default level does not look, and merges keep it. The code is unchanged since 6.2.0 (LUCENE-6766), and every release run from 6.2.0 on is affected. Keeping the sort only when a sub-reader that declares it serves every sort field, and advertising null otherwise, fixes it; addIndexes then refuses the reader with its existing error.

1. The sort comes from the first sub-reader that has one. ParallelLeafReader.java:126-142: if (indexSort == null) { indexSort = leafIndexSort; }; only a second, different non-null sort throws "cannot combine LeafReaders that have different index sorts". The sidecar's sort is null and main's is <long: "s">, so the reader keeps <long: "s">.
2. Each field comes from the first sub-reader that lists it. :154-159, // NOTE: first reader having a given field "wins": (fieldToReader.put(fieldInfo.name, reader)), and getNumericDocValues reads from that reader (:396-399). With the sidecar first, s comes from the sidecar.
3. The sort is published without asking who serves its fields. :193: new LeafMetaData(createdVersionMajor, minVersion, indexSort, hasBlocks). ParallelCompositeReader builds one ParallelLeafReader per leaf (ParallelCompositeReader.java:132-136), so it does the same.
4. Consumers trust the claim. TopFieldCollector decides per segment whether the search sort is a prefix of the index sort (TopFieldCollector.java:58-59) and then stops comparing once its queue is full. Sort.getPrimarySortField (Sort.java:145-146) feeds IndexSortSortedNumericDocValuesRangeQuery and the skip-index shortcut of the doc-values range queries. IndexWriter.validateMergeReader only checks that the claimed sort matches the writer's (IndexWriter.java:3245-3251), so addIndexes copies the documents in their served order under the declared sort.
5. CheckIndex looks only from level 2. The index-sort test runs for level >= MIN_LEVEL_FOR_INTEGRITY_CHECKS (2) (CheckIndex.java:1114, :1146-1149); the default level is 1 (:4748).

Controls that stay right: the sorted reader listed first; a sidecar without the sort field (the usual sidecar); a search sort that is not a prefix of the index sort.

The finder's bounded enumeration of 25,230 compositions of 2 and 3 sub-readers, with Z3 deciding per structure whether legal values can break the claimed sort, found 3,008 violable structures; all 3,008 witnesses replay on the stock jar. Two shapes beyond the sidecar: a two-field sort (s, t) declared by both readers, where the first serves s and the second t (served (0, 0), (1, 1), (1, 0)); and a points-only field with the sort field's name in the sidecar, which hides the sorted reader's doc values (then addIndexes fails loudly with IllegalStateException).

• Who is hit. An index sort (opt-in) plus a parallel reader (an expert class) whose sidecar, listed first, re-indexes a sort field with values in another order. The javadoc's use case is a sidecar for frequently changing fields (ParallelLeafReader.java:42-44), and a frequently changing sort key fits it.
• Read side. Silent wrong top hits for sorted searches; wrong hit sets from the index-sort range shortcuts, too many (10 instead of 6) or too few (0 instead of 6).
• Write side. addIndexes persists a segment whose documents break its declared sort. CheckIndex at its default level calls it clean; merges keep it; a sorted search on the merged index is still wrong.
• Scale and settings. 10 documents, the default codec and a plain IndexSearcher (in-memory directories in the repro; the logic does not depend on the directory); the same with assertions on or off.
• How likely: rare. The javadoc expects the indexes to "typically" contain different fields (ParallelLeafReader.java:36-37), the sidecar must hold a field of the sort with values in another order, and no report exists since 6.2.0 (2016).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected null, but was:<org.apache.lucene.index.CheckIndex$CheckIndexException: segment has indexSort=<long: "s"> but docID=0 sorts after docID=1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotNull(Assert.java:756)
       org.junit.Assert.assertNull(Assert.java:738)
       org.junit.Assert.assertNull(Assert.java:748)
       org.apache.lucene.index.TestParallelLeafReaderShadowedIndexSort.testAddIndexesOfSidecarFirstReaderKeepsIndexSort(TestParallelLeafReaderShadowedIndexSort.java:61)

• Contract. LeafMetaData.java:32-33: the sort is "the order in which documents from this index are sorted, or null if documents are in no particular order". ParallelLeafReader builds that metadata and already validates sorts (it throws for two different ones, :136-142, testWithIndexSort1).
• Overlap is endorsed. The class javadoc: matches come "from the first index added that has the field" (:38-40). TestDemoParallelLeafReader.java:178-182: "NOTE: important that parallelReader is first, so if there are field name overlaps, because changes to the schema overwrote existing field names, it wins".
• Sorted plus unsorted is a tested combination. testWithIndexSort2 ("ok to have one leaf w/ index sort and the other with no sort", TestParallelLeafReader.java:375-398), but with field-less documents, so the unsorted reader never held the sort field.
• Precedent. LUCENE-7581: "Lucene now prevents updating a doc values field that is used in the index sort, since this would lead to corruption" (CHANGES.txt:8700-8702); #16565 extended it to binary doc values (:729-730); updateNumericDocValue throws (IndexWriter.java:1965-1970). The sibling SlowCompositeCodecReaderWrapper publishes a null sort (:94).
• History. LUCENE-6766 (2016) set the rule; its review fixed only the order dependence of null sorts, and nobody discussed which reader serves the sort field.

The intended-behaviour checker and the adjudication built the strongest case: the class javadoc warns "It is up to you to make sure all indexes are created and modified the same way ... Failure to do so will result in undefined behavior" (ParallelLeafReader.java:46-49), and a sorted main index with an unsorted sidecar that changes the sort key was arguably not "created the same way"; Lucene treats changing sort keys after indexing as unsupported (LUCENE-7581, #16565); delegating wrappers pass metadata through, and addIndexes trusts reader metadata by design; LeafMetaData is @lucene.experimental; the trigger is rare.

Why it fails: the warning's one concrete example is "the same documents in the same order", which the repro meets; the javadoc does not say whether a sidecar may re-index a sort field with other values, but sorted plus unsorted sub-readers are tested as accepted, so "created the same way" cannot require the same index sort; the LUCENE-7581 precedent made Lucene refuse the same outcome, and CHANGES calls it corruption; unlike a delegating wrapper, ParallelLeafReader changes which values are served, by design, and derives the sort itself; @lucene.experimental is about API stability, not a segment that CheckIndex rejects; rarity lowers priority, not validity. Verdict: not intended. A maintainer may still answer "undefined behaviour per the javadoc", or prefer an exception or a javadoc note.

One fix was built and validated; three more options were not built.

A change in the ParallelLeafReader constructor (1 file, +28; the finder's sketch, which passes checkGoogleJavaFormat): after the field map is built, keep the sort only if some sub-reader declares exactly this sort and serves every field of it (or no sub-reader has the field); otherwise publish null. In the repro the sidecar owns s and declares no sort, so the reader reports null: the top 3 become docs 4, 5, 6, the range query counts 6, and addIndexes throws "cannot change index sort from null to <long: "s">" (a caller who wants the documents can wrap the reader in SortingCodecReader). The usual sidecar and the sorted reader first keep their sort.

// ParallelLeafReader constructor, after fieldInfos = builder.finish():
if (indexSort != null && isServedByDeclaringReader(indexSort) == false) {
  indexSort = null;
}
// isServedByDeclaringReader(sort) is true iff some sub-reader whose getMetaData().sort()
// equals sort is, for every SortField, the owner of that field in fieldToReader (or no
// sub-reader has the field); a SortField without a field name makes it false
this.metaData = new LeafMetaData(createdVersionMajor, minVersion, indexSort, hasBlocks);

Trade-offs

• Conservative. It also drops a sort that happens to hold: a multi-field sort split across sub-readers that all declare it, a same-named points-only field that shadows the sort field, or a SortField without a field name. In the finder's bounded model (2 or 3 sub-readers, fields s and t, 3 documents, values 0-4) it drops 2,388 of 5,684 always-true structures, never the single-declaring-owner case that covers the javadoc's sidecar use; the patched class matches the intended predicate on 8,692 of 8,692 sort-advertising structures and, within that model, never claims a violable sort. The model does not cover custom readers, other SortField types or other codecs.
• Test framework. wrapReader can split one reader into two FieldFilterLeafReader halves under a ParallelLeafReader (LuceneTestCaseParent.java:2402-2404); a multi-field index sort split across the halves would lose its claimed sort there (shortcuts off, results unchanged).
• Behaviour change. addIndexes of such a reader into a sorted index now throws instead of silently writing a broken segment; sorted searches on it lose the early-termination shortcut.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, OpenJDK 25: the repro test fails on the stock tree and passes with the fix, with assertions on and off.
• With the fix: TestParallel* (5 suites incl. the new test), TestIndexSorting, TestDemoParallelLeafReader, TestAddIndexes, TestSortingCodecReader, TestTopFieldCollectorEarlyTermination and TestIndexSortSortedNumericDocValuesRangeQuery: 11 suites, 168 tests, 0 failures; the whole org.apache.lucene.index package: 215 suites, 1,396 tests, 56 skipped, 0 failures (with the test's earlier revision, which differs only in the name of the unused catch parameter). The logs keep the test counts; the suite counts come from the JUnit XML of the removed worktree, and an independent fact check re-ran both selections with the final test revision and got the same counts.
• :lucene:core:check -x test passes: checkGoogleJavaFormat, ecjLintTest, forbiddenApisTest and the license and regeneration checks executed, ecjLintMain and forbiddenApisMain were up to date on the same inputs; errorProne was skipped (off by default outside CI) and eclint not applied (no tool location set).
• Earlier, by the hunt: the finder's 9-test JUnit and the split-sort test pass; 86 upstream parallel-reader and index-sorting tests and 352 early-termination and range-query tests over 8 seeds pass stock and fixed; the independent reproducer's JUnit passes 8 of 8.
• Not run: nightly tests, a benchmark, the merge-time wrapForMerge route.

Reject the combination, as the constructor already does for two different sorts. Not built.

Trade-offs

Loud and simple, but it rejects every composition in which an unsorted earlier reader holds a sort field, also for users who never sort or call addIndexes; the test framework's split case would need the same ownership rule to avoid throwing on multi-field sorts.

A javadoc sentence on both parallel readers: the claimed index sort is valid only if the sort fields come from a reader sorted by it. Not built.

Trade-offs

Cheap, but alone it leaves the silent corruption for anyone who misses the note.

Have addIndexes(CodecReader...) check the incoming documents against the declared sort, as CheckIndex.testSort does. Not built.

Trade-offs

Also catches other readers that publish a wrong sort, but costs a pass over the sort fields of every added reader, fixes only the write side and leaves the wrong search results.

Method: non-formal first signal, widened by a formal leg (the hunt's hunt unit for parallel-reader index-sort metadata, 2026-10-28, main a9779ef3c193).

• Lead. Item 7 of an outside reviewer's idea pack (OpenAI GPT-5.6, code reading): "parallel-reader metadata must not advertise an unsafe index sort after a mixed sort declaration". A planning probe on the stock jars showed the sidecar-first composite advertising <long: "s"> with a wrong top 3 and wrong range counts; a second probe extended it to addIndexes, CheckIndex levels and forceMerge.
• Formal leg. A bounded-exhaustive enumeration on the stock jar: 29 legal sub-reader types (combinations of an index sort none, (s), (t), (s, t) or (t, s) with s and t each absent, doc values or points-only) in ordered lists of 2 and 3, 25,230 compositions through the real ParallelLeafReader and ParallelCompositeReader. For each of the 8,692 structures that advertise a sort, Z3 decided whether legal values (every sorted sub-reader in its own declared order) can make the served values break it: 3,008 can. All 3,008 witnesses were rebuilt as real indexes and violate the claimed sort on the stock jar. A vacuity mutant without the sorted-sub-reader constraint flips the single-owner structures to violable, so the UNSAT answers depend on that constraint.
• JUnit. Reduced to a public-API JUnit (9 tests, 2 controls; 7 fail with assertions on and off) and a second test for the split two-field sort; this doc's repro is a one-method version of the addIndexes test.