CheckIndex -exorcise with more than one thread writes the surviving segments in size order, not commit order

CheckIndex -exorcise repairs a damaged index by writing a new commit that keeps the good segments and drops the broken ones. Since 8.11.0 and 9.0.0 the check runs on several threads by default (threadCount = the number of cores). In that mode it sorts the segments by file size so that small ones print first, collects the results in that sorted order, and exorciseIndex commits that list as it is. A one-thread repair keeps the commit order. The repaired index has the right documents, deletes and values, but the order of whole segments, and so of all documents, depends on the thread count: commits are usually largest (oldest) first, so an insertion-ordered index typically comes back reversed and Sort.INDEXORDER returns the newest documents first.

1. Default is concurrent. threadCount = Runtime.getRuntime().availableProcessors() (:541); checkIndex(List) creates a fixed pool whenever threadCount > 1 and calls the executor overload (:588-600).
2. The sequential branch keeps commit order. With no executor it walks lastCommit.info(i) for i = 0..n-1 and calls processSegmentInfoStatusResult (:823-845).
3. The concurrent branch sorts by size. It copies the commit into segmentCommitInfos (:850-854), sorts the copy by sizeInBytes() ascending (:856-869, commented "smaller segment tends to finish faster, and hence its output can be printed out faster"; a size that throws IOException compares as equal) and submits the largest first (:871-896).
4. Results are collected in the sorted order. The result loop walks segmentCommitInfos.get(i) (:898-931) and calls processSegmentInfoStatusResult (:929), which appends to Status.segmentInfos and, for a good segment, result.newSegments.add(info.clone()) (:975-985).
5. The repair writes that list. exorciseIndex calls newSegments.changed() and newSegments.commit(dir) (:4606-4614), with no reordering.

Only the position of whole segments in segments_N changes, and with it the starting doc id of every segment that moves. Once this branch runs the order does not depend on timing: even a single-thread executor reorders. Each surviving segment keeps its deletes, soft deletes and doc-values updates.

In the repro's index the three commits of 10, 1 and 1 documents give _0 (787 bytes), _1 and _2 (751 bytes each). The size sort gives [_1, _2, _0] (a stable sort keeps _1 before _2). With _1 damaged (the Sort.INDEXORDER column numbers the documents 0 to 11 in insertion order, as the fact probe does):

The concurrent check also prints 1 of 3: name=_1, 2 of 3: name=_2, 3 of 3: name=_0: its progress labels use the sorted position (cosmetic).

• Who. The CheckIndex command line with -exorcise ("When not specified, this will default to the number of CPU cores"); Luke's check and "Try to Repair" (a default CheckIndex, then exorciseIndex on the status it returned); any caller of checkIndex() plus exorciseIndex() with the default thread count or setThreadCount(n > 1), or of checkIndex(null, executor) with any executor (a repair needs a full check: exorciseIndex rejects a partial status); and downstream shard-repair commands that call checkIndex(null) without setThreadCount (read in their source, not run).
• Default vs opt-in. Default whenever Runtime.getRuntime().availableProcessors() is more than 1, so on any multi-core host. -threadCount 1, or a JVM that reports one processor, is sequential and keeps the order.
• Scale. Independent of index size. Affected: indexes with more than one surviving segment whose commit order differs from ascending sizeInBytes() order. That is the usual case, because commits are usually largest (oldest) first, so the ascending sort typically reverses them. Observed in the intended-behaviour check: an insertion-ordered LogDocMergePolicy index _13(108) _17(12) _1b(12) _1c(4) _1d(1) with _17 broken comes back [_1d, _1c, _1b, _13] instead of [_13, _1b, _1c, _1d] (that run did not log byte sizes).
• What changes. Sort.INDEXORDER results, the order of equal-score hits (ties go by doc id), and which segments LogMergePolicy later treats as adjacent. Concurrent and sequential repairs of the same index write different commits, and the result depends on the machine's core count.
• What does not. No document is lost beyond the broken segments, none is added or altered, deletes, soft deletes and doc-values updates are kept, and the repaired index checks clean.
• How likely. Rare: -exorcise is an emergency action on an index that already has a broken segment, and only applications that rely on relative document order notice. The recorded searches found no report since 8.11.0 (2021).

Deterministic, single test thread; setThreadCount(2) makes the result independent of the machine (the default, the number of cores, behaves the same on any multi-core host). The default codec is set because LuceneTestCase randomizes it and the test names _1.fdt; no compound files so _1.fdt is a real file; NoMergePolicy keeps three segments. The single assertion also shows that the damage worked, because _1 is absent.

It asserts only the segment order, not doc ids. The same test is the suggested regression test: it passes with the recommended fix.

cp $REPRO/TestCheckIndexExorciseSegmentOrder.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestCheckIndexExorciseSegmentOrder -Ptests.seed=DEADBEEF
# assertions off fails the same way: add -Ptests.asserts=false

# gradle, stock main 645d4fe14a20, seed DEADBEEF, assertions on (trimmed)
TestCheckIndexExorciseSegmentOrder > testConcurrentExorciseKeepsCommitOrder FAILED
    java.lang.AssertionError: expected:<[_0, _2]> but was:<[_2, _0]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:6E1F4422DE9E97A8]:0)
        ...
        at org.apache.lucene.index.TestCheckIndexExorciseSegmentOrder.testConcurrentExorciseKeepsCommitOrder(TestCheckIndexExorciseSegmentOrder.java:62)
:lucene:core:test (FAILURE): 1 test, 1  failure

# fact probe on an index of the same shape whose documents store their insertion number as `id` (prints only, not the repro; abridged)
threadCount=1  status=[_0, _1 (broken), _2]  repaired=[_0(10), _2(1)]  INDEXORDER ids=[0..9, 11]
threadCount=2  status=[_1 (broken), _2, _0]  repaired=[_2(1), _0(10)]  INDEXORDER ids=[11, 0..9]

# command-line test (4 commits of 40/10/25/5 docs with a 100-character stored body each, the 10-doc segment broken), assertions off
-threadCount 1          after=[_0(40), _2(25), _3(5)]  Sort.INDEXORDER top5=[0, 1, 2, 3, 4]
-threadCount 2          after=[_3(5), _2(25), _0(40)]  Sort.INDEXORDER top5=[75, 76, 77, 78, 79]
default (32 cores)      after=[_3(5), _2(25), _0(40)]  Sort.INDEXORDER top5=[75, 76, 77, 78, 79]

• The thread count is a speed knob. CheckIndex.java:532 "Set threadCount used for parallelizing index integrity checking."; :621 "This method allows caller to pass in customized ExecutorService to speed up the check."; usage :4845-4847 "When '-threadCount 1' is used, index checking will be performed sequentially." The originating issue LUCENE-9662 (#10701) asked for concurrency so the check finishes faster.
• Exorcise only removes. Class javadoc :104-105 "write a new segments file that removes reference to problematic segments"; exorciseIndex :4603-4604 "effectively removing all documents in broken segments from the index"; :4657-4659 "With -exorcise, this tool will remove any segments that have issues and write a new segments_N file."
• The sort is for printing and scheduling. The code comment (:856-857) and the #128 review, where the author wrote that following the sequential order "is not really needed, especially under concurrent setting", about the printed output. No discussion found in #128, #10701, the 8.x backport or #281 identifies the sort's effect on newSegments or on the order of the repaired commit (exorcise appears there only in pasted tool output and in #281's copy of the command-line usage).
• Reordering tools warn; CheckIndex does not. IndexUpgrader.java:48-52: "If your application relies on \"monotonicity\" of doc IDs (which means that the order in which the documents were added to the index is preserved), do a full forceMerge instead", and :77 "WARNING: This tool may reorder document IDs!"; UpgradeIndexMergePolicy.java:44-48; TieredMergePolicy.java:46-47 "this policy freely merges non-adjacent segments; if this is a problem, use LogMergePolicy"; ParallelCompositeReader.java:36-44 recommends LogDocMergePolicy "as this one does not reorder documents during merging (like TieredMergePolicy)".
• History. Before 424192e1704 one loop walked the commit in order and filled newSegments; splitting it into two branches changed the concurrent result order as a side effect.
• Tests. No Lucene test calls exorciseIndex. TestUtil.checkIndex by default runs the concurrent check with 2-5 threads but never exorcises, so nothing would have caught it.

The lenses of an independent reviewer from another model family (OpenAI GPT-5.6) returned "not real", high confidence, after reproducing the reorder and agreeing with the root cause and the fix: no javadoc promises any order for Status.segmentInfos (public), Status.newSegments (package-private) or the repaired commit; CheckIndex is @lucene.experimental; doc numbers are "ephemeral" (IndexReader.java:51-54) and docids "internal implementation" (index/package-info.java:101-105); the default TieredMergePolicy already merges non-adjacent segments; and exorcise renumbers every document after a dropped segment anyway.

Why it fails: the ephemeral-docid text is about absolute numbers changing as documents are added, deleted and merged. Relative insertion order is something Lucene treats as relied upon: it names it ("monotonicity"), warns in the tools that may break it (IndexUpgrader, UpgradeIndexMergePolicy, TieredMergePolicy), and sends users who need it to LogMergePolicy, whose users then get a reversed index from exorcise with no warning. And two execution modes, documented only as a faster and a sequential way to run the same check, write different commits from the same input; that equivalence is an implied contract, not a written one. Concession: maintainers may land a fix as a consistency improvement rather than a bug fix (the analogous IndexSearcher parity change GITHUB#12516 was filed under "Changes in runtime behavior"). The hunt's verdict was "confirmed", medium confidence, after a "disputed" judge.

The core of the case is that the repaired index depends on -threadCount, with Sort.INDEXORDER reversed, not that doc ids change; no javadoc promises the order, and the Status.segmentInfos order alone is not a bug. Likely objections are the ephemeral-docid javadoc and #7405 ("Remove checkindex -exorcise"), and (the fact check adds) that readers can already order leaves differently (IndexWriterConfig#setLeafSorter, DirectoryReader.open(dir, leafSorter)): that is opt-in and explicit, while the reorder here is a side effect of the thread count.

Option A was built and checked with gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, 2026-10-31); B was checked by the tie-breaker on a9779ef3c193, whose CheckIndex.java is identical. C and D were not run.

In the concurrent branch, keep each status in an identity map instead of recording it at once, then walk the commit and record the results in its order. +11/-1 lines (one import, the map, the final loop).

      Map<SegmentCommitInfo, Status.SegmentInfoStatus> statuses = new IdentityHashMap<>();
      for (int i = 0; i < numSegments; i++) {
        // ... unchanged: wait for futures[i], print its output in size order ...
        statuses.put(info, segmentInfoStatus); // was: processSegmentInfoStatusResult(...)
      }

      // record the results in commit order, as the sequential branch does
      for (SegmentCommitInfo info : lastCommit) {
        if (statuses.containsKey(info)) {
          processSegmentInfoStatusResult(result, info, statuses.get(info));
        }
      }

Trade-offs

• newSegments and Status.segmentInfos match the sequential branch; #128's deliberate print and scheduling order stay.
• One map entry per segment. SegmentCommitInfo does not override equals, so an identity map is exact.
• Segments skipped by onlySegments are never put, so they stay skipped; that only affects Status.segmentInfos, since exorciseIndex rejects a partial status. The interrupted-caller path is unchanged (a null status is still passed on; that is a separate bug in the same loop).

What the checks validated

• Stock: the repro fails (gradle, assertions on). With A it passes with assertions on and off.
• TestCheckIndex, TestAllFiles*, TestSwappedIndexFiles, TestSegmentInfos, TestCodecHoldsOpenFiles and the repro: 26 tests, 0 failures, assertions on and off.
• org.apache.lucene.index.*: 1,396 tests, 56 skipped, 0 failures, assertions on and off (seed DEADBEEF); 22 of these test classes call TestUtil.checkIndex, which by default runs the concurrent check with 2-5 threads.
• checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass for lucene:core.
• The tie-breaker's command-line test (-threadCount 2 and the default) passes 3/3 with A and fails 2/3 without, assertions off. An equivalent variant (an array plus a map of sorted positions) passed every test on a9779ef3c193.

Walk the commit and look up each segment's future (the finder's fix).

Trade-offs

Fixes the order, but moves the per-segment log output back to commit order, which #128 changed on purpose ("so that the smaller ones finishing faster should be printed out earlier"); reviewers may object.

What the checks validated

By the tie-breaker on a9779ef3c193: the finder's, the independent reproduction's, the intended-behaviour check's and the command-line tests pass, and TestCheckIndex 9/9.

Rebuild newSegments in the commit's order before committing.

Trade-offs

Fixes the written index only; Status.segmentInfos stays size-ordered, and exorciseIndex would have to re-read the commit (Status keeps only segmentsFileName) or newSegments would have to carry each segment's position.

Say that with threadCount > 1, -exorcise writes the segments in ascending size order.

Trade-offs

Keeps two execution modes that write different indexes from the same input.

Method: non-formal (a differential sweep over a hand-picked set of executor behaviours is testing, not bounded-exhaustive checking; the finder tagged it semi-formal).

• A hunt unit (executor pluggable-behaviour sweep) ran a harness over 13 API surfaces and 9 executor behaviours on the stock jars, comparing each call with the executor-less call (same result fingerprint expected).
• The CheckIndex.checkIndex rows differed in every non-rejecting cell (caller-runs, single thread, LIFO pool, interrupted pool thread): on a clean 5-segment index Status.segmentInfos came back _1,_3,_2,_0,_4 instead of _0 to _4.
• The finder then read exorciseIndex, saw that it commits the same list, and wrote a public-API JUnit with one corrupted segment, which fails with assertions on and off while the sequential control passes.