Reordering merges break document blocks written before Lucene 9.9, because IndexWriter reads "not recorded" as "no blocks"

Since 9.11.0, IndexWriter lets a merge renumber documents (OneMerge#reorder) only when there are no document blocks or a parent field identifies them. It decides "no blocks" from the per-segment flag LeafMetaData#hasBlocks, which Lucene first recorded in 9.9.0 and which is documented as false for every segment written earlier. So on an index whose blocks were written by Lucene 9.0-9.8, which cannot have a parent field, every reordering merge of only such segments is let through, and BPReorderingMergePolicy interleaves the blocks: 139 of 257 children separated from their parent in the finder's repro, 199 of 200 wrong ToParentBlockJoinQuery parents in an independent one. There is no exception, CheckIndex passes, the merged segment keeps hasBlocks=false, and from 9.11.0 on the same documents written by 9.9.2 are left alone (9.9.0-9.10.0 had no guard at all). It affects every release from 9.9.0 to 10.6.0 (10.6.0 by reading) and main; the fix treats segments whose documents predate 9.9 as possibly holding blocks.

• Document block: documents added together with IndexWriter#addDocuments, children first and the parent last. Lucene keeps them adjacent, and block joins (ToParentBlockJoinQuery, ToChildBlockJoinQuery) find a child's parent as the next parent document (ToParentBlockJoinQuery.java:261, parentBits.nextSetBit(childDoc + 1)).
• Parent field: IndexWriterConfig#setParentField (since 9.10) marks every parent so that a reorderer can move whole blocks. An existing index that already has fields but no parent field cannot get one: "can't add a parent field to an already existing index without a parent field" (IndexWriter.java:1125-1131; only an index with no fields yet can, CHANGES GITHUB#13498/#13340), so an index holding pre-9.9 blocks never can.
• Reordering merge: a merge policy whose OneMerge#reorder returns a doc map renumbers the merged segment. BPReorderingMergePolicy (lucene/misc) does this to cluster similar documents and shrink the index; it asks to reorder every forced merge ("No minimum size for forced merges", BPReorderingMergePolicy.java:103-105), treats addIndexes as forced (:199-203) and asks to reorder natural merges from 1 document up by default (:46). "Reorder" here means the policy returns a doc map; whether documents actually move depends on BP's size thresholds (BPIndexReorderer.java:243-245), and its bp.reordered=true diagnostic does not prove that any did.

1. LeafMetaData#hasBlocks: "Note: This property was not recorded before LUCENE_9_9_0 this method will return false for all leaves written before LUCENE_9_9_0" (LeafMetaData.java:34-38). The 9.0-9.8 segment-info format reads it as false (Lucene90SegmentInfoFormat.java:157-170).
2. Both reorder guards, for merges (IndexWriter.java:5544-5558) and for addIndexes(CodecReader...) (:3549-3562), compute hasBlocksButNoParentField = anyMatch(LeafMetaData::hasBlocks) && anyMatch(parentField == null) under the comment "Don't reorder if blocks can't be identified using the parent field.", and call merge.reorder(...) when it is false and no index sort is set.
3. For a merge whose inputs all predate 9.9 the first operand is false, so the reorder runs although the blocks have no parent field. BP places each document on its own, ignoring blocks, so children land before other blocks' parents.
4. The merged segment's hasBlocks is the OR of its inputs (IndexWriter.java:5313-5319; addIndexes :3500-3506), so it stays false and later merges of it stay unprotected unless another input reports blocks. Its minVersion stays the oldest input's (SegmentMerger.java:78-91), which a fix can key on.
5. anyMatch means one input that records blocks protects the merge: the 9.8.0 segments plus one main-written 2-document block are not reordered (0 broken), while plus one main-written single document they still are (124 broken). An index sort also disables reordering.
6. CheckIndex does not notice: its only block-aware check is part of the index-sort test (CheckIndex.java:1247-1256).

Small example (written by 9.8.0: 4 product blocks, 2 color variants each, 12 documents, 2 segments), forceMerge(1) through BPReorderingMergePolicy on main, with BP's thresholds lowered for this tiny example (minPartitionSize 2, minDocFreq 1; with the defaults the guard still lets the reorder through and writes bp.reordered=true, but no document moves and all 8 variants keep their parent): the guard computes anyMatch(hasBlocks)=false, the merge is reordered (bp.reordered=true, merged hasBlocks=false, minVersion=9.8.0), and the order becomes [1a, 1b, P1 shirt, 2a, P2 shoe, 4b, 2b, 3a, 3b, P3 hat, 4a, P4 sock]: ToParentBlockJoinQuery maps 2b (the shoe's black variant) and 4b (the sock's white variant) to the hat, "products with a white variant" returns shirt, shoe, hat instead of shirt, shoe, sock, 6 of 8 variants keep their parent, and CheckIndex is clean. Written by 9.9.2, the same blocks are not reordered and all 8 keep their parent; with the fix, the 9.8.0 copy is not reordered either.

Finder's fixture (120 blocks, 377 documents, 257 children, 3 segments, written by 9.8.0), forceMerge(1) through BPReorderingMergePolicy(LogDocMergePolicy, BPIndexReorderer): 139 of 257 children separated from their parent, merged hasBlocks=false; the same documents written by 9.9.2: 0, not reordered. Independent fixture (200 blocks, 584 documents, 384 children): 381 of 384 children outside their block and 199 of 200 wrong ToParentBlockJoinQuery parents, through forceMerge(1) and through addIndexes(CodecReader...); CheckIndex passes. This doc's test with a merge policy that reverses doc IDs: 257 of 257 children paired with the wrong parent, both paths. The BP repros lower minPartitionSize (default 32) and minDocFreq (default 4,096) only so that a few hundred documents (12 in the small example) actually move; the guard decision does not depend on them.

• Who is exposed. Applications that opt into a reordering merge policy, BPReorderingMergePolicy from lucene/misc or any MergePolicy whose OneMerge#reorder (@lucene.experimental) returns a doc map, with no index sort, on an index that still holds blocks first written by Lucene 9.0-9.8 and never reindexed. Blocks written by 8.x are hit too on 9.9.0-9.12.3 (executed on 9.10.0, 9.11.0 and 9.12.3; 9.9.x by reading, as it has no guard at all; 10.0+ cannot open 8.x indexes).
• Trigger. A forced merge, a natural merge, forceMergeDeletes or addIndexes(CodecReader...) whose inputs all predate 9.9 (or all report no blocks). A forced merge right after enabling the policy hits the whole index; BP's defaults then ask to reorder every natural merge too (documents move only where BP's size thresholds allow).
• Silent and permanent. No exception, CheckIndex clean, the merged segment keeps hasBlocks=false. The merged segment does not record which documents formed a block, so a code fix cannot repair it; affected indexes need a reindex from source (or a restore of a copy taken before the merge). Each document's own contents stay intact.
• What breaks for users. Block joins attach children to the wrong parents (199 of 200 blocks in one repro): nested search, child retrieval, block-join filters and deletes.
• Reach (read, not tested downstream). Apache Solr ships a BP merge-policy factory since Solr 9.9.0, bundled Lucene 9.0-9.8 in Solr 9.0-9.4, and sets a parent field only together with an index sort; so Solr collections with nested documents first written by Solr 9.4 or earlier, never reindexed, on which an operator enables the BP factory, fit the pattern. No use of BPReorderingMergePolicy was found in Elasticsearch or OpenSearch; a search there for custom OneMerge#reorder overrides hit an API rate limit and was not completed. The open issue #12665 discusses enabling recursive graph bisection out of the box.
• How likely. Rare: it needs the opt-in policy plus never-reindexed pre-9.9 blocks plus no index sort. For an index that has all three it happens on every such merge.
• Spotting damaged segments. BP writes the diagnostic bp.reordered=true; a segment with that diagnostic and a minVersion before 9.9.0 went through a reorder without block protection (the marker says BP returned a doc map, not that documents moved; a reading of the values the runs print, not a tested tool). Other reordering policies write no such marker.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[c0p0, c1p1, c2p2]> but was:<[c0c1, c2p0, p1p2]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.misc.index.TestReorderPre99Blocks.testReorderingMergeKeepsPre99Blocks(TestReorderPre99Blocks.java:59)

• The guard's stated intent. #13128's commit message (4a3f4cfe9745): "This updates IndexWriter to only call OneMerge#reorder when it has a chance to preserve the block structure, ie. either there are no blocks or blocks are identified by a parent field." The code comment on both guards: "Don't reorder if blocks can't be identified using the parent field." (IndexWriter.java:3550, :5547).
• The block guarantee. IndexWriter#addDocuments: "the index does not currently record which documents were added as a block. Today this is fine, because merging will preserve a block." (:1499-1500). IndexWriterConfig#setParentField: "In order to maintain the API guarantee that the document order of a block is not altered by the IndexWriter a marker for parent documents is required." (IndexWriterConfig.java:587-588).
• Old indexes are supported input. MIGRATE.md:136-145: "indexes created with Lucene 9.x remain fully supported (open and write)"; CHANGES GITHUB#16741: "no reindexing required" (CHANGES.txt:10-11).
• The asymmetry. Blocks without a parent field written by 9.9.2 are skipped on every release since 9.11.0 (executed): "unsupported" is implemented as "do not reorder", not "scramble".
• Lucene's own tests rely on the guard. MockRandomMergePolicy reverses doc IDs when no parent field is configured (MockRandomMergePolicy.java:240-247, :266-285), which is safe only because IndexWriter skips the reorder for blocks without a parent field. No upstream test opens a pre-9.9 block index under a reordering merge.
• History. #12685 (@s1monw, 9.9.0) added hasBlocks; a reviewer, @jpountz, asked to document that indexes written before 9.9 "will return false regardless". #13128 (@jpountz, merged 2024-03-29) built the guard on that flag five months later (#12685 was merged 2023-10-23); the PR has no review discussion and does not mention pre-9.9 segments.

The devil's advocate built the strongest "documented limitation" case, which the cross-model reviews rated moderate: the LeafMetaData note documents that the flag is false for pre-9.9 leaves; CHANGES GITHUB#13125 (9.11.0) says recursive graph bisection "is now supported on indexes that have blocks, as long as they configure a parent field" (CHANGES.txt:2742-2743), which a pre-9.9 index can never do; the @lucene.experimental addDocuments javadoc warns that "in the future Lucene may merge more aggressively re-order documents ... in which case you may need to fully re-index" (IndexWriter.java:1506-1508) and calls offline re-sorting tools "your own risk"; and with an index sort, blocks without a parent field are rejected only for indexes created with 10.0 or later, while an older index with such blocks is sorted without keeping them together (Sorter.java:192-207).

Why it fails: the LeafMetaData note describes the flag, not the guard, and nothing tells users of BPReorderingMergePolicy (whose javadoc does not mention blocks) that pre-9.9 data is exempt from the guard; the same unsupported configuration with 9.9+ data is handled safely, which is what the guard was added for; the addDocuments javadoc states "Today this is fine, because merging will preserve a block" in the present tense and its "own risk" note names offline tools outside IndexWriter; no CHANGES or MIGRATE entry asks BP users to reindex pre-9.9 block indexes; and index sorting without a parent field never kept blocks together, while here intact blocks are destroyed for an optional compression gain. Verdict: not intended, a gap in the guard. Residual risk: setParentField's javadoc and CHANGES GITHUB#13125 already tie block-safe reordering to a parent field, so upstream may still prefer a documentation note over disabling reordering for every merge that touches a pre-9.9 segment; expect "real but narrow", not "by design".

Option A was validated in a scratch worktree at ea57360912ed (removed afterwards); B, C and D were not run. Until a fix ships, users should not enable a reordering merge policy on an index whose blocks were first written before 9.9; reindexing into a new index with IndexWriterConfig#setParentField lets BP keep blocks together (CHANGES GITHUB#13125; not executed here).

Add mayHaveBlocks(LeafMetaData) = hasBlocks() || minVersion() == null || minVersion() before 9.9.0, and use it instead of LeafMetaData::hasBlocks in the merge guard and the addIndexes(CodecReader...) guard. A pre-9.9 block index without a parent field is then left alone, exactly like the same layout written by 9.9+.

+  private static boolean mayHaveBlocks(LeafMetaData metaData) {
+    return metaData.hasBlocks()
+        || metaData.minVersion() == null
+        || metaData.minVersion().onOrAfter(Version.LUCENE_9_9_0) == false;
+  }
...
-        readers.stream().map(LeafReader::getMetaData).anyMatch(LeafMetaData::hasBlocks)
+        readers.stream().map(LeafReader::getMetaData).anyMatch(IndexWriter::mayHaveBlocks)
...
-          mergeReaders.stream().map(LeafReader::getMetaData).anyMatch(LeafMetaData::hasBlocks)
+          mergeReaders.stream().map(LeafReader::getMetaData).anyMatch(IndexWriter::mayHaveBlocks)

Trade-offs

• Block-free merges that contain pre-9.9 documents are no longer reordered (they cannot be told apart from block indexes), so such indexes lose BP's compression until they are reindexed. This is the cost upstream will weigh.
• Protection persists: a merged segment keeps its oldest input's minVersion, so later merges of it are guarded too, also segments that 9.9.0-9.10.0 merged from pre-9.9 inputs.
• Uses the deprecated Version.LUCENE_9_9_0; Lucene's build disables deprecation lint, and :lucene:core:check passes.
• The minVersion() == null branch is only defensive: a leaf of an index created with 7.0 or later must have a minVersion (LeafMetaData.java:58-60, IndexWriter.java:3238-3242).
• Prevents new damage only; already scrambled segments need a reindex.
• Edge: a segment written today by a codec whose segment-info format cannot store hasBlocks still reads false with a current minVersion; only test codecs in backward-codecs write such formats.

What the checks validated

• Through gradle in a scratch worktree at ea57360912ed: this doc's test fails 2 of 4 on stock with assertions on and off (257 of 257 children each time) and passes 4 of 4 with the fix, on and off.
• With the fix: the whole :lucene:misc:test (315 tests, 9 skipped), TestBPReorderingMergePolicy with -Ptests.nightly=true (3 tests), the whole :lucene:backward-codecs:test (2,343 tests, 184 skipped, including the backward-compatibility indexes) and the whole :lucene:core:test (8,865 tests, 322 skipped), all with assertions on and 0 failures; :lucene:core:check and :lucene:backward-codecs:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis) successful. The suite runs used the files before tidy re-wrapped two javadoc paragraphs; the repro and check runs used the final files. A first check run, before tidy, exited 1 (runs/validate.console.txt); the re-run after tidy passed.
• The finder's BPReorderingMergePolicy test through gradle (with backward-codecs added to lucene/misc's test dependencies): 1 of 2 fails on stock (139), 2 of 2 pass with the fix, assertions on and off.
• Earlier, as a shadow class of the same guard change: the finder's test (2 of 2), the independent reproducer's (4 of 4), the security probe (7 of 7), a probe, and upstream TestSegmentReordering and TestBPReorderingMergePolicy (also nightly) pass.

Log through infoStream, or record a segment diagnostic, when a reorder is skipped because a leaf predates 9.9, so a BP user can see why nothing is reordered.

Trade-offs

Small; no behaviour change beyond A.

Let the application declare that an old index has no blocks, for example with a setter on BPReorderingMergePolicy, to keep reordering block-free pre-9.9 indexes.

Trade-offs

New API surface; the application then owns the risk.

Javadoc on BPReorderingMergePolicy and OneMerge#reorder, plus a MIGRATE/CHANGES note: indexes holding blocks written before 9.9 must be reindexed, with a parent field, before a reordering merge policy is enabled.

Trade-offs

Leaves silent, permanent corruption in place for anyone who misses the note.

Method: code reading (a hunt unit, cross-version round trips of indexes written by old releases, 2026-10-26), while building a release-jar differential with content-keyed full dumps.

• The finder, an AI tool, listed the per-segment metadata that main consults but old releases never recorded. LeafMetaData#hasBlocks, recorded only since 9.9.0, stood out as the only block signal of IndexWriter's reorder guard.
• The unit's differential then reproduced it: variant T6 (BPReorderingMergePolicy plus forceMerge of release-written segments) reported "BLOCKPARENT broken" for a 9.0.0-written case and was clean for a 10.0.0-written control.
• The finder minimised it into a public-API JUnit with a 9.8.0 fixture and a 9.9.2 control and validated a guard fix as a shadow class.
• Then: read-only reviews by an independent reviewer from another model family (OpenAI GPT-5.6, so no cross-model reviewer ran Java), an independent reproducer, a root-cause discriminator, a devil's advocate, an adjudication, a per-release history sweep, a prior-report search, a security assessment and probe, a credibility audit, and this doc's own test and fix validation. No model checker or prover was involved.