UniformSplit postings formats fail to flush or merge a field when deltaNumLines > targetNumBlockLines / 3, a setting the constructor accepts

UniformSplitPostingsFormat and STUniformSplitPostingsFormat (codecs module, opt-in) take targetNumBlockLines and deltaNumLines in their public constructors; the javadoc asks only 0 <= deltaNumLines < targetNumBlockLines and promises blocks of target ± delta terms, and validateSettings enforces exactly that. When the writer finishes a field, it writes the leftover terms as one block only if there are at most target − delta of them; otherwise it looks for a split point in the last 2 × delta terms, starting at index numLines − 2 × delta, and only asserts that this is at least 1. For a split of target − delta + 1 to 2 × delta terms, a range that exists exactly when 3 × delta > target, the start is 0 or negative; at a field's end such a split is either the whole leftover or what a first split of a larger leftover (up to target + delta − 1 terms) leaves. The flush, commit or merge then throws (AssertionError with assertions on; IndexOutOfBoundsException with them off, at exactly 2 × delta terms only for some terms), the IndexWriter closes through its tragic-event path and everything buffered since the last commit is lost; the last commit stays intact and passes CheckIndex. The smallest case is one document with 3 values under (4, 2). The defaults (32, 3) cannot fail. The code is unchanged since the formats were added (LUCENE-8753, 8.3.0).

1. Ordinary splits. BlockWriter.addLine appends one line per term and calls splitAndWriteBlock once the block holds targetNumBlockLines + deltaNumLines lines (BlockWriter.java:137-139). There the window starts at (target + delta) − 2 × delta = target − delta >= 1.
2. The end of a field. UniformSplitTermsWriter.writeFieldTerms calls finishLastBlock after the last term ("Flush remaining terms.", UniformSplitTermsWriter.java:333-334), which loops splitAndWriteBlock while lines remain (BlockWriter.java:147-153); the leftover has 1 to target + delta − 1 lines. When the first call splits normally (deltaStart >= 1), it leaves at most 2 × delta lines, and the loop splits those again.
3. The split. splitAndWriteBlock (BlockWriter.java:161-191) writes numLines <= target − delta as one block (:165-169); otherwise int deltaStart = numLines - deltaNumLines * 2; (:170), assert deltaStart >= 1 : "blockLines size: " + numLines; (:171), then blockLines.get(i) for i from deltaStart (:175-176), keeping the last line with the shortest prefix (<= at :178), and writes subList(0, minMdpEndIndex) (:184). A call with target − delta + 1 to 2 × delta lines has deltaStart <= 0. At a field's end it comes either directly (the leftover r has that many lines) or second: for a leftover r of 2 × delta + 1 to target + delta − 1 lines, the first call ends the block at the last shortest-prefix line k of the window and leaves r − k lines, which land in that range when k < r − (target − delta). With 3 × delta <= target every remainder is at most 2 × delta <= target − delta and is written as one block.
4. What is thrown. Assertions on: the assert fails. Assertions off, deltaStart < 0: blockLines.get(deltaStart) throws at :176 (here Index -1). Assertions off, deltaStart == 0 (a call of exactly 2 × delta lines): the loop picks index 0 only when line 0 has the strictly shortest prefix, and then addBlockKey's blockLines.get(0) on the empty sub-list throws (BlockWriter.java:278-282); otherwise (last shortest prefix at k >= 1) the split writes a valid block of k lines, but finishLastBlock splits the remaining 2 × delta − k lines again, and that split throws (Index -k at :176) when k < 3 × delta − target, possible when 3 × delta >= target + 2, e.g. (6, 4) or (32, 12); otherwise the field is written. The fact check ran this under (6, 4) with one 8-value field: a to h commits, a, aa … ag throws Index 0, a, b, ba, bb, bc, bd, be, bf throws Index -1 out of bounds for length 7; with assertions on all three throw blockLines size: 8, and with the fix all three commit. So with assertions off the failures are a strict subset of those with assertions on: each follows a call with deltaStart <= 0.
5. Both formats. STBlockWriter.finishLastBlock only calls super (STBlockWriter.java:86-90).
6. Validation lets it through. validateSettings (UniformSplitTermsWriter.java:238-260), called by the constructor (UniformSplitPostingsFormat.java:105), checks only target > 0, delta >= 0, delta < target and target + delta <= 1,000.
7. Why the writer closes. A flush failure is recorded as aborting, abort() runs and the exception is passed to IndexWriter.onTragicEvent (DocumentsWriterPerThread.java:585-589, :594-605; IndexWriter.java:453-454), the same for both exceptions; prepareCommitInternal then rolls the writer back (IndexWriter.java:3879-3883, :6098-6106). A merge failure calls tragicEvent(t, "merge") (IndexWriter.java:5109-5113).

The failing splitAndWriteBlock call always has target − delta + 1 to 2 × delta lines. It is either the field's whole leftover or what finishLastBlock's first split of a larger leftover leaves, so any leftover from target − delta + 1 to target + delta − 1 lines can fail, the larger ones only for some terms.

The (4, 2) 5-term case, run by the diagram probe: a, b, ba, bb, bc throws blockLines size: 4 with assertions on and Index 0 out of bounds for length 0 with them off; a, ab, b, ba, bb throws blockLines size: 3 and Index -1 out of bounds for length 3; a to e commits.

• Writer state (UniformSplit, both assertion modes): under (4, 2), commit 1 with one value succeeds; a 3-value document plus a document in another field then make commit 2 throw. The writer is closed (isOpen() false, a tragic exception recorded, the next addDocument throws AlreadyClosedException) and the reopened index has 1 of the 3 documents added; CheckIndex is clean.
• Merge path (both formats): two committed segments with 2 and 1 values each flushed fine; forceMerge(1) writes a 3-term field and throws, and the writer closes; the last commit (2 documents, all 3 values readable) is intact and CheckIndex-clean.
• Moderate settings: (32, 12) with a 22-value field, (64, 22) with 43 values and (100, 34) with 67 values throw with assertions on and off.
• Sweep: all 78 accepted settings with target 1..12, fields of 1..40 values, two term shapes, both formats: 0 failures when 3 × delta <= target; with assertions off 1,164 of 3,120 flushes fail for sequential terms and 914 of 3,120 for random terms. Fields of more than 2 × delta values, which reach the failing range only after a first split, account for 806 of 1,268 (sequential) and 600 of 1,268 (random) of them.

• Who. Applications that build UniformSplitPostingsFormat or STUniformSplitPostingsFormat through the public 5-argument constructors, or through a subclass and the protected constructors, with deltaNumLines > targetNumBlockLines / 3. The formats are opt-in (codecs module, @lucene.experimental) and the setting is non-default. Default-constructed and SPI-loaded instances use (32, 3) and cannot fail.
• No known user. A code search found no code that builds these formats with non-default settings (only copies of Lucene's own default-settings test), and Solr selects postings formats by SPI name, which uses the defaults. The search is not exhaustive.
• What it costs. The flush, commit or merge throws and the IndexWriter closes; everything buffered since the last commit is lost, including other threads' documents, since it is a writer-wide rollback (this follows from the rollback; all runs were single-threaded). The committed index stays consistent and CheckIndex-clean, and in the executed checks every assertions-off flush that did not throw read back all its terms. It is not index corruption.
• Merges stay blocked. A merge whose output field needs such a split fails again in each new writer session until the setting is changed; switching the field back to the defaults recovers (forceMerge(1) succeeds, CheckIndex clean).
• Scale. None needed: one document with 3 values under (4, 2). With a wide window failures are frequent: about a third of fields, e.g. 143 of 400 random fields under (32, 16) with assertions off. Just past the boundary failures are rare on ordinary data. With assertions off: 1 of 400 random fields under (32, 11), 4 of 400 under (64, 22), 5 of 400 under (100, 34), 26 of 400 under (32, 12).
• How likely: rare. Loud, and limited to a non-default setting nobody is known to use.

(4, 2) meets the documented rule 0 <= deltaNumLines < targetNumBlockLines. The test sits in org.apache.lucene.codecs.uniformsplit, where an upstream test would go, but uses only the public constructor; ByteBuffersDirectory and an explicit postings format fix the directory and codec choices that matter; the test framework still randomizes unrelated things such as locale and time zone, which do not affect the path. Line 45 is w.commit(); line 48 is the test's own wrapper, which says the commit was expected to succeed. If you add it upstream, wrap the long comment line for the formatter.

The writer close, the merge path and STUniformSplitPostingsFormat share the root cause and were executed by the longer checks (in the collapsed explanation above); they are not part of this repro.

Regression test: this same test; it passes with the recommended fix.

cp $REPRO/TestUniformSplitLargeDeltaFlush.java $LUCENE/lucene/codecs/src/test/org/apache/lucene/codecs/uniformsplit/
cd $LUCENE && ./gradlew :lucene:codecs:test --tests TestUniformSplitLargeDeltaFlush -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails too, with IndexOutOfBoundsException: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed)
TestUniformSplitLargeDeltaFlush > testCommitThreeTermsWithTarget4Delta2 FAILED
    java.lang.AssertionError: commit with UniformSplitPostingsFormat(4, 2) threw java.lang.AssertionError: blockLines size: 3
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:778092D94C5D7E5]:0)
        at org.apache.lucene.codecs.uniformsplit.TestUniformSplitLargeDeltaFlush.testCommitThreeTermsWithTarget4Delta2(TestUniformSplitLargeDeltaFlush.java:48)
        Caused by:
        java.lang.AssertionError: blockLines size: 3
            at org.apache.lucene.codecs.uniformsplit.BlockWriter.splitAndWriteBlock(BlockWriter.java:171)
            at org.apache.lucene.codecs.uniformsplit.BlockWriter.finishLastBlock(BlockWriter.java:149)
            at org.apache.lucene.codecs.uniformsplit.UniformSplitTermsWriter.writeFieldTerms(UniformSplitTermsWriter.java:334)
            at org.apache.lucene.codecs.uniformsplit.UniformSplitTermsWriter.write(UniformSplitTermsWriter.java:274)
            at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:162)
            at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:133)
            ...
            at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4215)
            at org.apache.lucene.codecs.uniformsplit.TestUniformSplitLargeDeltaFlush.testCommitThreeTermsWithTarget4Delta2(TestUniformSplitLargeDeltaFlush.java:45)

# same, -Ptests.asserts=false (JDK frames trimmed)
    java.lang.AssertionError: commit with UniformSplitPostingsFormat(4, 2) threw java.lang.IndexOutOfBoundsException: Index -1 out of bounds for length 3
        Caused by:
        java.lang.IndexOutOfBoundsException: Index -1 out of bounds for length 3
            at java.base/java.util.ArrayList.get(ArrayList.java:428)
            at org.apache.lucene.codecs.uniformsplit.BlockWriter.splitAndWriteBlock(BlockWriter.java:176)
            at org.apache.lucene.codecs.uniformsplit.BlockWriter.finishLastBlock(BlockWriter.java:149)

• The settings are inside the documented contract. The constructor javadoc (UniformSplitPostingsFormat.java:73-76, the same at UniformSplitTermsWriter.java:169-172): deltaNumLines "Must be greater than or equal to 0 and strictly less than {@code targetNumBlockLines}. The block size will be {@code targetNumBlockLines}+-{@code deltaNumLines}." The class javadoc: "Each block size (number of terms) is {@code targetNumBlockLines}+-{@code deltaNumLines}" (UniformSplitTermsWriter.java:48-50) and "A good sweet spot for the target block size is 32 with delta of 3 (10%) (default values). This can be tuned in the constructor." (:70-71). The range is loose at a field's end even today: the stock end-of-field split can write a block below target − delta (e.g. 24 lines for a 30-line leftover under the defaults (32, 3), BlockWriter.java:165-185); the defect is the exception on accepted settings, not the block size.
• Validation accepts them. validateSettings (UniformSplitTermsWriter.java:238-260) checks exactly the documented rule; (4, 2), (32, 11), (32, 12) and (64, 22) pass.
• A valid result exists. A failing split has at most 2 × delta < target + delta terms, so one block keeps the promised range, and the reader needs no block-size setting: with the fix every value reads back and CheckIndex is clean.
• Nothing limits delta further. No javadoc, test, CHANGES.txt or MIGRATE.md entry. Upstream tests use only (32, 3) (TestUniformSplitPostingFormat.java:43-48, TestSTUniformSplitPostingFormat.java:32-37) and BlockWriter(..., 10, 2, ...) (TestBlockWriter.java:45), all with 3 × delta <= target. The design discussion (#9798, lucene-solr #633, #676, #828) never discusses delta bounds or the last block, and git log -L on the body of splitAndWriteBlock finds only the introducing commit (with its javadoc, also a reformatting); the file's other commits do not touch the method.
• Sibling practice. BlockTree's validateSettings rejects settings its splitter cannot handle: "maxItemsInBlock must be at least 2*(minItemsInBlock-1)" (Lucene103BlockTreeTermsWriter.java:354-360). UniformSplit accepts the settings and then fails at flush.

The strongest case: these are @lucene.experimental formats in the codecs module; the javadoc calls 10% the sweet spot and the design describes delta as an "allowed delta variation (10%)"; the assert records the author's precondition; and nobody uses such settings.

Why it fails: @lucene.experimental warns that the API may change, not that documented arguments may crash the writer. The 10% is a recommended default, never a stated limit, and the same javadoc says the defaults can be tuned. The assert is internal, invisible to users and false for every accepted setting with 3 × delta > target. Low reach bears on priority, not on whether it is a bug. If large deltas are unwanted, the fix is to reject them in validateSettings, not to fail the flush and close the writer. The reading-only GPT reviews rated the intended-behaviour case "weak"; a later cross-model fact check rated it "moderate" (experimental, opt-in, an artificial setting, a loud failure that keeps the last commit) and noted that maintainers may reasonably prefer rejecting such settings in validation. That bears on which fix to lead with, not on whether the flush should throw.

Option A was built and run here. B, a simpler clamp, was run by the study and has a drawback. C, tightening validation, was not run.

Replace the assert deltaStart >= 1 in BlockWriter.splitAndWriteBlock (BlockWriter.java:171) with a branch that writes the remaining lines as one block and returns. addLine always splits at target + delta lines, where deltaStart >= 1, so the branch is reached only from finishLastBlock, directly or on its second call, and the blocks written before it are unchanged; settings with 3 × delta <= target behave exactly as today.

int deltaStart = numLines - deltaNumLines * 2;
if (deltaStart < 1) {
  // Only reachable from finishLastBlock, for the last lines of a field, when
  // 3 * deltaNumLines > targetNumBlockLines: write them as one block (numLines <= 2 *
  // deltaNumLines < targetNumBlockLines + deltaNumLines, so the block size stays in range).
  writeBlock(blockLines, dictionaryBuilder);
  blockLines.clear();
  return;
}

Trade-offs

• Block size. The last block then has target − delta + 1 to 2 × delta lines, inside the promised target ± delta; its split point is not chosen by the shortest-prefix rule, which affects only the dictionary key of one block per field.
• No format change. The reader needs no setting; with the fix every value reads back (iteration and seekExact), merges succeed and CheckIndex is clean.
• It drops the assert; the condition it guarded is now handled.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-31):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off;
• the org.apache.lucene.codecs.uniformsplit.* tests (the 7 upstream classes plus the minimal test) pass with assertions on and off: 8 classes, 83 tests, 0 failures, 4 skipped;
• the whole :lucene:codecs:test passes (26 classes, 693 tests, 0 failures, 78 skipped; run after the copied test was removed, assertions on); checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for codecs pass;
• earlier, with the fix loaded over the stock jars: the settings sweeps give 0 failures (3,120 flushes in the study, 12,480 in the tie-break), and the independent reproducer's 5 tests and the tie-break's production-API check pass with assertions on and off.

int deltaStart = Math.max(1, numLines - deltaNumLines * 2); and no assert.

Trade-offs

The finder's 4-case test (which includes the repro's (4, 2) 3-term case), the settings sweep (0 of 3,120 flushes fail) and the 7 upstream uniformsplit test classes pass. But, as the study's refute reviewer noted, it can write a 1-line block that is not the last block of the field (when the shortest prefix is at index 1). The stock end-of-field split also writes blocks below target − delta, but not 1-line ones when 3 × delta < target; so the first option is preferred. Not run in this doc's validation.

Add the check to validateSettings and the constructor javadoc, like BlockTree's 2*(minItemsInBlock-1) rule.

Trade-offs

Simple and fails fast at construction. But it narrows a documented and accepted range: applications whose data happens to avoid the window today (under (32, 11) about 1 field in 400 fails) would fail at construction after an upgrade; existing indexes stay readable, because the reader does not use the settings. The minimal test would have to expect an IllegalArgumentException from the constructor instead of a successful commit.

Method: semi-formal. An assertion failure seen while building fixtures for another hunt unit (a codec differential of automaton term intersection), outside that unit's target files; then code reading of BlockWriter.splitAndWriteBlock.

• Bounded-exhaustive sweep. All 78 settings with targetNumBlockLines <= 12 that the constructor accepts, every field size 1..40, on the stock jar: 1,164 of 3,120 flushes fail with assertions off, every one with 3 × delta > target and none otherwise; 0 with either fix sketch.
• Widening. An independent reproducer found the merge path and that the assertions-off failures are a subset; a devil's-advocate verifier reached it from another package through production API; the follow-up measured failure rates on random data just past the boundary.
• The minimal repro above is the smallest case: 3 terms under (4, 2).