LogMergePolicy with mergeFactor = Integer.MAX_VALUE makes commits throw once two size levels exist

LogMergePolicy sorts segments into size levels (the log, base mergeFactor, of each segment's size) and looks at windows of mergeFactor segments within a level; the end of each window is int end = start + mergeFactor. setMergeFactor accepts any value of 2 or more, including Integer.MAX_VALUE, which applications have used to mean "never merge". For every level after the first, start is at least 1, so the sum overflows to a negative number; the negative end passes the loop test, end - start overflows back to Integer.MAX_VALUE, and the minimum-size packing step reads levels.get(end) with that negative index (-2147483648 when the level starts at position 1). With the default IndexWriterConfig, IndexWriter asks the merge policy on every commit, near-real-time reader open and flush of new documents, so all of them throw IndexOutOfBoundsException until the factor is lowered. On main a 20,000,000-document segment followed by a 1-document flush is enough (a size ratio above 9,975,792); up to 9.5.0, 2,000 + 1 documents were. The overflow is as old as the class: 19 releases from 2.3.0 to 10.5.2 and main fail, only the exception changed over time, and on 9.5.0-10.1.0 (by code reading from 9.3.0) it is an OutOfMemoryError that closes the writer and loses uncommitted documents.

• Size level: LogMergePolicy gives each segment the level log(size) / log(mergeFactor) (size = documents for LogDocMergePolicy, bytes for LogByteSizeMergePolicy). A level runs from its first segment to the last segment whose level is within LEVEL_LOG_SPAN = 0.75 of the largest remaining level (2 x 0.75 when that largest one is below the minimum merge size); the segments after it start the next level.
• Window: inside a level, findMerges looks at runs of mergeFactor segments, from start to end = start + mergeFactor, and merges a run once it is full. start is the position of the level's first segment in the segment list.
• Packing step (since #14166, 10.2.0): when a full window is still smaller than the minimum merge size, it keeps adding the next segments, levels.get(end), levels.get(end + 1), ...

1. setMergeFactor checks only mergeFactor < 2 (LogMergePolicy.java:109-112). With Integer.MAX_VALUE = 2,147,483,647, log(mergeFactor) is 21.49.
2. Segments of 20,000,000 and 1 documents get levels 0.782 and 0. The first level spans 0.032 to 0.782 and holds only segment 0; segment 1 starts the second level, so start = 1. In general a second level exists exactly when the last segment in the list (normally the newest flush) is more than mergeFactor^0.75 = 9,975,792 times smaller than the largest one (if the largest is above the minimum merge size); order matters, so [20M, 1, 20M] stays one level.
3. int end = start + mergeFactor (:596) = 1 + 2,147,483,647 = 2,147,483,648, which wraps to -2,147,483,648. The same expression is at :680.
4. while (end <= 1 + upto) (:597) is true for a negative end; the window loop for (i = start; i < end; ...) runs zero times, so the window looks empty and small; end - start wraps back to 2,147,483,647, so the packing test at :629-632 passes.
5. The packing loop calls levels.get(end) (:636): IndexOutOfBoundsException: Index -2147483648 out of bounds for length 2. For a level that starts at position start, end is Integer.MIN_VALUE + (start - 1), so the index is -2147483647 when it starts at 2. Before 10.2.0 the same state reached new ArrayList<>(end - start), an array of 2,147,483,647 elements: OutOfMemoryError: Requested array size exceeds VM limit; on main, settings that make the packing test false (for example a minimum merge size of 0) reach that allocation too (read, not run).
6. IndexWriter.commit() asks the policy for merges through MergePolicy.findFullFlushMerges (MergePolicy.java:693), so the exception comes out of commit(); the same happens on near-real-time reader open and on flushes of new documents.

Direct findMerges calls on [N, 1] documents at Integer.MAX_VALUE: N = 2,000, 1,000,000, 9,000,000 and 9,900,000 return no merge; N = 10,000,000, 12,000,000 and 20,000,000 throw. Integer.MAX_VALUE - 1 and 10 return no merge on [20,000,000, 1]. The overflow needs mergeFactor > Integer.MAX_VALUE - start, so in practice only Integer.MAX_VALUE or values within about the segment count of it (Integer.MAX_VALUE - 1 fails once the second level starts at position 2). 1,000,000,000 passed the audit's 12,000,000 + 1 scenario and could overflow only for a level that starts after segment 1,147,483,647.

• Who is exposed. The default policy is TieredMergePolicy, so only applications that choose LogDocMergePolicy or LogByteSizeMergePolicy and set the factor to Integer.MAX_VALUE (or within about the segment count of it). The idiom is known: LUCENE-2331 (#3407, 2010) quotes the advice "if you want to prevent merges, set mergeFactor to a high enough value" and names "mergeFactor to Int.MAX_VAL", and third-party code calls setMergeFactor(Integer.MAX_VALUE). In 2.3.0-3.1 LogByteSizeMergePolicy was the default and IndexWriter.setMergeFactor reached it.
• What it takes on 9.6.0 and later. The last segment (normally the newest flush) more than about 1e7 times smaller than the largest one: for example a 1-document flush after a 12,000,000- or 20,000,000-document segment, or a small segment after a bulk load and forceMerge(1) of 12 million documents followed by a near-real-time reader open. With LogByteSizeMergePolicy, for example 50 GiB and 4 KiB segments (checked by a direct findMerges call only). Up to 9.5.0 a boundary at the minimum merge size also split levels whenever the largest segment was above it, so a 2,000-document segment and a 1-document flush were enough.
• Effect on main and 10.2.0+. With the default IndexWriterConfig, that writer's commits, near-real-time reader opens and flushes of new documents throw at once (2-5 ms) until the factor is lowered. The writer stays open; in the probes the last commit was intact (CheckIndex clean) and a committed delete stayed hidden; calling setMergeFactor(10) on the live policy makes the next commit persist every document. close() under the setting throws, closes the writer and loses the uncommitted documents. With the merge-on-commit/refresh wait maxFullFlushMergeWaitMillis set to 0 (default 500 ms), a commit or reader open without changes succeeds and a commit with changes throws only after it was written (read, not run).
• Effect on 9.5.0-10.1.0 (run) and, by code reading, 9.3.x-9.4.x. OutOfMemoryError: Requested array size exceeds VM limit: one rejected allocation, not heap exhaustion, but IndexWriter treats it as tragic, so the writer closes and uncommitted documents are lost; the last commit stays intact. With -XX:+ExitOnOutOfMemoryError the JVM exits inside commit() (run on 9.12.3). On 5.5.5-8.11.4 the same error left the writer open. By reading, it became tragic in 9.3.0 (LUCENE-10078), which raised the default merge-on-commit wait from 0 to 500 ms, so the commit asks the policy inside its tragic-error handler; on 9.0-9.2 it asks after the commit was written, as on 8.11.4 (9.0-9.4 not run).
• How likely. Rare: a non-default policy, a boundary value and, since 9.6.0, a 1e7 size ratio. No report was found in 19 years.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected findMerges to return null, but it threw java.lang.IndexOutOfBoundsException: Index -2147483648 out of bounds for length 2
       org.apache.lucene.index.TestLogMergePolicyMaxMergeFactor.testFindMergesWithMaxMergeFactor(TestLogMergePolicyMaxMergeFactor.java:61)
     Caused by: java.lang.IndexOutOfBoundsException: Index -2147483648 out of bounds for length 2
       java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
       java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
       java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
       java.base/java.util.Objects.checkIndex(Objects.java:365)
       java.base/java.util.ArrayList.get(ArrayList.java:428)
       [...]

• The setter accepts it. setMergeFactor (LogMergePolicy.java:102-112) throws only for values below 2 ("mergeFactor cannot be less than 2") and says "larger values ({@code > 10}) are best for batch index creation". No javadoc, CHANGES or MIGRATE entry states an upper bound.
• Integer.MAX_VALUE means "unbounded" in this very class: DEFAULT_MAX_MERGE_DOCS = Integer.MAX_VALUE (:57). getMergeFactor says the factor "controls the total number of segments allowed to accumulate in the index" (:94-97).
• The expected answer is clear. findMerges returns the merges that are "now necessary" (MergePolicy.java:602-610), nothing documents an exception for a valid setting, and Integer.MAX_VALUE - 1 and 10 return no merge on the same segments.
• Tests use only small factors: random 2-9 and 10-50 in the test framework, fixed values from 2 to 1,000, or 1 + the segment count; none near Integer.MAX_VALUE, and none expects an exception.
• History. The overflow came with the class in 2007; later commits changed only the exception (LUCENE-3084, LUCENE-6892, #14166) and the trigger (#12113), without discussing large factors.
• Precedent. In the sibling TieredMergePolicy, the byte-size setters setMaxMergedSegmentMB and setFloorSegmentMB saturate the double-to-long conversion instead of overflowing (TieredMergePolicy.java:107, :173); its integer setters are not clamped.

The strongest case for "intended" or "misuse" (built by a dedicated checker; the works-as-designed reviewer from another model family rated it moderate): NoMergePolicy is the documented way to stop merging ("Use it if you want to prevent segment merges"), added by LUCENE-2331 as the clean alternative to a high mergeFactor (its CHANGES entry: disable merges "without tweaking a particular MergePolicy parameters, such as mergeFactor"); a factor of 2^31 has no useful meaning as "segments merged at once"; MergePolicy is @lucene.experimental; and LogMergePolicy is not the default.

Why it fails: the setter accepts the value and its javadoc steers batch users to large values; LUCENE-2331 itself says NoMergePolicy "can REALLY allow you disable merges (except for maybe set mergeFactor to Int.MAX_VAL)", treating the value as working, and it was never deprecated or rejected; "experimental" allows API changes, not an IndexOutOfBoundsException from an accepted setting; and Integer.MAX_VALUE - 1 behaves correctly on the same data, so the failure is an arithmetic accident, not a limit. Verdict: not intended, a defect. Expect "use NoMergePolicy" as the first reply anyway, and offer both fixes; maintainers may reasonably prefer to reject such values instead of clamping.

One fix was built and validated in a clean scratch worktree at main a9779ef3c193; the alternatives are untested.

LogMergePolicy.java only: one import, two changed lines (:596 and :680) and a comment. start is between 0 and the segment count and mergeFactor between 2 and Integer.MAX_VALUE, so the sum read as an unsigned 32-bit value is exact; unsignedMin returns it unchanged whenever it did not overflow, and Integer.MAX_VALUE otherwise. With Integer.MAX_VALUE the window simply never fills, exactly as with Integer.MAX_VALUE - 1: no natural merge, which is what "never merge" users want. Lucene already writes window ends this way: MathUtil.unsignedMin(Integer.MAX_VALUE, windowMin + minWindowSize) in MaxScoreBulkScorer and MathUtil.unsignedMin(Integer.MAX_VALUE, offset + bitSet.length()) in BitSetDocIdStream.

+import org.apache.lucene.util.MathUtil;
 ...
       // Finally, record all merges that are viable at this level:
-      int end = start + mergeFactor;
+      // start >= 0 and mergeFactor >= 2: clamp instead of wrapping past Integer.MAX_VALUE
+      int end = MathUtil.unsignedMin(start + mergeFactor, Integer.MAX_VALUE);
 ...
         start = end;
-        end = start + mergeFactor;
+        end = MathUtil.unsignedMin(start + mergeFactor, Integer.MAX_VALUE);

Trade-offs

• No behaviour change for any value that does not overflow; no allocation, one unsigned compare per window.
• Keeps accepting Integer.MAX_VALUE as "never merge", so existing callers keep working.
• Equivalent spelling without the helper: int end = (int) Math.min((long) start + mergeFactor, Integer.MAX_VALUE); (not run).
• Backport: both lines read the same on branch_10x and in the 9.x and 10.x releases (up to 10.1 the overflow reaches new ArrayList<>(end - start)). MathUtil.unsignedMin exists only from 10.3.0, so 9.x and 10.0-10.2 need the long-arithmetic spelling or a backport of the helper (read, not run there). Add a CHANGES entry under Bug Fixes.
• Open PR #16847 (more than 2^31 documents) edits findMerges and keeps int end = start + mergeFactor, so one of the two will need a rebase.
• It gives Integer.MAX_VALUE the meaning "never merge"; maintainers who prefer to call such values invalid would choose the setter bound instead (or both).

What the checks validated

• Clean scratch worktree at a9779ef3c193, gradle, seed DEADBEEF, removed afterwards. Stock: the repro fails 2 of 3 with assertions on and off. With the fix: 3 of 3 pass.
• With assertions on: the repro plus TestLogMergePolicy (13 tests), TestIndexWriterMergePolicy (21), TestIndexWriterMerging (13) and TestForceMergeForever (1): 51 tests, 0 failures, 4 skipped. With assertions off: the repro plus every test matching org.apache.lucene.index.*Merg*: 52 classes, 231 tests, 0 failures, 8 skipped. With -Ptests.nightly=true: TestLogMergePolicy 13 of 13 and TestIndexWriterMergePolicy 19 of 21 (2 skipped).
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest of lucene/core pass with the fix and the test in place.
• Earlier, the same fix shadowed onto the stock jars: the finder's repro 3 of 3 and the independent reproducer's 5 of 5 pass (main ea57360912ed); on a9779ef3c193 the diagram probe gets no merge for every size and both commits succeed.

Reject values above some limit with an IllegalArgumentException, or clamp them when they are set (untested).

Trade-offs

Turns a delayed commit failure into an immediate, clear error, but rejecting breaks code that passes Integer.MAX_VALUE today to mean "never merge" (LUCENE-2331's idiom) and needs a MIGRATE note; the safe bound depends on the segment count, which the setter does not know. Clamping on set (for example to Integer.MAX_VALUE / 2) avoids the break but changes what getMergeFactor returns. Can be combined with the clamp.

Point users who want to stop merges to NoMergePolicy in the setMergeFactor javadoc.

Trade-offs

Useful in addition, but an accepted value still breaks commits.

Use NoMergePolicy to stop merges (available from 3.1), or a large but not extreme factor such as 1,000,000,000 (passed the audit scenario). On main and 10.2.0+, a writer that is already failing recovers when setMergeFactor(10) (any sane value) is called on its live policy before the next commit; do not close it under the bad setting. On 9.5.0-10.1.0 (by reading also 9.3-9.4) the first failure already closed the writer: open a new writer with a sane factor on the intact last commit and add the uncommitted documents again.

Trade-offs

Needs the user to know about the bug.

Method: a bytecode inventory plus SMT proofs (the study's "formal" tier).

• An abstract interpreter over the class files (written with the JDK 25 java.lang.classfile API) listed every iadd of two non-constant ints whose result flows into a loop comparison. One row was LogMergePolicy.findMerges add-var L596 -> cmp@597.
• The hunting AI tool wrote three bit-vector obligations for Z3, cross-checked with cvc5: LM1 asks whether start + mergeFactor can wrap with start >= 1 and mergeFactor >= 2 (counterexample: yes); LM1s proves the sum is exact whenever mergeFactor <= Integer.MAX_VALUE - start; LM2 proves that in the wrapped case end is negative, both loop tests (end <= 1 + upto, end < 1 + upto) hold and end - start >= mergeFactor. The rest of the packing test (minMergeSize < maxMergeSize, mergeSize < minMergeSize, no merging segment) is not modelled: it holds for the default settings by reading the code and in every executed run.
• The same AI tool then wrote a JUnit test on the stock jars (a direct findMerges call and an end-to-end IndexWriter run), which failed on main with assertions on and off, on 10.5.2 and, with OutOfMemoryError, on 9.12.3.