TemporalMergePolicy with maxThreshold Integer.MAX_VALUE: forceMergeDeletes() throws OutOfMemoryError

The finding in 30 seconds

TemporalMergePolicy groups segments into time windows and merges at most maxThreshold of them at once per window. For forceMergeDeletes() it collects each window's segments into batches of maxThreshold, and it allocates the batch list at that full size before adding a single segment.

A diagram in which an index of two segments in one time window, each with one deleted document, flows through the three steps of TemporalMergePolicy (keep segments over 10% deleted, group them by time window, cut each window into merges of at most maxThreshold segments, with step 3 outlined in red) into one merged segment without deleted documents, above an orange Expected panel where the two segments become one segment with no deleted documents and a blue Actual panel where step 3 asks for room for 2,147,483,647 segments and fails with OutOfMemoryError. — With maxThreshold set to Integer.MAX_VALUE, the step that cuts a time window into merges asks for room for 2,147,483,647 segments up front, so forceMergeDeletes() throws OutOfMemoryError and merges nothing. Generated with AI, checked against the code.

setMaxThreshold(Integer.MAX_VALUE) passes the check 'maxThreshold < minThreshold'; forceMergeDeletes() finds one time window with two 2-document segments, each with 1 deleted document; buildSequentialMerges calls new ArrayList<>(maxThreshold), an array of 2,147,483,647 slots for 2 segments, and the JVM throws java.lang.OutOfMemoryError: Requested array size exceeds VM limit at TemporalMergePolicy.java:957; nothing is merged and both deletes stay. With new ArrayList<>(Math.min(maxThreshold, candidates.size())) the list has 2 slots and the two segments merge into one with 0 deleted documents. — The repro's path: the setter's lower-bound check lets `Integer.MAX_VALUE` through; `forceMergeDeletes()` reaches `buildSequentialMerges` with 2 segments, which asks for a list of 2,147,483,647 slots and gets `OutOfMemoryError`. With the capacity bounded by the candidates, the 2 segments merge into 1 with 0 deletes. Select the still to open it at full size. Generated with AI, checked against the code.

Configure new TemporalMergePolicy().setTemporalField("ts").setMaxThreshold(Integer.MAX_VALUE). The setter only checks that the value is at least minThreshold (4), and its javadoc says larger values merge more aggressively. Natural merges and forceMerge(1) work with it.
Index 4 documents with the same timestamp, committing after every 2: one time window now holds two 2-document segments. Delete one document in each, 50% of each segment, above the default forceMergeDeletesPctAllowed of 10%.
Call forceMergeDeletes(). You expect one merged segment with 0 deleted documents, which is what the default maxThreshold of 8 gives for the same index.
Instead it throws java.lang.OutOfMemoryError: Requested array size exceeds VM limit (on the tested OpenJDK 25 HotSpot JVM, with a 256 MB and a 4 GB heap), with assertions on and off. Nothing is merged and both deletes stay; the writer stays usable unless the JVM is set to exit on OutOfMemoryError.
Why: for each window with 2 or more such segments, buildSequentialMerges first creates new ArrayList<>(maxThreshold) (TemporalMergePolicy.java:957), an array of 2,147,483,647 slots for 2 segments, which the JVM refuses outright.

Who is affected

Only applications that opt in to the experimental TemporalMergePolicy (new in 10.5.0), set maxThreshold too large to allocate (Integer.MAX_VALUE at any heap size; huge finite values on a small heap) and call forceMergeDeletes(). Every such call fails once one window has two segments with enough deletes. The deletes stay; a JVM with -XX:+ExitOnOutOfMemoryError exits.

Mechanism, terms and numbers492 words

TemporalMergePolicy (@lucene.experimental, first released in 10.5.0) merges segments only within time windows, at most maxThreshold at once. setMaxThreshold rejects only values below minThreshold (TemporalMergePolicy.java:246-249 at 5adff78bb168) and says "Larger values allow more aggressive merging", so Integer.MAX_VALUE is accepted, and natural merges and forceMerge(1) work with it. But IndexWriter.forceMergeDeletes() sends every time window with two or more segments over forceMergeDeletesPctAllowed to buildSequentialMerges, which first allocates new ArrayList<>(maxThreshold) (:957). The JVM refuses an array of 2^31-1 slots, so the call throws java.lang.OutOfMemoryError: Requested array size exceeds VM limit (on the tested HotSpot JVM at any heap size), for two 2-document segments, before any merge is registered, and the deletes stay. The writer is not tragic and stays usable, unless the JVM runs with -XX:+ExitOnOutOfMemoryError, which then exits. Huge finite values fail too when the heap is too small. A one-line fix bounds the capacity by the number of candidates.

The bug, step by step

Setter (TemporalMergePolicy.java:245-252 at 5adff78bb168): setMaxThreshold throws only when maxThreshold < minThreshold (:246-249; minThreshold defaults to 4, :103). Integer.MAX_VALUE is stored.
Forced-deletes merges (findForcedDeletesMerges, :528-597): segments whose share of deleted documents is strictly above forceMergeDeletesPctAllowed (default 10, :111; the test at :555) are grouped by time window (:568; the window comes from each segment's min and max LongPoint value of the temporal field, read from the segment files, :615-622, :675-707, so equal timestamps share a window); windows with fewer than two are skipped (:573); the rest go to buildSequentialMerges(bucketSegments) (:582).
Batching (buildSequentialMerges, :951-978): List<SegmentCommitInfo> batch = new ArrayList<>(maxThreshold); (:957) runs before any candidate is added. Then each candidate is added, a merge is emitted when batch.size() == maxThreshold (:961), and a final merge takes any rest of two or more (:970).
The JVM limit: new ArrayList<>(Integer.MAX_VALUE) asks for an Object[] of length 2^31-1, above HotSpot's maximum array length, so the JVM throws OutOfMemoryError: Requested array size exceeds VM limit without trying to allocate. On the tested OpenJDK 25 HotSpot JVM the heap size does not matter (independent checks got the same Error with a 256 MB and a 4 GB heap); other JVMs were not tested.
To the caller: IndexWriter.forceMergeDeletes asks the policy inside synchronized (this) before registering anything (IndexWriter.java:2358-2359), so the Error propagates and no merge runs.

Every other use of maxThreshold in the merge code is a bound or a comparison: Math.min at :495 and :935, loop and equality bounds at :878, :890 and :961 (the remaining uses are the field, the setters' checks and store, the getter and toString: :108, :191, :246, :250, :260, :605). That is why natural merges and forceMerge(1) work with the same setting; :957 is the only place that turns the value into an allocation size.

Exposure and likelihood352 words

Opt-in only. TemporalMergePolicy is not a default and is @lucene.experimental (:94); the default maxThreshold is 8 (:108). forceMergeDeletes() is documented as "often a horribly costly operation; rarely is it warranted" (IndexWriter.java:2413).
No scale needed. Two 2-document segments in one window trigger it, on every such call (a second call on the same writer fails again; independent check).
Effect. An Error, not an IOException, so a caller that catches Exception does not catch it. No merge runs and the deletes stay until a later merge picks up those segments: a natural merge looks at a window only once it holds at least minThreshold (default 4) segments (:833-835), so for a two-segment window it may never come, while an explicit forceMerge(1) does merge them (independent side check). In a JVM that is not set to exit on OutOfMemoryError, the writer is not tragic: getTragicException() is null, isOpen() is true, and it accepts more documents and a commit (independent side checks). A JVM run with -XX:+ExitOnOutOfMemoryError exits with code 3 (independent checks: a plain new ArrayList<>(Integer.MAX_VALUE), and the full forceMergeDeletes() path). A JVM with -XX:+HeapDumpOnOutOfMemoryError writes a heap dump on the first such Error (HotSpot dumps once per JVM: three failing calls in one JVM wrote one dump). Lucene's own test JVM, which uses that flag, wrote one of about 50 MB per run here; in a service the dump size follows the heap in use (not measured).
Not affected. Natural merges and forceMerge(1) with the same setting, and every maxThreshold small enough to allocate (such as the default 8).
Huge finite values. :957 sizes the list by any value. On the tested JVM, with the repro's index: Integer.MAX_VALUE - 1 gives the same "Requested array size exceeds VM limit" with a 256 MB and a 4 GB heap; Integer.MAX_VALUE - 2 and - 8 give OutOfMemoryError: Java heap space even with 4 GB; 10^8 and 10^9 give Java heap space with 256 MB and work with 4 GB; 8 works with both (independent check). Only values near Integer.MAX_VALUE fail whatever the heap.

Minimal repro

One JUnit test method on the public API: maxThreshold Integer.MAX_VALUE, two 2-document segments in one time window, one delete in each, forceMergeDeletes(), then assert 0 deleted documents. One test thread; it fails with every seed tried and with assertions on and off; the test method takes under a second.

TestTemporalMergePolicyMaxThreshold.java (one test, public API)50 lines

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.index;

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.LongPoint;
import org.apache.lucene.document.StringField;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.util.LuceneTestCase;

public class TestTemporalMergePolicyMaxThreshold extends LuceneTestCase {

  public void testForceMergeDeletesWithMaxValueMaxThreshold() throws Exception {
    TemporalMergePolicy mp =
        new TemporalMergePolicy().setTemporalField("ts").setMaxThreshold(Integer.MAX_VALUE);
    try (Directory dir = newDirectory();
        IndexWriter w = new IndexWriter(dir, new IndexWriterConfig().setMergePolicy(mp))) {
      for (int i = 0; i < 4; i++) {
        Document doc = new Document();
        doc.add(new StringField("id", Integer.toString(i), Field.Store.NO));
        doc.add(new LongPoint("ts", 1_700_000_000L));
        w.addDocument(doc);
        if (i % 2 == 1) {
          w.commit();
        }
      }
      w.deleteDocuments(new Term("id", "0"), new Term("id", "2"));
      // expected: deletes expunged; actual: OutOfMemoryError (array size exceeds VM limit)
      w.forceMergeDeletes();
      try (DirectoryReader r = DirectoryReader.open(w)) {
        assertEquals(0, r.numDeletedDocs());
      }
    }
  }
}

Command

cp $REPRO/TestTemporalMergePolicyMaxThreshold.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestTemporalMergePolicyMaxThreshold -Ptests.seed=5EED1213 -Ptests.asserts=true
# and with -Ptests.asserts=false (assertions off)

Expected: The two segments are merged into one and numDeletedDocs() is 0, as with the default maxThreshold of 8.
Observed: forceMergeDeletes() throws java.lang.OutOfMemoryError: Requested array size exceeds VM limit at TemporalMergePolicy.java:957, with assertions on and off.

All files of this repro: repro/LFV-1213/

More files, environment, full output178 words

Pass -Ptests.asserts explicitly to choose assertions on or off; the result is the same. Lucene's test JVM runs with -XX:+HeapDumpOnOutOfMemoryError, so each failing run leaves a heap dump of about 50 MB (java_pid<N>.hprof) in lucene/core/build/tests-cwd/; delete it afterwards.

The test framework still randomizes the codec and the directory, but the failure did not depend on the seed (fixed and fresh seeds), and the Error is thrown in the test thread before any merge is registered. As a regression test, the same method can go into the existing TestTemporalMergePolicy.

Environment

lucene	`main 5adff78bb168 (12.0.0-SNAPSHOT): this test fails; an earlier, equivalent variant (two 4-document segments) also fails on 10.5.0, 10.5.1, 10.5.2 and 10.6.0`
jdk	`OpenJDK 25.0.3 (HotSpot)`
os	`Linux aarch64 (the only platform run)`

Observed output

TestTemporalMergePolicyMaxThreshold > testForceMergeDeletesWithMaxValueMaxThreshold FAILED
    java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.base/java.util.ArrayList.<init>(ArrayList.java:157)
        at org.apache.lucene.index.TemporalMergePolicy.buildSequentialMerges(TemporalMergePolicy.java:957)
        at org.apache.lucene.index.TemporalMergePolicy.findForcedDeletesMerges(TemporalMergePolicy.java:582)
        at org.apache.lucene.index.IndexWriter.forceMergeDeletes(IndexWriter.java:2359)
        at org.apache.lucene.index.IndexWriter.forceMergeDeletes(IndexWriter.java:2425)
        at org.apache.lucene.index.TestTemporalMergePolicyMaxThreshold.testForceMergeDeletesWithMaxValueMaxThreshold(TestTemporalMergePolicyMaxThreshold.java:44)

# main 5adff78bb168, gradle, seed 5EED1213; the same with -Ptests.asserts=true and =false
# with the fix: BUILD SUCCESSFUL, 1 test, 0 failures (assertions on and off)

Why it looks unintended

The setter checks only a lower bound and says "Larger values allow more aggressive merging"; there is no documented "no limit" value.
Everywhere else in the merge code maxThreshold is a bound, so natural merges and forceMerge(1) work with it (executed); only :957 allocates by it.
Upstream fixed the same shape before: LUCENE-4938 (min(maxDoc, n), tested with Integer.MAX_VALUE) and LUCENE-504.

The full argument198 words

Contract. setMaxThreshold (TemporalMergePolicy.java:234-244): "Sets the maximum number of segments to merge at once within a time window. Larger values allow more aggressive merging (reducing segment count faster) but increase the cost of individual merge operations." and "Must be at least equal to setMinThreshold". Only that lower bound is enforced (:246-249); values below minThreshold are rejected, so there is no documented "no limit" value. A finite value above any segment count a window can reach would act as one, but nothing says how large is safe.
Idiom. Lucene itself uses Integer.MAX_VALUE as a merge-policy "no limit": LogMergePolicy.DEFAULT_MAX_MERGE_DOCS = Integer.MAX_VALUE (LogMergePolicy.java:57).
Tests and review. No javadoc, CHANGES or MIGRATE entry mentions the pre-allocation. The upstream tests use maxThreshold 3, 4 and 10. The review of #15620 (17 issue comments, 21 review comments, 14 reviews) discussed the default thresholds, never an upper bound or the batch capacity.
Precedent. LUCENE-4938 (#6002, 2013): IndexSearcher pre-allocated by n when sorting and was fixed with min(maxDoc, n); Robert Muir asked for "simple tests ... that tests with Integer.MAX_VALUE". LUCENE-504 (15743fc1793, 2009): FuzzyQuery's queue was changed to grow dynamically "to support BooleanQuery.maxClauseCount(Integer.MAX_VALUE) without exhausting all memory".

The devil's-advocate check163 words

The strongest case for "intended": no index has 2^31 segments in one window, so Integer.MAX_VALUE is not a real segment count; the javadoc warns that larger values cost more; the class is experimental and new; forceMergeDeletes is rarely warranted; and the failure is loud. So it is an absurd setting and the user's problem.

Why it fails: the setter accepts the value, invites larger values and documents no other "no limit" value; the same setting works on the other merge paths (natural merges and forceMerge(1), executed); the failure is an OutOfMemoryError from two tiny segments, not a validation error that names the argument; and upstream treated the same pre-size-by-a-huge-limit shape as a bug (LUCENE-4938, LUCENE-504). Having no upper bound is not a promise that every extreme value is cheap, but a list for two segments should not need 2^31-1 slots: what remains is mainly an impact argument, which is why it is Low. Verdict: a real Low bug with a one-line fix.

Suggested fix

Recommended A. Bound the initial capacity by the number of candidates (validated)

Size the batch list by what it can hold: new ArrayList<>(Math.min(maxThreshold, candidates.size())) at TemporalMergePolicy.java:957. The repro passes with assertions on and off, and the existing TemporalMergePolicy tests pass. A plain new ArrayList<>() would also avoid it (not run); an upper bound in the setter is not recommended.

     MergeSpecification spec = null;
-    List<SegmentCommitInfo> batch = new ArrayList<>(maxThreshold);
+    List<SegmentCommitInfo> batch = new ArrayList<>(Math.min(maxThreshold, candidates.size()));

All options and trade-offs3 options · 253 words

Option A was applied in a scratch worktree at 5adff78bb168 and checked with gradle; option B was not run; option C is listed only to say why not.

A. Bound the initial capacity by the number of candidates (validated) (recommended)

One line at TemporalMergePolicy.java:957. A batch never holds more than min(maxThreshold, candidates.size()) segments, so nothing else changes.

     MergeSpecification spec = null;
-    List<SegmentCommitInfo> batch = new ArrayList<>(maxThreshold);
+    List<SegmentCommitInfo> batch = new ArrayList<>(Math.min(maxThreshold, candidates.size()));

Trade-offs

None in behaviour: the capacity is only a sizing hint, and the merges emitted are the same for every maxThreshold that worked before.

What the checks validated

The repro passes with assertions on and off (gradle, seed 5EED1213).
With the fix, TestTemporalMergePolicy (7 tests) and TestTemporalMergePolicyAcrossBuckets (4 tests) pass, 0 failures, with assertions on and, in a separate run, off.
:lucene:core:checkGoogleJavaFormat passes with the fix and the test file.
Independent checks shadowed the same change and got the same result with their own tests.

B. No capacity hint

new ArrayList<>(); the list grows as needed and is reused after clear().

Trade-offs

Equally small; a few extra array copies for windows with more than 10 candidates. Not run for this doc.

C. An upper bound in setMaxThreshold (not recommended)

Reject values above some limit with an IllegalArgumentException.

Trade-offs

It rejects a "no limit" setting that the other merge paths handle (natural merges and forceMerge(1), executed), and any bound is arbitrary; unless the bound is low, large finite values below it would still over-allocate.

Downstream note

A product that lets its users set this merge-policy threshold and passes Integer.MAX_VALUE through would see every forced-deletes call that finds two eligible segments in one time window fail; a modest value such as the default 8 avoids it.

How it was found

A proof-obligation audit of TemporalMergePolicy's setMaxThreshold contract: the setter admits every value from minThreshold up to Integer.MAX_VALUE, so every path that uses maxThreshold must work for all of them, and buildSequentialMerges allocates by it, which fails for the top value. A one-method JUnit test on the public API then witnessed it.

The method in full73 words

The obligation came from the setter's own validation (lower bound only) and its javadoc ("Larger values allow more aggressive merging"), checked against each use of the field in the class: every use is a bound or a comparison except one, :957, which is an allocation size. The witness test (an earlier, equivalent variant of the repro above) fails on main with assertions on and off and on the 10.6.0 and 10.5.0 jars.

Prior reports, history and checks

Prior reports

No prior report found in apache/lucene issues, PRs and commits (incl. the JIRA archive), the mailing lists, Solr, Elasticsearch and OpenSearch, or the web (searched 2026-11-03; GitHub re-checked the same day). The only item on this code is #15620, the PR that added the class.

Searches3 searches

Nearest items, none of them this bug: #15620 (the introducing PR; its review asked how the default thresholds were chosen and never discussed large values or the batch capacity), #6906 (LUCENE-5844, ArrayUtil.grow and the VM array limit, another class), #9025 (LUCENE-7976) and #9300 (LUCENE-8253), forced deletes in other merge policies. No open PR touches TemporalMergePolicy.java. Public forks that sync upstream main carry the same line.

Related, not a duplicate: LFV-213, in the same method (forceMergeDeletes skips a segment that is the only eligible one in its time window; the size guards at :573, :952 and :970, another fix).

What was searched, when, and what turned up
Where	When	How	Found
GitHub issues, PRs and commits (apache/lucene incl. the migrated JIRA archive)	2026-11-03; re-checked the same day	a scripted prior-report search (16 issue and PR, 3 commit, 4 JIRA-archive queries) and manual searches: TemporalMergePolicy, temporal merge, time-series merge policy, maxThreshold, buildSequentialMerges, setMaxThreshold, "Requested array size exceeds VM limit", forceMergeDeletes OutOfMemoryError; git log -S on the line	no report; #15620 is the introducing PR; #6906, #9025, #9300 are other classes or causes
Open and recent PRs	2026-11-03	the touched files of the 196 open apache/lucene PRs updated since 2026-03-17, the 13 closed or merged since 2026-11-01, and the 11 updated on 2026-11-03	none touches TemporalMergePolicy.java or its tests
Mailing lists, downstream, web	2026-11-03	dev@, java-user@, issues@, commits@ (20 queries); Solr (GitHub and JIRA), Elasticsearch, OpenSearch; 6 web queries; all comments and reviews of #15620	nothing about the batch capacity or an OutOfMemoryError from forceMergeDeletes

History

Introduced with the class by #15620 (1596b7604b6, merged 2026-03-17); first released in 10.5.0. 10.5.0, 10.5.1, 10.5.2, 10.6.0 and main fail; 10.4.0 and 9.x have no such class. Not fixed upstream.

Versions tested4 rows

#15620 "Add Temporal Merge Policy for time-series data" (churromorales) was merged to main as 1596b7604b6 on 2026-03-17; the branch_10x copy 4a2c3fd0901 (same author and date) is what the 10.5.x and 10.6.0 releases ship. git log -S 'new ArrayList<>(maxThreshold)' over the full history finds only these two commits; the line is already in the PR's first revision (2026-01-26), and the review changed neither it nor the setter's check. TemporalMergePolicy.java is byte-identical in 10.5.0 to 10.6.0. CHANGES.txt lists #15620 under the 10.4.0 heading, but the 10.4.0 core jar has no such class, so CHANGES.txt cannot date this bug. The release runs used an earlier, equivalent variant of the repro (two 4-document segments, one delete each).

Versions and branches executed, with results
Version or commit	Verdict	Result
main `5adff78bb168`	affected	gradle: fails with assertions on and off at `:957`; passes with the fix
10.6.0	affected	fails at `:957`, assertions on and off
10.5.0, 10.5.1, 10.5.2	affected	fails at `:957`
10.4.0, 9.12.3	no class	TemporalMergePolicy does not exist; the test does not compile

Independent checks

An independent reproducer, an executed cross-model review (OpenAI GPT-5.6, real, high), a devil's-advocate verifier and an adjudication agree, and separate re-runs reproduced it. This doc's gradle runs reproduced it on main and validated the fix.

Checks and verdicts313 words

The cross-model review ran its Java on main ea57360912ed, whose TemporalMergePolicy.java is identical to 5adff78bb168's.

Each independent check and its verdict
Check	Verdict	What it found
independent reproducer	real	Own test, written from the description alone (two 2-document segments, 50% deletes), fails on main, 10.6.0 and 10.5.2. Side checks: `maxThreshold` 8 merges the same layout; `forceMerge(1)` and a natural merge work with `Integer.MAX_VALUE`; after the Error the writer is not tragic and the 2 deletes remain.
cross-model review, refute lens (OpenAI GPT-5.6, executed)	real (high)	Own test fails with assertions on and off (three seeds) while the same test with `maxThreshold` 4 passes; option A passes its tests and the existing TemporalMergePolicy test classes; the "intended" argument rated weak.
devil's-advocate verifier	real	No doc, test or review supports the pre-allocation; precedents LUCENE-4938 and LUCENE-504. Probe: natural merges and `forceMerge(1)` work, the writer stays usable; `-XX:+ExitOnOutOfMemoryError` exits with code 3.
adjudication and separate re-runs	confirmed, Low	Fresh-seed re-runs fail on main with assertions on and off; the minimal test passes with option A; one minimal deterministic test.
this doc's fix check	reproduced; option A validated	Gradle in a scratch worktree at `5adff78bb168`: the repro fails without and passes with option A, assertions on and off; `TestTemporalMergePolicy` and `TestTemporalMergePolicyAcrossBuckets` (11 tests) pass.
independent fact check of this doc	real; wording corrected	Re-ran the repro, the fix and side checks in its own worktree; JVM probes showed one heap dump per JVM, the result for huge finite values and `-XX:+ExitOnOutOfMemoryError` exit 3 on the full path. Its two wording errors (heap dump per call; the downstream note) are corrected.
cross-model fact check of this doc (OpenAI GPT-5.6)	real; Low framing right	Re-read the code and logs; asked for narrower wording on huge finite values, later merges, "every other use" and the JVM tested, now applied.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

PNG 528 KB

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,364 words

Finding record

LFV-1213, snapshot of 2026-11-06

Registry title: TemporalMergePolicy with setMaxThreshold(Integer.MAX_VALUE), a value the setter accepts, makes IndexWriter.forceMergeDeletes() throw OutOfMemoryError ('Requested array size exceeds VM limit') and merge nothing, because buildSequentialMerges pre-sizes its batch list to maxThreshold
Area: index / merge policy (TemporalMergePolicy forced-deletes merges, IndexWriter.forceMergeDeletes)
Priority: Low, rank 1011: TemporalMergePolicy with maxThreshold set to Integer.MAX_VALUE, which the setter accepts, makes forceMergeDeletes throw OutOfMemoryError for two tiny segments and leave the deletes in place, in 10.5.0 through 10.6.0. It needs an experimental opt-in policy, an extreme setting and a rarely warranted call, and it fails loudly with the writer still usable, though a JVM set to exit on OutOfMemoryError exits.
Impact class: crash
Impact note: Hits only applications that choose TemporalMergePolicy (opt-in, @lucene.experimental, first released in 10.5.0), set maxThreshold to Integer.MAX_VALUE to mean 'no batch limit', and call forceMergeDeletes() while a time window holds two or more segments over forceMergeDeletesPctAllowed: the call throws an OutOfMemoryError before any merge is registered, so the deletes stay, at any heap size and with no scale (two 4-document segments); natural merges and forceMerge are unaffected and the writer is not tragic and stays usable, but a JVM run with -XX:+ExitOnOutOfMemoryError exits.
Summary: You configure TemporalMergePolicy with setMaxThreshold(Integer.MAX_VALUE), which the setter accepts (it checks only maxThreshold >= minThreshold and says larger values merge more aggressively), and call IndexWriter.forceMergeDeletes() while one time window holds two segments whose share of deleted documents is above forceMergeDeletesPctAllowed, for example two 4-document segments with one deleted document each. You expect the two segments to be merged into one with no deleted documents left, as happens with the default maxThreshold of 8, and as natural merges and forceMerge do with the same setting. Instead buildSequentialMerges creates its batch list with new ArrayList<>(maxThreshold) before adding anything, so forceMergeDeletes() throws java.lang.OutOfMemoryError: Requested array size exceeds VM limit at any heap size, no merge is registered and the deleted documents stay.
Introduced: 1596b7604b67ed47314aa648f14b91dc75ce8134 (apache/lucene#15620 'Add Temporal Merge Policy for time-series data' by churromorales, merged to main 2026-03-17; the branch_10x copy 4a2c3fd0901cbd4a31704b619e0dde15ac65f284, same author and author date, committed 2026-03-17, is the code that 10.5.0 to 10.6.0 ship) [read: git log --all -S 'new ArrayList<>(maxThreshold)' over the full history finds only these two commits, and git blame gives buildSequentialMerges :951-978, including :957, to 1596b7604b6; the line is already in the PR's first revision bd45e5dbce8 (2026-01-26) and the review changed neither it nor the setter's lower-bound-only check]. Original code, not a regression: the class and the line arrived together, and the parent 4a3cdbcd5f9 has no TemporalMergePolicy (2026-03-17 (author and commit date on main 2026-03-17T19:32:33Z; branch_10x copy committed 2026-03-17T21:49:06Z))
First affected version: 10.5.0 [exec] (tag commit f6eaee8148b, 2026-06-19; the first release with TemporalMergePolicy); 10.4.0 and 9.12.3 do not have the class [exec: compile error, cannot find symbol]. CHANGES.txt lists GITHUB#15620 under the 10.4.0 heading, but the 10.4.0 core jar has no TemporalMergePolicy class, so CHANGES.txt cannot date this bug
Affected versions: 10.5.0, 10.5.1, 10.5.2 and 10.6.0 (4ae946e5a9b), plus unreleased main 5adff78bb168 (12.0.0-SNAPSHOT). [exec] fail with the same OutOfMemoryError at TemporalMergePolicy.java:957: main with -ea and -da (finder, the independent reproduction, the intended-behaviour check, adjudication, re-run; the GPT reviewer on main ea57360912ed, which has the same file), 10.6.0 with -ea and -da, and 10.5.2, 10.5.1 and 10.5.0 with -ea. [read] TemporalMergePolicy.java is byte-identical in 10.5.0, 10.5.1, 10.5.2, 10.6.0 and 4a2c3fd0901 (blob b3e4e927d4a); main's copy differs only in an unnamed lambda parameter. Not applicable: 10.4.0 and every earlier release, all of 9.x included (class absent; 10.4.0 and 9.12.3 executed: compile error). Not fixed upstream: main, branch_11x and branch_10_6 still have the line at :957, and no commit after #15620 touches the file (GitHub commits and contents API, 2026-11-03). Not new in 10.6 (10.5.x fails). All runs used JDK 25.0.3 on Linux aarch64
Status: documented (brief, checks and a per-finding doc exist)
Verification: Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by: semi-formal: proof-obligation audit
Upstream and prior reports: none found (2026-11-03, on top of the finder's, the independent reproduction's, the intended-behaviour check's and the adjudication's searches; rechecked the same day): a scripted prior-report search (16 GitHub issue/PR, 3 commit, 2 downstream-repository, 20 mailing-list queries over dev@, java-user@, issues@ and commits@, 4 JIRA-archive queries and 6 web queries; 93 distinct hits, 1 high: the introducing PR), manual GitHub searches on apache/lucene (TemporalMergePolicy, temporal merge, time-series merge policy, maxThreshold, buildSequentialMerges, setMaxThreshold, 'Requested array size exceeds VM limit', forceMergeDeletes OutOfMemoryError) and on Solr, Elasticsearch and OpenSearch, the Solr JIRA, the dev@, java-user@ and issues@ archives, the 17 issue comments, 21 review comments and 14 reviews of #15620, and the touched files of the 196 open apache/lucene PRs updated since 2026-03-17 plus the 13 closed or merged since 2026-11-01: no issue, PR, commit, JIRA entry, list thread or downstream issue reports the batch-list pre-sizing at TemporalMergePolicy.java:957 or an OutOfMemoryError from forceMergeDeletes with a large maxThreshold. The only item on this code is #15620 (the introducing PR by churromorales, merged 2026-03-17 as 1596b7604b6, author date 2026-03-17), whose review asked how the default thresholds were chosen and never discussed an upper bound or the batch capacity. Nearest items, not this bug: #6906 (LUCENE-5844, ArrayUtil.grow and the VM array limit, closed 2014), #9025 (LUCENE-7976, TieredMergePolicy singleton merges) and #9300 (LUCENE-8253, forceMergeDeletes and soft deletes). Re-check the same day: GitHub search finds nothing new (TemporalMergePolicy and maxThreshold: only #15620; buildSequentialMerges, setMaxThreshold and 'temporal merge' updated since 2026-11-01: 0); TemporalMergePolicy.java's only commit on main is still 1596b7604b6, and none of the 11 apache/lucene PRs updated since 2026-11-03 touches it.
Lucene commit tested: main 5adff78bb168 (12.0.0-SNAPSHOT): gradle in a scratch worktree for this doc (the repro fails with assertions on and off and passes with the fix; with the fix TestTemporalMergePolicy (7 tests) and TestTemporalMergePolicyAcrossBuckets (4 tests) pass; the formatter check passes). An earlier, equivalent variant of the test (two 4-document segments, one delete each) fails the same way on the 10.5.0, 10.5.1, 10.5.2 and 10.6.0 jars, and does not compile against 10.4.0 and 9.12.3, which have no such class.

Lucene and upstream sources

Finding record: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:94 (at 5adff78bb168) (@lucene.experimental), :108 (default maxThreshold 8), :234-252 (setMaxThreshold javadoc 'Larger values allow more aggressive merging', 'Must be at least equal to setMinThreshold', lower-bound check at :246-249), :495 and :935 (Math.min bounds), :878, :890 and :961 (loop and equality bounds), :565-590 (findForcedDeletesMerges; windows with two or more eligible segments reach buildSequentialMerges at :582), :951-978 (buildSequentialMerges; new ArrayList<>(maxThreshold) at :957, the fix site)
lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2357-2359 (at 5adff78bb168) (forceMergeDeletes calls findForcedDeletesMerges inside synchronized (this)); lucene/core/src/java/org/apache/lucene/index/LogMergePolicy.java:57 (DEFAULT_MAX_MERGE_DOCS = Integer.MAX_VALUE, Integer.MAX_VALUE as a merge-policy 'no limit')
The finding in 30 seconds: Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:94 (@lucene.experimental), :103 (minThreshold = 4), :108 (maxThreshold = 8), :111 (forceMergeDeletesPctAllowed = 10.0)
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:234-252 (setMaxThreshold javadoc and setter; :246-249 the lower-bound check)
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:528-597 (findForcedDeletesMerges; :555 delete-share test, :568 grouping, :573 'bucketSegments.size() < 2', :582 buildSequentialMerges call)
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:951-978 (buildSequentialMerges; :957 'new ArrayList<>(maxThreshold)', :961, :970)
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:615-622, :675-707 (the temporal field's LongPoint min and max read from the segment files)
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:495, :878, :890, :935 (maxThreshold as a bound); :108, :191, :246, :250, :260, :605 (field, setter checks and store, getter, toString)
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2358-2359 (synchronized; mergePolicy.findForcedDeletesMerges)
Who is affected: Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:94, :108
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2413 (forceMergeDeletes javadoc)
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:815-855 (findMergeCandidates; :833-835 windows with fewer than minThreshold segments are skipped)
Why it looks unintended: Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:234-252
Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/LogMergePolicy.java:57
Lucene main 5adff78bb168: lucene/core/src/test/org/apache/lucene/index/TestTemporalMergePolicy.java:44, :64, :271 (maxThreshold 4, 3, 4); TestTemporalMergePolicyAcrossBuckets.java:46, :140, :194, :268 (maxThreshold 10)
GitHub apache/lucene #6002 (LUCENE-4938, migrated JIRA thread), #15620 (comments and reviews)
Diagrams: Lucene main 5adff78bb168: lucene/core/src/java/org/apache/lucene/index/TemporalMergePolicy.java:246-249, :582, :957

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

TestTemporalMergePolicyMaxThreshold.java (one test, public API): repro/LFV-1213/TestTemporalMergePolicyMaxThreshold.java (50 lines)

About the diagrams

One diagram of the call path in the repro, made separately; its notes are in the diagram directory.

assets/LFV-1213/bug.png (528 KB)

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.