TieredMergePolicy accepts segmentsPerTier = NaN, and the next flush or commit never returns

TieredMergePolicy.setSegmentsPerTier rejects values below 2.0 with "segmentsPerTier must be >= 2.0", but its only check is if (v < 2.0), which is false for NaN, so NaN is stored. The next findMerges call then never returns: mergeFactor = (int) segsPerTier is 0, the level size of the allowed-segment-count loop drops to 0, and neither exit test of its while (true) can become true (with any maxMergedSegmentMB above 0, such as the default 5 GB). findMerges runs inside the synchronized IndexWriter.updatePendingMerges, so the spinning thread holds the writer's monitor: commit, close, rollback, getDocStats, hasPendingMerges and other indexing threads block behind it, and neither Thread.interrupt() nor setting a valid value on the live policy ends it. One document and one flush(), commit() or DirectoryReader.open(writer) are enough, with the default IndexWriterConfig, assertions on or off. Every release since TieredMergePolicy appeared (3.2.0, 2011) through 10.6.0, and main, hangs. The class's four other double setters accept NaN through the same kind of check; with forceMergeDeletesPctAllowed = NaN, forceMergeDeletes() merges nothing.

TieredMergePolicy.java:219-225 on main 645d4fe14a20:

public TieredMergePolicy setSegmentsPerTier(double v) {
  if (v < 2.0) {
    throw new IllegalArgumentException("segmentsPerTier must be >= 2.0 (got " + v + ")");
  }
  segsPerTier = v;
  return this;
}

NaN < 2.0 is false, so NaN is stored.

TieredMergePolicy.java:400-413 (annotated: the end-of-line comments are added and the comment line :401 is left out):

final int mergeFactor = (int) segsPerTier;                      // (int) NaN == 0
long levelSize = Math.max(minSegmentBytes, floorSegmentBytes);  // 16 MB by default
long bytesLeft = totIndexBytes;
while (true) {
  final double segCountLevel = bytesLeft / (double) levelSize;
  if (segCountLevel < segsPerTier || levelSize == maxMergedSegmentBytes) {
    allowedSegCount += Math.ceil(segCountLevel);
    break;
  }
  allowedSegCount += segsPerTier;
  bytesLeft -= segsPerTier * levelSize;                          // (long) NaN == 0
  levelSize = Math.min(maxMergedSegmentBytes, levelSize * mergeFactor);  // 16 MB * 0 == 0
}

1. One flushed document makes a segment of about 1 KB (994 bytes in one probe run). Pass 1: levelSize = 16 MB, segCountLevel = 994 / 16,777,216, about 0.00006. With the default 8.0 the loop breaks here, because 0.00006 < 8.
2. With NaN, 0.00006 < NaN is false and 16 MB is not the maximum merged segment size (5 GB), so there is no break. bytesLeft becomes (long) NaN = 0 and levelSize becomes min(5 GB, 16 MB * 0) = 0.
3. Pass 2 and every later pass: segCountLevel = 0 / 0.0 = NaN, NaN < NaN is false and 0 is not 5 GB. The state never changes, so the loop never ends.

flush() calls maybeMerge(FULL_FLUSH) (IndexWriter.java:4378-4379), which calls the private synchronized updatePendingMerges (:2442-2446, :2454), which calls findMerges (:2508). commit() and DirectoryReader.open(writer) reach it through findFullFlushMerges (:2496), whose default calls findMerges (MergePolicy.java:693). The findMerges javadoc says the call "is always synchronized on the IndexWriter instance" (MergePolicy.java:602-605). By reading, an automatic flush during addDocument takes the same path (IndexWriter.java:6353-6359); not executed.

The loop has no blocking call, so Thread.interrupt() has no effect. Its state lives in locals (mergeFactor is a final local; levelSize and bytesLeft are already 0), so calling setSegmentsPerTier(8.0) on the live policy does not help either. Both were executed on main, 10.6.0, 10.5.2 and 9.12.3.

+Infinity is accepted too but harmless: segCountLevel < Infinity breaks on pass 1. The second exit, levelSize == maxMergedSegmentBytes, would fire only with maxMergedSegmentMB = 0.

setMaxMergedSegmentMB (v < 0.0, :103), setDeletesPctAllowed (v <= 0 || v > 50, :139), setFloorSegmentMB (v <= 0.0, :169) and setForceMergeDeletesPctAllowed (v < 0.0 || v > 100.0, :196) accept NaN the same way. With forceMergeDeletesPctAllowed = NaN, pctDeletes > forceMergeDeletesPctAllowed (:947) is never true, so forceMergeDeletes() merges nothing (the study left 6 deleted documents in place, against 0 with the default 10%). maxMergedSegmentMB = NaN is stored as 0 bytes and behaves like the accepted value 0, which is a separate bug.

• Who. Applications that set segmentsPerTier on the default TieredMergePolicy from a value that can be NaN: a configuration string parsed with Double.parseDouble (which accepts "NaN") or a computed ratio such as 0.0 / 0.0.
• Effect. The first flush, commit or NRT reopen after the setting takes effect never returns and burns one core. The spinning thread holds the IndexWriter monitor, so commit, close, rollback, getDocStats and hasPendingMerges from other threads stay BLOCKED with it as lock owner, and a concurrent indexing thread stops when it publishes its next flushed segment (in one probe, with a 1 MB RAM buffer, it added 1,580 documents in the first second, then none for the remaining 7 s of the probe, BLOCKED in publishFlushedSegment with the spinner as lock owner). There is no exception and no log line; only a JVM restart recovers.
• What it does not do. Nothing committed is lost or corrupted: the last commit stays intact and the pending commit simply never happens, so documents added since the last commit are gone after the restart.
• Scale. Tiny: one document, the default IndexWriterConfig, any directory, assertions on or off.
• Likelihood: rare. Normal code passes constants, Lucene's own randomized tests draw values from 2 to 50, and nobody has reported it since 3.2.0 (2011). Whoever does feed a NaN gets a certain, unrecoverable hang instead of an error.

The class sits in org.apache.lucene.index next to TestTieredMergePolicy, but uses nothing package-private. Default IndexWriterConfig, one empty document, no threads, and the test draws no random values (LuceneTestCase still picks its usual randomized defaults, such as the codec). The bound is randomizedtesting's @Timeout(millis = 10_000), the idiom upstream uses to bound a test that could hang.

Because the spinning thread cannot be interrupted, a failing run does not end at the timeout: the runner also reports a suite-level thread leak (the test thread still RUNNABLE in findMerges), a zombie-thread error and a sysout-limit error from the leak dumps, and lingers for the thread, so a failing run takes about 35 s. All of these come from the one loop.

Regression test: this same test, with one change for Lucene's build: write the catch as catch (IllegalArgumentException _), because the core module's ECJ lint rejects an unused exception parameter. That form fails on stock main with the same timeout and passes with the fix and the format, ECJ and forbidden-apis checks.

cp $REPRO/TestTieredMergePolicySegmentsPerTierNaN.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestTieredMergePolicySegmentsPerTierNaN -Ptests.seed=DEADBEEF
# about 35 s for a failing run; assertions off fails the same way: -Ptests.asserts=false

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on (stacks trimmed)
TestTieredMergePolicySegmentsPerTierNaN > testFlushWithSegmentsPerTierNaN FAILED
    java.lang.Exception: Test timeout exceeded (>= 10000 msec).
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:4A4471D852F73166]:0)
TestTieredMergePolicySegmentsPerTierNaN > executionError FAILED
    org.gradle.internal.exceptions.DefaultMultiCauseException: Multiple Failures (3 failures)
    	java.lang.AssertionError: The test or suite printed 20980 bytes to stdout and stderr, even though the limit was set to 8192 bytes. ...
    	com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestTieredMergePolicySegmentsPerTierNaN:
       1) Thread[id=21, name=TEST-TestTieredMergePolicySegmentsPerTierNaN.testFlushWithSegmentsPerTierNaN-seed#[DEADBEEF]-worker, state=RUNNABLE, ...]
            at org.apache.lucene.index.TieredMergePolicy.findMerges(TieredMergePolicy.java:413)
            at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2508)
            at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2445)
            at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4379)
            at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4357)
            at org.apache.lucene.index.TestTieredMergePolicySegmentsPerTierNaN.testFlushWithSegmentsPerTierNaN(TestTieredMergePolicySegmentsPerTierNaN.java:39)
    	com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated: ...
2 tests completed, 2 failed

# plain main() probe, main 645d4fe14a20, assertions off (not the repro)
setSegmentsPerTier(NaN) accepted; getSegmentsPerTier() = NaN
flush: did NOT return within 5000 ms; state=RUNNABLE; thread CPU 999 ms in a 1000 ms window
  second thread calling hasPendingMerges(): state=BLOCKED, ... lock owner=probe-flush (the flush thread)
  after interrupt(): flush thread alive=true, state=RUNNABLE
  after setSegmentsPerTier(8.0) on the live policy: flush thread alive=true, state=RUNNABLE
commit: did NOT return within 5000 ms; state=RUNNABLE; thread CPU 1000 ms in a 1000 ms window
nrt-open: did NOT return within 5000 ms; state=RUNNABLE; thread CPU 1000 ms in a 1000 ms window
RESULT: 3 of 3 writer calls did not return (segmentsPerTier=NaN)
# the same probe with 2.0: flush, commit and NRT open returned in 43, 6 and 35 ms

• The setter enforces a domain. The message "segmentsPerTier must be >= 2.0 (got ...)" (TieredMergePolicy.java:221) states the legal range; NaN is outside it. With the one-line fix, the same call throws "segmentsPerTier must be >= 2.0 (got NaN)".
• Nothing makes NaN intended. The javadoc (:213-218) gives only the meaning and the default 8.0. TestTieredMergePolicy.testSetters (:759-793) accepts +Infinity and rejects -2.0 but never tries NaN; git log -G isNaN over the merge-policy sources finds no commit; no issue discusses it.
• Lucene's practice. LUCENE-6982 (#8039): "Rather than strange behavior, we should throw an exception if these values are wrong"; its commit added NaN checks for BM25's k1 and b (BM25Similarity.java:53-60 today). On #14412 jpountz asked to keep LRUQueryCache "rejecting skipCacheFactor=Float.NaN" (review comment r2019386671, 2025-03-28); the check reads if (skipCacheFactor >= 1 == false) { // NaN >= 1 evaluates false (LRUQueryCache.java:178).
• A hang, not a trade-off. The result is not a different merge choice but a call that never returns while holding the writer lock. Lucene's SECURITY.md (line 46) says issues outside its security scope "are still valid bug reports and hardening suggestions" and names "a change that prevents an endless loop" as a welcome example (framed there around broken indexes); this fix touches no hot path.

The strongest case for "not a bug" (made by the cross-model refute review, OpenAI GPT-5.6, read-only, and by the devil's-advocate verifier): garbage in. NaN is not a segment count; the javadoc gives no range and no @throws; the class is @lucene.experimental; and core has other ordered-only checks that admit NaN (LiveIndexWriterConfig.setRAMBufferSizeMB, CompoundFormat.setMaxCFSSegmentSizeMB) or no check at all (the LogByteSizeMergePolicy MB setters).

Why it fails: the setter already validates its argument and names the range, so a value outside that range that gets through is the defect; the reviewer itself agreed with the root cause and called the guard a reasonable hardening change. @lucene.experimental is about API stability, not permission to hang. The other NaN-blind setters show inconsistency, not intent, and none of them was shown to hang anything. Verdict: a defect, most likely treated upstream as a small validation fix.

One fix was built and validated through gradle in a scratch worktree of 645d4fe14a20; the one-line variant was validated earlier with a standalone runner. The other options are sketches, not built.

Add Double.isNaN(v) || to the five double setter checks of TieredMergePolicy (+5/-5 lines). The setSegmentsPerTier line is the one that removes the hang; the other four close the same gap in setMaxMergedSegmentMB, setDeletesPctAllowed, setFloorSegmentMB and setForceMergeDeletesPctAllowed.

public TieredMergePolicy setSegmentsPerTier(double v) {
  if (Double.isNaN(v) || v < 2.0) {
    throw new IllegalArgumentException("segmentsPerTier must be >= 2.0 (got " + v + ")");
  }
  segsPerTier = v;
  return this;
}

// the same `Double.isNaN(v) ||` prefix in setMaxMergedSegmentMB, setDeletesPctAllowed,
// setFloorSegmentMB and setForceMergeDeletesPctAllowed

Trade-offs

• Changes behaviour only for NaN, which is outside every one of these setters' stated ranges. Where its effect was run it never worked: segmentsPerTier hung the writer and forceMergeDeletesPctAllowed made forceMergeDeletes() a no-op; by reading, the two MB setters store it as 0 bytes (a value setFloorSegmentMB itself rejects). The effect of deletesPctAllowed = NaN was not measured. Code that passes NaN now fails at configuration time with the setter's existing message ("... (got NaN)").
• +Infinity stays accepted; testSetters relies on that for the MB setters, and it is harmless for segmentsPerTier.
• No change on any hot path: the check runs once per setter call.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, OpenJDK 25.0.3, seed DEADBEEF. Stock: the repro times out (10.07 s test, 35.1 s suite).
• With the fix: the repro passes (the setter now throws, so the test returns early), and TestTieredMergePolicy runs 17 tests, 1 skipped (@Nightly), 0 failures, with assertions on and off.
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest of :lucene:core pass with the regression-test form of the repro (catch (IllegalArgumentException _)); that form fails on stock main with the same timeout.

if (Double.isNaN(v) || v < 2.0) in setSegmentsPerTier alone.

Trade-offs

The smallest change that removes the hang; it leaves the four sibling setters accepting NaN (and forceMergeDeletes() a no-op with forceMergeDeletesPctAllowed = NaN).

What the checks validated

With the standalone runner on 645d4fe14a20: the minimal test passes, and a plain main() probe prints "setSegmentsPerTier(NaN) rejected: segmentsPerTier must be >= 2.0 (got NaN)".

Invert each condition so NaN fails it, as LRUQueryCache does: if (v >= 2.0 == false).

Trade-offs

Same effect as the recommended option; a matter of style. Not built separately.

Defensive: break out of the allowed-segment-count loop when levelSize stops growing (for example when it is 0 or mergeFactor < 2).

Trade-offs

Protects against future bad states, but on its own it would hide a broken configuration behind odd merge decisions instead of an error. Not built.

Method: semi-formal (a bytecode inventory and an abstract path exploration, then dynamic probes).

• Inventory. A static pass with the JDK 25 class-file API over 27 core and non-sandbox module jars listed every public or protected constructor, factory, setter and method of a public class whose bytecode reaches an IllegalArgumentException guard (1,231 of them), with the compared parameters and constants.
• Prediction. For every float or double parameter, an abstract exploration of the guard's paths with the parameter fixed to NaN, +Infinity and -Infinity predicted accept or reject; for the five TieredMergePolicy double setters it predicted ACCEPT for NaN.
• Confirmation. A dynamic probe on the real jar confirmed the acceptance; a merge-policy sweep (272 configurations) saw no completion within 15 s for segmentsPerTier = NaN, with the worker in findMerges:413; a public-API JUnit then witnessed the hang. An independent reproducer, a devil's-advocate verifier, a cross-model review (OpenAI GPT-5.6), an adjudication, a follow-up (history, prior reports, security), a credibility audit, a re-verification on current main and a tie-break followed.