Resetting an open writer's ConcurrentMergeScheduler to AUTO_DETECT makes the next merge stall forever

ConcurrentMergeScheduler.setMaxMergesAndThreads accepts the pair (AUTO_DETECT_MERGES_AND_THREADS, AUTO_DETECT_MERGES_AND_THREADS), documented as the "Dynamic default ... based on CPU core count", and stores it as -1/-1. Lucene resolves AUTO on its own only in initialize(), which IndexWriter calls once from its constructor (an explicit setDefaultMaxMergesAndThreads(false) call also sets concrete values). Called on the scheduler of an open writer, the reset leaves both limits at -1. The next thread other than a merge thread that hands pending merges to the scheduler's merge() (forceMerge, commit or close, all three executed) enters maybeStall, whose condition "running merge threads >= maxMergeCount" reads 0 >= -1 and never becomes false, so the thread waits in 250 ms naps forever: no merge thread starts and nothing is thrown, with assertions on and off; the only trace is a verbose InfoStream line. Setting concrete limits again releases it. From 5.0.0 to 8.5.2 merge() resolved AUTO on every call; LUCENE-9345 (8.6.0) moved that call into initialize(). Every release run since, through 10.6.0 and main, hangs.

1. The setter stores -1/-1. setMaxMergesAndThreads (:132-158) takes the AUTO/AUTO pair with the comment // OK and sets both fields to -1 (:133-137). It rejects mixed pairs and values below 1, and its javadoc (:122-131) has no "before the writer is opened" note.
2. Only initialize() resolves AUTO by itself. initDynamicDefaults (:442-453) calls setDefaultMaxMergesAndThreads(false) (maxThreadCount = max(1, cores/2), maxMergeCount = maxThreadCount + 5, :190-191) when maxThreadCount is AUTO. Its only caller is initialize() (:533-539), which in production code only IndexWriter calls, once, from its constructor (IndexWriter.java:960; a unit test also calls it directly, TestConcurrentMergeScheduler.java:471). The public setDefaultMaxMergesAndThreads(false) also sets concrete values, but only when the application calls it.
3. merge() stalls before it starts anything. merge() (:542) loops if (maybeStall(mergeSource) == false) break; (:566) and only then takes the next merge and starts its thread.
4. The stall check never passes. maybeStall loops while (mergeSource.hasPendingMerges() && mergeThreadCount() >= maxMergeCount) (:611). With maxMergeCount = -1 that holds while any merge is pending, so it calls doStall(), a wait(250) (:647-654), again and again. A merge thread is never stalled (maybeStall returns false for it, :623-627). In the repro, where no merge runs, the merge the thread waits for is the one it was about to start itself.

• forceMerge(1): Object.wait <- doStall <- maybeStall <- merge <- IndexWriter.executeMerge <- maybeMerge <- forceMerge; after 3 s hasPendingMerges() is true and 0 merge threads run; setMaxMergesAndThreads(6, 1) released it after 66 ms and the merge left 1 segment (this doc's probe).
• commit(): in the post-commit maybeMerge (the finder's repro, LogDocMergePolicy), or on the merge-on-commit path in prepareCommitInternal, before the new commit point is written (the independent reproducer: the commit generation stays 1).
• close(): through IndexWriter.flush (the finder's probe). By reading (not run), an indexing thread whose flush triggers a merge reaches the same call.

• Setting concrete limits again ends the loop (executed: 66 ms).
• rollback() from another thread freed the stalled forceMerge after 7 ms, which then threw AlreadyClosedException (executed, the fact check's probe). By reading it is not a general way out: on the merge-on-commit path the stalled thread holds commitLock (IndexWriter.java:3775, :3892), which rollbackInternal needs first (:2589), and during a stalled close() a rollback() waits for the close to finish (:2581, :1387).

• Who. Code that changes ConcurrentMergeScheduler limits on a live writer (a normal, tested use: see "Why it is a bug") and resets them with the documented constant, for example to map an "auto" setting back to the defaults.
• Default vs opt-in. It needs this Expert setter call on the scheduler of an open writer. The default configuration never calls it; AUTO set before the writer opens is resolved by initialize().
• Effect. A permanent hang of the next thread (other than a merge thread) that hands pending merges to the scheduler: forceMerge, commit, close, or (by reading) an indexing thread whose flush triggers a merge. No exception; the only trace is the verbose InfoStream line "too many merges; stalling..." (ConcurrentMergeScheduler.java:633).
• Scale. None: two one-document segments.
• Downstream (read on default branches, not run). Solr calls the setter on a scheduler it has just created, before the writer is built, so initialize() resolves AUTO; Elasticsearch calls it on a live scheduler, but its settings parse with a minimum of 1, so it never passes -1. Hence likelihood "rare" (a rating from these readings, not a measured prevalence).

TestCMSLiveResetToAutoDetect (one method, extends LuceneTestCase, package org.apache.lucene.index of the lucene/core tests, next to TestConcurrentMergeScheduler). It uses a ByteBuffersDirectory and a default IndexWriterConfig; the second thread only bounds the hanging call, and the finally sets concrete limits so the test can clean up after a failure. It carries the Apache license header and passes checkGoogleJavaFormat, and it is the regression test unchanged. The outcome does not depend on the seed. catch (TimeoutException _) needs Java 22+, fine on main; a copy for branch_10x (Java 21) needs a named variable.

cp $REPRO/TestCMSLiveResetToAutoDetect.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestCMSLiveResetToAutoDetect -Ptests.seed=DEADBEEF

# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF -Ptests.asserts=false
TestCMSLiveResetToAutoDetect > testResetToAutoDetectOnOpenWriter FAILED
    java.lang.AssertionError: forceMerge(1) hangs after resetting a live ConcurrentMergeScheduler to AUTO_DETECT
        at org.apache.lucene.index.TestCMSLiveResetToAutoDetect.testResetToAutoDetectOnOpenWriter(TestCMSLiveResetToAutoDetect.java:51)

# gradle, same, assertions on: the same failure
# with the fix: PASS with assertions on and off

# supporting probe (the repro's steps plus prints), main, -da; stack frames abridged
PROBE after open maxMergeCount/maxThreadCount=7/2        (the test framework randomizes the core count)
PROBE after reset maxMergeCount/maxThreadCount=-1/-1
PROBE after 3 s: forceMerge running, hasPendingMerges=true mergeThreadCount=0 limits=-1/-1
PROBE   at org.apache.lucene.index.ConcurrentMergeScheduler.doStall(ConcurrentMergeScheduler.java:650)
PROBE   at org.apache.lucene.index.ConcurrentMergeScheduler.maybeStall(ConcurrentMergeScheduler.java:636)
PROBE   at org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:566)
PROBE   at org.apache.lucene.index.IndexWriter.executeMerge(IndexWriter.java:2451)
PROBE   at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2446)
PROBE   at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2281)
PROBE released by setMaxMergesAndThreads(6, 1) after 66 ms; segments=1

• Contract. AUTO_DETECT_MERGES_AND_THREADS is the "Dynamic default for maxThreadCount and maxMergeCount, based on CPU core count" (ConcurrentMergeScheduler.java:61-66). The setter takes the pair at any time (// OK, :135). Its @param maxMergeCount says the incoming thread "will block until a merge thread has completed" when "we already have this many threads running" (:125-128); here no merge thread exists to complete.
• Tests. testLiveMaxMergeCount changes the limits through w.getConfig().getMergeScheduler() (TestConcurrentMergeScheduler.java:666), testChangeMaxMergeCountyWhileForceMerge changes them during a forceMerge (:995), and the test framework's maybeChangeLiveIndexWriterConfig calls the setter at random on live writers (LuceneTestCaseParent.java:1946-1960, from RandomIndexWriter), but only with concrete limits (1 to 4 threads, :1950-1951), never AUTO, so randomized tests cannot hit this. testResetToAutoDefault (:873-896) resets to AUTO but only checks the getters of a scheduler no writer uses.
• History. When the constant was added (#7013, LUCENE-5951), Chris Hostetter noted in review that the AUTO reset branch was a no-op, and Michael McCandless replied "I'll add a test that exposes this then fix it" (the origin of testResetToAutoDefault). From 5.0.0 to 8.5.2 merge() began with initDynamicDefaults(writer). LUCENE-9345 (#10385, d7e0b906abc) moved the call into the new initialize() when merge() started to receive a MergeSource, which has no Directory (the diff turns initDynamicDefaults(IndexWriter) into initDynamicDefaults(Directory), needed then for the SSD probe, unused since LUCENE-9576 in 9.0.0); its PR, review, commit message and CHANGES entry do not mention AUTO or say why the call moved.

The strongest case for "not a bug": AUTO is an initial sentinel that initialize() resolves once; the setter is Expert and MergeScheduler is @lucene.experimental, so callers should choose AUTO before opening the writer, or call setDefaultMaxMergesAndThreads(false) to restore defaults.

Why it fails: nothing documents that precondition, the setter accepts the pair without complaint, the reset worked from 5.0.0 to 8.5.2 and was asked for in review, and setDefaultMaxMergesAndThreads(false) is not documented as the way to restore defaults. Even if maintainers decide the reset is unsupported, a silent forever-hang is not a defensible outcome: the setter would have to reject the pair. The argument bears on likelihood, hence priority Low. A credibility audit also rated a silent forever-hang on a legal public call worth a one-line fix.

Option A was built and checked through gradle on 5adff78bb168; B and C were not built. All three keep the minimal test as the regression test (C would turn it into an expectThrows). Open PR #16889 edits the same merge()/maybeStall code for another bug.

Restore the pre-8.6 behaviour: merge() calls initDynamicDefaults before anything else. It changes nothing unless the limits are AUTO, and since LUCENE-9576 (9.0.0) it no longer uses its Directory argument, so null is safe.

  public synchronized void merge(MergeSource mergeSource, MergeTrigger trigger) throws IOException {

    // setMaxMergesAndThreads(AUTO_DETECT, AUTO_DETECT) may be called after initialize(): resolve it
    // here too, otherwise maybeStall compares against -1 and stalls forever
    initDynamicDefaults(null);

Trade-offs

One field check per merge() call, already under the scheduler's lock. Between the reset and the next merge() call the getters still return -1, and the other readers of the two fields (updateMergeThreads, the I/O throttle, the intra-merge executor; ConcurrentMergeScheduler.java:372, :840, :971) see -1 until then (by reading; not exercised by the repro). That matters when a merge is already running at the reset: updateMergeThreads computes doPause = threadIdx < bigMergeCount - maxThreadCount (:372), which with -1 sets the rate of at least the largest running merge to 0 (stopped), and setForceMergeMBPerSec, enableAutoIOThrottle and disableAutoIOThrottle call it (:201, :217, :227; by reading, not run). So fix A is validated for the reproduced case (no merge running at the reset); a reset during a running merge needs its own test, and fix B closes that window.

What the checks validated

• Gradle in a scratch worktree at 5adff78bb168, seed DEADBEEF: the repro fails without the fix (assertions on and off) and passes with it (on and off).
• With the fix: the repro, TestConcurrentMergeScheduler (21), TestIndexWriterMerging (13), TestIndexWriterForceMerge (5, 1 skipped by the suite) and TestMergeSchedulerExternal (2): 42 tests, 0 failures. :lucene:core:checkGoogleJavaFormat passes.

In setMaxMergesAndThreads, when the pair is AUTO and initialize() has already run, call setDefaultMaxMergesAndThreads(false) right away.

Trade-offs

Every reader sees concrete values at once. Needs an "initialized" flag (or a check of state that initialize() sets); resolving unconditionally would break testResetToAutoDefault, which expects the getters to return AUTO on an unused scheduler.

Throw IllegalStateException for the AUTO pair once the scheduler is initialized, and document that AUTO is an initial value only.

Trade-offs

Loud instead of silent, but it takes away a reset that worked from 5.0.0 to 8.5.2. testResetToAutoDefault (an unused scheduler) is unaffected; the regression test would expect the exception.

Method: non-formal (code reading of ConcurrentMergeScheduler, witnessed by an executable repro).

• The finder's code reading found that initDynamicDefaults, the only code that resolves AUTO, runs only from initialize(), which IndexWriter calls once (the call moved there from merge() in 8.6.0); the setter accepts AUTO at any time, and maybeStall's check cannot pass with -1.
• The finder then wrote a public-API JUnit test (commit path) that fails on main with assertions on and off and on 10.6.0 and 9.12.3, and a stack probe for the commit and close paths.
• An independent reproducer, two executed cross-model reviews, a devil's-advocate verifier and an adjudication reproduced it with their own tests; a final repro check reduced it to the one-test forceMerge repro shown here ("Prior reports, history and checks").