A failed IndexRearranger.execute() never shuts down its thread pool, so its threads keep the JVM alive

IndexRearranger.execute() creates its own fixed pool, Executors.newFixedThreadPool(min(availableProcessors, #selectors), new NamedThreadFactory("rearranger")), for every call, and calls executor.shutdown() only as the last statement of the method, not in a finally. Any exception after the first task was submitted skips that line: a selector that throws, an addIndexes error, and since 10.0.0 also the class's own overlapping-selector IllegalStateException or a failing deletes selector. execute() still throws the right exception, but the pool's threads are non-daemon (NamedThreadFactory) and a fixed pool's threads "will exist until it is explicitly shutdown" (JDK javadoc), and the caller has no reference to the pool. So each such failed call leaves min(availableProcessors, #selectors) idle threads alive until the JVM ends, and a command-line program whose rearrange failed does not exit on its own after the exception leaves main.

The one-test repro fails on main (assertions on and off) and passes with a try/finally fix that also keeps the misc module's tests and :lucene:misc:check green. An equivalent one-test check fails on every release that was run, from 8.9.0 to 10.6.0; the releases in between have identical source or bytecode. It is minor: an experimental offline tool, a run that already failed loudly, the input index not changed.

• Segment selector (IndexRearranger.DocumentSelector): user code, BitSet getFilteredDocs(CodecReader reader) throws IOException (IndexRearranger.java:283).
• Fixed thread pool: Executors.newFixedThreadPool(n, factory) is a ThreadPoolExecutor(n, n, 0L, ...). Its idle threads never time out; the JDK javadoc: "The threads in the pool will exist until it is explicitly shutdown."
• Non-daemon thread: the JVM does not exit while one is alive. NamedThreadFactory.newThread calls t.setDaemon(false) (NamedThreadFactory.java:73) and names threads <prefix>-<pool number>-thread-<n>: the pool number comes from a static, so JVM-wide, counter (:30, used with NAME_PATTERN in the constructor, :33, :41-48) and <n> from the factory's own counter (:70-71), hence rearranger-2-thread-1.

1. execute() (:113-137) creates the pool as a local variable (:114-117).
2. createRearrangedIndex (:180-200) submits one task per selector (:194); each runs addOneSegment (:202-210), whose filtered reader's constructor calls selector.getFilteredDocs(in) (:256), then writer.addIndexes(readers) (:209).
3. It waits with for (Future<Void> future : futures) { future.get(); } (:197-199). A failed task makes get() throw ExecutionException, which leaves execute().
4. executor.shutdown() is the last statement (:136), outside any finally, so it is skipped. Nothing else references the pool: no executor parameter, no field, no close().

The finder's exit-path harness ran these five cases on the stock jar (the three failure cases leak) and on a try/finally shadow (none leaks). Any other exception after the first submit skips :136 the same way (read, not run): an interrupted caller in either future.get() loop (:198, :231), an I/O error in finalizeRearrange (:146-173) or in opening or closing the second phase (:132-135), and tryDeleteDocument's IllegalStateException (:241). The fix's finally covers them too.

• Who is hit. Callers of IndexRearranger (@lucene.experimental, IndexRearranger.java:66) whose execute() fails after it submitted at least one task. In 8.9.0-9.12.3 only an exception out of the future.get() loop leaks (a failing segment selector or addIndexes, or an interrupted caller); from 10.0.0 (#11815) also the overlapping-selector IllegalStateException and a failing deletes selector. 8.9.0-9.0.0 have neither the ordering check nor a delete phase, and 9.1.0-9.12.3 call shutdown() before the ordering check (read from source).
• Triggers are ordinary. getFilteredDocs is declared throws IOException; addIndexes can fail on I/O; the overlap check is the class's own validation.
• Default vs opt-in. The tool is opt-in (a misc module, mainly for building benchmark indexes). Inside it nothing needs to be configured: every exception that leaves execute() after the first task was submitted skips shutdown().
• Scale. min(availableProcessors, #selectors) idle threads per such failed call, until the JVM ends. The repro shows one; Lucene's CI showed 10 after a failed TestIndexRearranger task (2022).
• Visible effects. A batch JVM does not exit on its own after main throws (executed with one selector: still alive at 20 s, 0.52 s with the fix); a long-lived process that retries collects threads; tests get ThreadLeakError.
• What does not happen. execute() throws the right exception, the input index is only read (what a failed run leaves in the output directory was not examined), a successful run is unaffected, and assertions on or off make no difference.
• Likelihood: rare. It needs a run that already fails; the searches found no user report since 8.9.0 (2021). Priority Low.

The selector records Thread.currentThread() before it throws, so the test checks the exact pool worker and cannot pass vacuously (if the selector never ran, worker.get() is null and the test errors). @ThreadLeakScope(NONE) only keeps the framework's suite-level leak check from replacing the test's own assertion; without it the stock run also fails with ThreadLeakError. The test checks only the exception type; that its cause is the selector's IOException shows in the plain main() run below and is asserted by an independent verifier's longer test (e.getCause().getMessage() is "selector failed").

Plain main(), same scenario: prints Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: selector failed and is still running at a 20 s timeout; the thread dump shows rearranger-2-thread-1 WAITING (parking) in ThreadPoolExecutor.getTask and DestroyJavaVM waiting. With the fix this one-selector run exits in 0.52 s.

As the regression test: the same test. For :lucene:misc:check, rename the unused lambda parameter reader to _ (ecj lint) and wrap the comment line at 100 columns (google-java-format); that copy fails on the stock code (assertions on), passes with the fix (assertions on and off) and passes the check. With the fix, the @ThreadLeakScope(NONE) line and its import can also go. The version shown keeps reader, because _ needs Java 22+. As shown, it uses only APIs that exist from 9.1.0 on (an equivalent test ran on 9.1.0-10.6.0); 8.9.0-8.11.4 and 9.0.0 have no org.apache.lucene.tests.util.LuceneTestCase and need import org.apache.lucene.util.LuceneTestCase;, and List.of needs Java 9+.

cp TestIndexRearrangerFailureShutsDownPool.java $LUCENE/lucene/misc/src/test/org/apache/lucene/misc/index/
cd $LUCENE && ./gradlew :lucene:misc:test --tests org.apache.lucene.misc.index.TestIndexRearrangerFailureShutsDownPool -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

# gradle, main e1faeba08bde, seed DEADBEEF, assertions off (the same with assertions on)
TestIndexRearrangerFailureShutsDownPool > testFailedExecuteStopsWorker FAILED
    java.lang.AssertionError: rearranger-2-thread-1 still alive
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:79E4B6940A9732BB]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.assertTrue(Assert.java:42)
        at junit@4.13.2/org.junit.Assert.assertFalse(Assert.java:65)
        at org.apache.lucene.misc.index.TestIndexRearrangerFailureShutsDownPool.testFailedExecuteStopsWorker(TestIndexRearrangerFailureShutsDownPool.java:49)
1 test completed, 1 failed
# the test took 10.3 s (the full join); with the fix: 1 test, 0 failures, 0.28 s

# the same test without @ThreadLeakScope(NONE), stock, assertions on
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.lucene.misc.index.TestIndexRearrangerFailureShutsDownPool:
   1) Thread[id=21, name=rearranger-2-thread-1, state=WAITING, group=TGRP-TestIndexRearrangerFailureShutsDownPool]
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:

• Ownership. execute() creates the pool (IndexRearranger.java:114-117) and calls shutdown() on success (:136). There is no executor parameter, field or close(), so only execute() can release it.
• In-tree code does it right. CheckIndex.checkIndex(List<String>) creates Executors.newFixedThreadPool(threadCount, new NamedThreadFactory("async-check-index")) and shuts it down in a finally (CheckIndex.java:588-616). The benchmark module's JoinIndices makes its pool daemon: "daemon, since Benchmark never closes PerfRunData, and the pool must not keep the JVM alive" (JoinIndices.java:90).
• Precedent. Issue #14971 (Dawid Weiss, 2025): "ConcurrentMergeScheduler is used and it creates an instance of CachedExecutor but - upon failure - the thread pool is never shut down (LuceneTestCase does have a grace period of 20 seconds, so it's indeed a leak)." Fixed by #15675: "GITHUB#14971: Fix CachedExecutor thread pool leak in ConcurrentMergeScheduler when IndexWriter initialization fails." (lucene/CHANGES.txt:362-363).
• The test framework calls it a leak. Without @ThreadLeakScope(NONE) the repro fails with ThreadLeakError: 1 thread leaked from SUITE scope ... rearranger-2-thread-1, state=WAITING and "There are still zombie threads that couldn't be terminated"; the 2022 CI mails show the same for TestIndexRearranger.
• Legal trigger, nothing documented. getFilteredDocs is declared throws IOException (:283). execute() has no javadoc, the class javadoc says nothing about threads or failures, and the existing tests call execute() only on successful runs. No review comment on lucene-solr#2246 or #11815 discusses failure cleanup.

The strongest case for "works as designed": IndexRearranger is an @lucene.experimental offline tool, execute() promises nothing on failure, the run has already failed loudly, and a command-line caller can call System.exit.

Why it fails: @lucene.experimental is about API stability, not resource leaks; the caller cannot reach the pool; a JVM that does not exit after main throws is exactly what the in-tree JoinIndices comment avoids; and maintainers treated the same failure-path pool leak as a bug in #14971. These points lower the severity (Low), not the fact that it is a bug. There is no explicit contract.

Option A was run through gradle in a scratch worktree of e1faeba08bde and is recommended. B and C are alternatives a maintainer may prefer; neither was run as written here. No open PR changes the executor lines (#14996 and #16847 touch the file elsewhere).

Wrap both phases of execute() (everything after the pool creation) in try { ... } finally { executor.shutdown(); }. The finder's and an independent verifier's versions of this fix are equivalent.

--- a/lucene/misc/src/java/org/apache/lucene/misc/index/IndexRearranger.java
+++ b/lucene/misc/src/java/org/apache/lucene/misc/index/IndexRearranger.java
@@ -123,17 +123,20 @@
     createSegmentsConfig.setMergePolicy(NoMergePolicy.INSTANCE);
     applyDeletesConfig.setMergePolicy(NoMergePolicy.INSTANCE);
 
-    try (IndexWriter writer = new IndexWriter(output, createSegmentsConfig);
-        IndexReader reader = DirectoryReader.open(input)) {
-      createRearrangedIndex(writer, reader, segmentSelectors, executor);
-    }
-    finalizeRearrange(output, segmentSelectors);
+    try {
+      try (IndexWriter writer = new IndexWriter(output, createSegmentsConfig);
+          IndexReader reader = DirectoryReader.open(input)) {
+        createRearrangedIndex(writer, reader, segmentSelectors, executor);
+      }
+      finalizeRearrange(output, segmentSelectors);
 
-    try (IndexWriter writer = new IndexWriter(output, applyDeletesConfig);
-        IndexReader reader = DirectoryReader.open(writer)) {
-      applyDeletes(writer, reader, deletedDocsSelector, executor);
+      try (IndexWriter writer = new IndexWriter(output, applyDeletesConfig);
+          IndexReader reader = DirectoryReader.open(writer)) {
+        applyDeletes(writer, reader, deletedDocsSelector, executor);
+      }
+    } finally {
+      executor.shutdown();
     }
-    executor.shutdown();
   }
 
   /**

Trade-offs

• shutdown() neither waits nor interrupts running tasks (JDK: "previously submitted tasks are executed ... This method does not wait"; it interrupts only idle workers, which is how they exit), so tasks still running or queued when another task fails run to the end, and only then do the threads exit; a task that blocks keeps its thread, and the JVM, alive until it returns. That they run on a reader and writer being closed is the separate LFV-1011; this fix neither fixes nor worsens it.
• Error path only; the success path already called shutdown(). No API or format change.

What the checks validated

• Gradle, JDK 25, scratch worktree of e1faeba08bde (removed afterwards): the repro fails on the stock build with assertions on and off (10.3 s) and passes with the fix with assertions on and off (0.28 s).
• With the fix: TestIndexRearranger, TestBinaryDocValueSelector and the repro with -Ptests.iters=5 and -XX:ActiveProcessorCount=4: 30 tests, 0 failures; the whole :lucene:misc:test: 24 classes, 317 tests, 0 failures, 9 skipped.
• With the fix, the committable copy of the test passes :lucene:misc:check -x test; a copy without @ThreadLeakScope(NONE) passes the framework's leak check (on stock it fails with ThreadLeakError).
• Earlier: the finder's and the verifier's tests and the exit-path harness (all 5 exit cases) pass with the fix shadowed; a cross-model reviewer's run kept TestIndexRearranger green.

Also drops queued tasks and interrupts running ones, so a failed run stops sooner.

Trade-offs

Interrupting a thread blocked on I/O is risky in Lucene: NIOFSDirectory's javadoc warns it "can close the underlying file descriptor immediately" (NIOFSDirectory.java:41-46). Not tried.

Wait for every task before rethrowing (LFV-1011's fix) and shut the pool down in a finally. Or give each phase its own pool as the last resource of its try-with-resources, so that ExecutorService.close() (JDK 19+; main requires Java 25) shuts down and waits before the reader and writer close.

Trade-offs

The combined diff (+35/-13) was validated for LFV-1011 on 645d4fe14a20, whose IndexRearranger.java is the same file; the close() variant was not tried, and a 9.x backport would need explicit shutdown() and awaitTermination().

Make the pool's threads daemon, as the benchmark module's JoinIndices does.

Trade-offs

Lets the JVM exit, but a long-lived process still collects idle threads per failed call.

Method: non-formal (code reading during an executor sweep, 2026-10-26). The sweep passed unusual executors into the Lucene components that accept one; IndexRearranger creates its own, so none could be injected, and the hunting AI tool read its failure paths instead and saw executor.shutdown() as the last statement, outside any finally.

• A JUnit that counts live rearranger- threads after a failed execute() failed 2 of 3 (failing selector, overlapping selectors; the success control passed). A cross-model review (an independent reviewer from another model family, OpenAI GPT-5.6, refute lens, executed) reproduced both paths.
• A later hunt unit (2026-11-02) re-ran the lead on main e1faeba08bde, ran five exit cases of execute() through an exit-path harness on the stock jar and on a try/finally shadow, and reduced the repro to one test.