A Monitor#match that throws leaves ParallelMatcher's worker tasks blocked on the executor forever

ParallelMatcher's constructor submits threads worker tasks to the caller's ExecutorService before any candidate query exists. Each worker loops on queue.take() until it sees an END marker, and END is put only by finish(). Monitor#match creates the matcher, runs the presearcher and the query-index search, and only then calls finish(); when the search throws, its try-with-resources closes the document batch, but CandidateMatcher has no close or abort method, so nothing releases the workers. They stay parked until something interrupts them, each holding a pool thread; on a fixed pool sized to threads, one failed match makes every later ParallelMatcher match on that pool hang until shutdownNow() (an interrupt) or a restart. The headline trigger is legal input: a document whose terms hit more stored-query fields than the clause limit makes the presearcher throw TooManyClauses, which the serial matcher throws too. Two executor-side triggers hit the same missing release: a rejected submit leaks the workers already accepted, and a worker run on the calling thread (CallerRunsPolicy on a saturated pool, or a direct executor) blocks Monitor#match itself. Monitor#debug has the same shape. The default serial matcher and PartitionMatcher are not affected. Every lucene-monitor release since 8.2.0 and main are affected.

• Workers start in the constructor (ParallelMatcher.java:63-74), before matchQuery has queued anything:

for (int i = 0; i < threads; i++) {
  MatcherWorker mw = new MatcherWorker(matcherFactory);
  futures.add(executor.submit(mw));
}

• Each worker blocks until END (ParallelMatcher.java:124-138): while ((task = queue.take()) != END) { ... }.
• Only finish() puts END: doFinish puts one END per worker and waits on each future (ParallelMatcher.java:91-113); CandidateMatcher.finish is its only caller (CandidateMatcher.java:105-106), and CandidateMatcher has no other lifecycle method.
• Monitor#match has no failure path for the matcher (Monitor.java:218-227):

try (DocumentBatch batch = DocumentBatch.of(analyzer, docs)) {
  LeafReader reader = batch.get();
  CandidateMatcher<T> matcher = factory.createMatcher(new IndexSearcher(batch.get()));
  StandardQueryCollector<T> collector = new StandardQueryCollector<>(matcher);
  long buildTime = queryIndex.search(t -> presearcher.buildQuery(reader, t), collector);
  return matcher.finish(buildTime, collector.queryCount);
}

• Where the exception comes from: TermFilteredPresearcher adds one TermInSetQuery clause per document field whose terms appear in the query index (TermFilteredPresearcher.java:224-231), and BooleanQuery.Builder.add throws TooManyClauses at the limit (BooleanQuery.java:82-83). A diagnostic stack on main shows it thrown from TermFilteredPresearcher.buildQuery <- WritableQueryIndex.search <- Monitor.java:224: after createMatcher (:222), before finish (:225). With MultipassTermFilteredPresearcher(4) each pass's builder stays under the limit, and the search's clause-count check in IndexSearcher.rewrite throws IndexSearcher.TooManyNestedClauses (a TooManyClauses subclass) instead, from 257 fields on (256 pass); that is also between createMatcher and finish. Any other exception that escapes the query-index search takes the same path, for example the filter-field batch check (TermFilteredPresearcher.java:183-188) or an I/O error from a persistent query index. An exception from a single candidate query does not: StandardQueryCollector.matchQuery catches it and reports it as a match error (Monitor.java:294-303).
• Monitor#debug (Monitor.java:317-330) has the same create-search-finish shape and leaks the same way (executed for this doc).

• Rejected submit: if the k-th executor.submit throws RejectedExecutionException (k >= 2), it leaves the constructor with k-1 workers accepted and parked. AbortPolicy is ThreadPoolExecutor's default. Executed: a 1-thread SynchronousQueue pool with threads = 2 still runs the accepted worker 10 s after shutdown().
• Caller runs: with CallerRunsPolicy on a saturated pool, or a direct executor, submit runs a worker on the calling thread inside the constructor, where it blocks in take() before any task exists. Executed: a pool sized to threads (2 threads, ArrayBlockingQueue(2)) shared by two in-progress matches (built through the public MatcherFactory.createMatcher as stand-ins, not two concurrent Monitor#match calls); a third Monitor#match was still blocked 10 s after the pool went idle. An interrupt ends it with RuntimeException("Interrupted during match").

The serial matcher (same TooManyClauses, no executor) and PartitionMatcher, which takes the same ExecutorService parameter but submits only in doFinish via invokeAll (PartitionMatcher.java:70-102): after the same failure its pool terminates, and a caller-run task simply runs.

• Who. Code that matches with ParallelMatcher.factory(...) from lucene-monitor; the package javadoc offers it "to increase performance in low-concurrency systems" (package-info.java:50-51). The default serial matcher and PartitionMatcher are not affected.
• Effect. Every Monitor#match or Monitor#debug whose query-index search throws strands the matcher's threads worker tasks until an interrupt; each one that runs holds a pool thread. On a fixed pool of threads threads, the sizing the upstream tests use (a pool of 10 with threads = 10, ConcurrentMatcherTestBase.java:40, :51-52), one failure makes every later ParallelMatcher match on that pool hang, even after the offending document is gone. The caller gets an exception that gives no hint that pool threads are lost; only an interrupt (shutdownNow()) or a restart recovers.
• Pool shape. On a fixed pool with more threads than threads, later matches keep working until failed matches have taken every thread; on a cached pool parked threads pile up instead (neither executed). A later match on a used-up pool waits in doFinish's future.get() (ParallelMatcher.java:99, executed) or, once its 1,024-slot queue is full, in matchQuery's put (:49, :80, read). The 2-argument factory uses availableProcessors() workers (ParallelMatcher.java:170-184), so one failure can strand that many.
• Scale. Ordinary: one document and one pool thread in the repro. The headline trigger needs stored queries over more fields than the clause limit (1,024 by default, about 1,024 / passes with MultipassTermFilteredPresearcher) and a document with matching terms in all of them; an I/O error from a persistent query index or a mixed filter-field batch takes the same path. The executor triggers need a rejecting pool or a caller-runs or direct executor.
• Not executed: an unbounded cached pool (expected: a growing number of parked threads rather than a hang), a pool shared by several Monitors, an end-to-end service scenario.
• Workaround until fixed. The serial matcher and PartitionMatcher are not affected. An application that keeps ParallelMatcher can wrap its factory, keep the matcher it creates and call its public finish(0, 0) after a failed match; that frees the worker (executed, awaitTermination(5 s) true). Nothing documents this, and finish() also runs any candidates already queued.
• Likelihood: rare. Each trigger is uncommon among ParallelMatcher users, no downstream user was found and nothing has been reported since 2019; once it happens on a fixed pool, the hang is permanent.

The class sits in org.apache.lucene.monitor next to Lucene's monitor tests, but uses nothing package-private. Smallest shape: one pool thread, threads = 1, 1,025 one-term queries, one document; deterministic, no threads besides the pool. The finally block's shutdownNow() frees the parked worker, so the suite reports no thread leak. The test takes about 6-7 s, 5 of them the bounded wait; the whole gradle command took 20-62 s here.

Regression test: this same test, with one change for Lucene's build: create the pool as Executors.newFixedThreadPool(1, new NamedThreadFactory("matchers")) (import org.apache.lucene.util.NamedThreadFactory), because Lucene's forbidden-apis check rejects Executors#newFixedThreadPool(int) in test code. That form fails on stock main with the same message and passes with the fix and the format, ECJ and forbidden-apis checks.

cp $REPRO/TestParallelMatcherFailedMatch.java $LUCENE/lucene/monitor/src/test/org/apache/lucene/monitor/
cd $LUCENE && ./gradlew :lucene:monitor:test --tests org.apache.lucene.monitor.TestParallelMatcherFailedMatch -Ptests.seed=DEADBEEF
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF (verbatim, log lines 94-99)
TestParallelMatcherFailedMatch > testFailedMatchReleasesWorkers FAILED
    java.lang.AssertionError: expected the pool to terminate after the failed match, but its worker is still running
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:5AE480DC4E23BD8C]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.monitor.TestParallelMatcherFailedMatch.testFailedMatchReleasesWorkers(TestParallelMatcherFailedMatch.java:49)
# (line 49 is the awaitTermination assert; the expectThrows(TooManyClauses) before it passed)

# diagnostic run on main 645d4fe14a20, assertions off (a separate observation test, not the repro); selected lines: the Monitor lambda, Monitor.match(:242), test-method, JDK park and executor frames left out
DIAG parallel: match threw org.apache.lucene.search.IndexSearcher$TooManyClauses
DIAG parallel:   thrown via org.apache.lucene.monitor.TermFilteredPresearcher$2.build(TermFilteredPresearcher.java:227)
DIAG parallel:   thrown via org.apache.lucene.monitor.TermFilteredPresearcher.buildQuery(TermFilteredPresearcher.java:132)
DIAG parallel:   thrown via org.apache.lucene.monitor.WritableQueryIndex.search(WritableQueryIndex.java:222)
DIAG parallel:   thrown via org.apache.lucene.monitor.Monitor.match(Monitor.java:224)
DIAG parallel: awaitTermination(5s) after shutdown = false
DIAG parallel: diag-pool-thread state=WAITING
DIAG parallel:   at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
DIAG parallel:   at org.apache.lucene.monitor.ParallelMatcher$MatcherWorker.call(ParallelMatcher.java:127)
DIAG parallel: terminated after shutdownNow = true
DIAG partition: awaitTermination(5s) after shutdown = true

# standalone runner, seed DEADBEEF: main 645d4fe14a20 fails with -ea and -da; 10.5.2 (-ea, -da) and 10.6.0 fail the same way

• No documented executor requirement. The javadocs say only "Matcher class that runs matching queries in parallel", "an ExecutorService to use for parallel execution" and "the number of threads to execute on" (ParallelMatcher.java:34-46, :55-62, :157-184); nothing about rejection, caller-thread execution, sizing or cleanup after a failed match. Lucene states such limits where it has them: "The provided Executor must not reject tasks" (BPIndexReorderer.java:834).
• Lucene's own code drops the matcher. Monitor#match closes its DocumentBatch on failure (Monitor.java:220) but has nothing to call on the matcher. In the normal call, monitor.match(doc, ParallelMatcher.factory(...)), the caller holds no reference to it; releasing the workers needs an undocumented workaround (wrap the public MatcherFactory, keep the matcher, call its public finish() after the failure; executed).
• The siblings clean up. The serial matcher throws the same TooManyClauses and uses no executor; PartitionMatcher, with the same ExecutorService parameter, throws it and its pool terminates (executed on main and on 8.2.0).
• The JDK and core Lucene allow caller-runs and rejection. The Executor javadoc lets an executor run a task "immediately in the caller's thread"; ThreadPoolExecutor's default handler is AbortPolicy. Core's TaskExecutor runs a rejected task on the calling thread "to ensure a rejecting executor only reduces parallelism and does not result in failure" (TaskExecutor.java:55-67).
• Tests. No upstream test makes a ParallelMatcher match fail. The upstream concurrent test runs threads = 10 on a pool of 4 (ConcurrentMatcherTestBase.java:62, :76-77), so "the pool must have threads free threads" is not an upstream assumption.

The strongest case for "intended" (built by the devil's-advocate verifier; every verifier rated it weak): the class javadoc says the matcher "distributes queries amongst its worker threads using a BlockingQueue" and threads is "the number of threads to execute on" (ParallelMatcher.java:40-42, :61). So the caller must supply an executor that really runs threads long-lived consumer tasks asynchronously; a caller-runs, direct or saturated rejecting pool is a provisioning error, the leaks happen only on exceptional paths where the caller does get an exception, and shutdownNow() frees a dedicated pool.

Why it fails: no text states that precondition, and the upstream test itself runs 10 workers on 4 threads. It does not touch the headline at all: the failed-match leak happens on a correctly sized Executors.newFixedThreadPool with the default policy, because Monitor#match creates the matcher and lets the exception escape without releasing it, and shutdownNow() is not an option for the shared pool the factory invites. The caller-runs part is a separate, more defensible question (which executors are allowed): there a javadoc constraint is a reasonable maintainer answer, which is why the claim leads with the failed-match leak.

One fix was built and validated through gradle in a scratch worktree of 645d4fe14a20; it covers the failed-match and rejected-submit triggers and leaves the caller-runs hang. The other options are design sketches, not built.

Add a no-op protected void doAbort() to CandidateMatcher, next to doFinish(). Monitor#match and Monitor#debug call it when the query-index search throws, then rethrow. (A throw from finish() itself needs no hook for ParallelMatcher: doFinish puts every END before it waits.) ParallelMatcher overrides it to call future.cancel(true) on each worker; its constructor does the same for the workers already accepted when a submit throws (for example RejectedExecutionException), and now creates collectorMatcher before the loop.

// CandidateMatcher: new hook, no-op by default
protected void doAbort() {}

// Monitor#match (the same in Monitor#debug)
long buildTime;
try {
  buildTime = queryIndex.search(t -> presearcher.buildQuery(reader, t), collector);
} catch (Throwable e) {
  matcher.doAbort();
  throw e;
}
return matcher.finish(buildTime, collector.queryCount);

// ParallelMatcher: doAbort() and a failed submit call future.cancel(true) for each worker

Trade-offs

• A running worker is interrupted out of take(), a queued one never starts, and the original exception propagates (ParallelMatcher's doAbort() does not throw; an outside override that throws would replace it, so a PR may add such a failure as suppressed). Successful matches are unchanged; candidate tasks already queued are dropped, which is fine because the match has failed. Cancelled worker tasks still waiting in a bounded executor queue stay there, as no-ops, until a thread takes them.
• It adds a protected method to a public abstract class. Maintainers may prefer another name, a public abort() (a wrapping CandidateMatcher in another package cannot forward a protected call) or making CandidateMatcher Closeable.
• cancel(true) interrupts the pool threads that run this matcher's workers, as shutdownNow() does today for the whole pool.
• It does not fix the caller-runs hang (the worker blocks inside the constructor, before any failure). The study's TLA+ model predicted this: releasing the workers on failure clears the failed-match and rejection deadlocks, not the caller-runs one.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31). Stock: the repro fails, and so does a probe that calls Monitor#debug instead. With the fix: both pass with assertions on and off; :lucene:monitor:test runs 183 tests (the module's own 181, the repro and the probe), 1 skipped (@Nightly), 0 failures; checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass.
• The study's other trigger tests on the fixed build: failed match, next match after a failed match, rejected submit, multipass presearcher and filter-field batch now pass; both caller-runs tests still fail, as expected.

Record the constructing thread and let a worker that starts on it return at once; doFinish drains what is left in the queue on the calling thread, and matchQuery runs a task inline when no worker can take it (otherwise its put blocks once the 1,024-slot queue is full). Or submit workers lazily and let the caller help drain the queue.

Trade-offs

Fixes the remaining caller-runs and direct-executor hang, but adds more code to the matching path than the release fix and changes how work is split between threads. Not built or run.

A note on ParallelMatcher.factory that the executor must run the threads workers asynchronously (no caller-runs or direct executor) and that a rejected submit fails the match.

Trade-offs

A plausible maintainer answer for the caller-runs part; alone it leaves the failed-match and rejection leaks in place.

No API change: on failure Monitor#match calls matcher.finish(...), which puts END and joins the workers.

Trade-offs

Runs (for PartitionMatcher, starts) every candidate query already collected for a match that has failed, delays the exception until they finish, and can itself throw. Not built.

Method: non-formal first signal (a differential sweep of the real main jar), then a TLA+ model.

• Sweep. 13 API surfaces that accept an executor, times 9 executor behaviours; oracles: the same result as the executor-less call, a return within 8 s, and an executor that terminates after shutdown(). The ParallelMatcher cells: caller-runs = hang, rejection at the 2nd or 3rd submit = pool never terminates (first run 2026-10-26).
• Model. ParallelMatcherLifecycle.tla (TLC, deadlock checking on) encodes a failure between createMatcher and finish on purpose; TLC found the failed-match deadlock (12 distinct states) as well as the caller-runs and abort deadlocks, none in the two capacity controls. A fix mutant ("one END per accepted worker on failure") clears the failed-match and abort deadlocks but not the caller-runs one. The verifiers did not re-check the model.
• Confirmation. A public-API JUnit on stock builds, then an independent reproducer, a devil's-advocate verifier, cross-model reviews (OpenAI GPT-5.6) that ran their own tests, an adjudication and a credibility audit; a follow-up ran the version history, a prior-report search and a security screen.