IndexSearcher#count can hang forever when Weight#count throws on a segment split into intra-segment partitions

With an executor and the opt-in intra-segment partitions, IndexSearcher#count uses a partition-aware collector in TotalHitCountCollectorManager. The first partition of a segment to arrive registers an incomplete CompletableFuture and calls Weight#count; the other partitions of that segment wait on it with no timeout. The future is completed only on success or CollectionTerminatedException, so when Weight#count throws anything else (a stock ExitableDirectoryReader timeout or an IOException, both run), the other partitions wait forever, TaskExecutor waits for them, and count never returns. Introduced by #13542 in 10.0.0; every release to 10.5.2, branch_10_6 and main are affected. Each hanging call pins its threads for good.

• Intra-segment partitions (10.0.0, #13542): a segment can be split into doc-id ranges that are searched as separate slices on separate threads. Off by default. You turn it on by overriding IndexSearcher#slices(List) to call the public static slices(leaves, maxDocsPerSlice, maxSegmentsPerSlice, true), which splits any segment with more than maxDocsPerSlice documents into at most 5 partitions of maxDoc / n documents, the last one also taking the remainder.
• Weight#count(ctx): the per-segment shortcut that count tries first, for example TermQuery's docFreq. It is declared to throw IOException.
• The count barrier: with partitions, TotalHitCountCollectorManager hands out a LeafPartitionAwareTotalHitCountCollector. Per segment, the first partition decides "count from statistics" or "collect", and the others follow, so that a whole-segment count is never added twice and the query cache cannot answer some partitions with a whole-segment count.

1. IndexSearcher#count builds new TotalHitCountCollectorManager(getSlices()) (IndexSearcher.java:525), makes one task per slice and calls TaskExecutor#invokeAll (:778-788), which forks all tasks but one and runs tasks on the calling thread too (TaskExecutor.java:91-114).
2. Each task calls collector.setWeight(weight) (IndexSearcher.java:811); its first call for a partition is then collector.getLeafCollector(ctx) (:836). The first partition of the segment puts a new, incomplete CompletableFuture F into the shared map with putIfAbsent (TotalHitCountCollectorManager.java:110-115) and calls Weight#count through super.getLeafCollector (TotalHitCountCollector.java:48).
3. F is completed only on success (complete(false), :120) or on CollectionTerminatedException (complete(true), :122-124). There is no other catch and no finally. Every other partition of the segment waits in earlyTerminated.get() with no timeout (:131).
4. When Weight#count throws anything else, F stays incomplete. The failing task's setException cancels only tasks that have not started; cancel does nothing to a started task, by design (TaskExecutor.java:175-198). collectResults then waits with an untimed future.get() (:118-123).
5. So the caller parks in collectResults while a pool thread parks at line 131, or, when the caller ran the waiting partition itself, the caller parks at line 131 after the producing thread has already finished. Nothing can complete F any more: the hang is permanent.

The stock trigger: count uses a no-scores weight, so TermQuery looks its term up lazily inside Weight#count (TermQuery.java:257). On an ExitableDirectoryReader that opens an ExitableTermsEnum, whose constructor checks the timeout (and the thread's interrupt flag) and throws ExitingReaderException (ExitableDirectoryReader.java:950-952, :976-999). IndexSearcher#setTimeout does not reach this path: it wraps only the bulk scorer, after getLeafCollector (IndexSearcher.java:842-858). The stock trigger needs a segment without deletions (with deletions TermWeight#count returns -1 without opening the terms enum, TermQuery.java:255-266, and the timeout then throws normally later), the timeout detected inside the first partition's Weight#count (here: a budget already spent), no query cache answering the count first (off by default on main), and the other partition's task already started.

The same collector serves search(q, TotalHitCountCollectorManager), MultiCollectorManager wrappers and FacetsCollectorManager.search(searcher, q, 0, fcm) (its n == 0 branch).

• Who is exposed. Applications that pass an Executor to IndexSearcher and override slices(List) with slices(..., true), and that call IndexSearcher#count, search(q, TotalHitCountCollectorManager), a MultiCollectorManager containing one, or FacetsCollectorManager.search(searcher, q, 0, fcm).
• What triggers it. Any exception from the first-arriving partition's Weight#count: a time budget enforced through ExitableDirectoryReader (or an interrupt it detects), an IOException or a custom Weight (these run), or, by reading, an AlreadyClosedException when a reader is closed under a running search (close to misuse: SearcherManager's acquire/release keeps the reader open while a search runs).
• Not exposed. Searchers without an executor (one whole-segment slice), default slicing, and a sequential Runnable::run executor (the second task is cancelled before it starts). Downstream, read only: Elasticsearch builds whole-segment slices; OpenSearch has intra-segment partitioning but counts with its own manager, and no path into Lucene's class was established.
• How likely. It hangs whenever the other partition's task has started before the first one throws. With no test hooks: 124 and 176 of 200 calls in the study (50-doc segment, 5 partitions, 4 threads), 121 of 200 at ea57360912ed, 11-15 of 16 in the tie-break, 44 of 50 on 10.5.2 and 48 of 50 on 10.0.0. These rates hold for the recorded 1- and 4-thread pools on one host; other pool sizes were not measured.
• Damage. The call never returns, with no exception and no log line. The calling thread and every thread running a waiting partition of the segment (at most 4 per split segment) stay parked until each is interrupted; usually that is the caller plus a pool thread, but in 1 of 12 tie-break hangs on 10.0.0 and 1 of 27 hangs in the fact check's run on main the caller ran the waiting partition itself and only it was pinned. IndexSearcher's javadoc advises against shutdownNow on the search executor with NIOFSDirectory. With a fixed pool the leak stops once every worker is parked (later searches then run on their calling threads, TaskExecutor.java:102-114); how far a service degrades depends on its sizing (not measured).
• Scale. None needed: 10 documents in the test, 2,000 in the hook-free check.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => junit.framework.AssertionFailedError: Unexpected exception type, expected ExecutionException but got java.util.concurrent.TimeoutException
       org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1437)
       org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
       org.apache.lucene.search.TestIndexSearcherCountPartitionHang.testCountRethrowsTimeoutFromPartitionedSegment(TestIndexSearcherCountPartitionHang.java:77)
     Caused by: java.util.concurrent.TimeoutException
       java.base/java.util.concurrent.FutureTask.get(FutureTask.java:206)
       org.apache.lucene.search.TestIndexSearcherCountPartitionHang.lambda$testCountRethrowsTimeoutFromPartitionedSegment$2(TestIndexSearcherCountPartitionHang.java:77)
       org.apache.lucene.tests.util.LuceneTestCaseParent._expectThrows(LuceneTestCaseParent.java:1587)
       org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1427)
       [...]

• TaskExecutor#invokeAll (TaskExecutor.java:71-75): "If an exception is thrown by more than one callable, the subsequent ones will be added as suppressed exceptions to the first one that was caught. Additionally, if one task throws an exception, all other tasks from the same group are cancelled".
• Weight#count (Weight.java:196): "@throws IOException if there is a low-level I/O error".
• ExitableDirectoryReader (ExitableDirectoryReader.java:32-37): "If QueryTimeout#shouldExit() returns true, an ExitingReaderException is thrown."
• CollectorManager (CollectorManager.java:35-39): several leaf collectors may be requested for the same leaf, and "Any computation or logic that needs to happen once per segment requires specific handling in the collector manager implementation".
• Dead catch. TotalHitCountCollectorManager.java:138-139 catches ExecutionException from get(); that can only happen if someone calls completeExceptionally, and no code path does.
• Documented feature. The slices javadoc and the 10.0.0 CHANGES entry ("GITHUB#13542: Add initial support for intra-segment concurrency") present partitions as supported; the only documented limitation is performance.
• Tests and history. No upstream test throws from Weight#count with partitions (TestTotalHitCountCollector has one test; TestExitableDirectoryReader searches without an executor). The #13542 threads discuss double counting, the query cache and clearing the map, never a failing first partition. 9.x and the parent of #13542 propagate the exception across concurrent slices (2 of 2 pass).

Strongest case for "intended" (built by the cross-model reviewer and the tie-break adjudication): partitions are opt-in and much of their path is not stable API (the executor constructors, LeafSlice, LeafReaderContextPartition, Collector, CollectorManager and TaskExecutor are @lucene.experimental, slices is Expert, and the test's ConstantScoreWeight is @lucene.internal); TaskExecutor deliberately waits for started tasks so that no task outlives invokeAll; SECURITY.md line 28 makes bounding untrusted input the application's job.

Why it fails: @lucene.experimental marks API instability, not permission to hang, and count and the manager are not tagged. Waiting for started tasks is safe only if they can finish; here the manager's own coordination future is never completed once its producer throws, which is the manager's defect, not the executor's. The whole-segment path and the dead ExecutionException catch show the intended outcome. The reviewer's "not real" was procedural, for a read-only session that could run nothing; on substance it agreed with the root cause and rated the intended-behaviour argument weak. Verdict: a defect.

Two options were built and validated in a scratch worktree at ea57360912ed (removed afterwards): A, the minimal fix, recommended for this issue, and C, which also fixes the related reuse bug LFV-053 and surfaced the original exception at top level in every recorded trial. B and D are untested.

In LeafPartitionAwareTotalHitCountCollector#getLeafCollector, after the CollectionTerminatedException catch, catch any Throwable, call firstEarlyTerminated.completeExceptionally(t) and rethrow. Every waiter's get() then returns, every started task finishes, and invokeAll rethrows. +5 lines, TotalHitCountCollectorManager.java only; no Java 22 syntax, so it backports to 10.x as is.

} catch (CollectionTerminatedException e) {
  firstEarlyTerminated.complete(true);
  throw e;
} catch (Throwable t) {
  // e.g. Weight#count threw: release the other partitions of this leaf, which wait on
  // this future, so that the search fails with t instead of hanging
  firstEarlyTerminated.completeExceptionally(t);
  throw t;
}

Trade-offs

• Wrapped exception, sometimes. A waiter rethrows new RuntimeException(ExecutionException(cause)) (the existing lines 138-139). When the waiter's task comes first in the result list, the caller sees that wrapper, with the original as its cause and as a suppressed exception: 2 of 200 IOException calls in this doc's run, 7 of 400 in the tie-break, 8 of 200 in the study. Code that catches ExitingReaderException misses those.
• Reuse is not fixed (LFV-053): a failed search's entries stay in the map because reduce() never runs, so a reused manager fails with a spurious wrapped exception instead of hanging. IndexSearcher#count and FacetsCollectorManager build a new manager per call, so for them A ends the hang and the reuse problem does not arise (the wrapped-exception caveat still applies).
• No cost on the success path; no benchmark was run.

What the checks validated

• Stock: the test fails 2 of 3 through gradle, with assertions on and with -Ptests.asserts=false.
• Patched, gradle, assertions on and off: the new test, the hook-free test, TestTotalHitCountCollector, TestIndexSearcher, TestTaskExecutor, TestExitableDirectoryReader, TestMultiCollectorManager, TestMultiCollector, TestCollectorManager, TestSearchWithThreads, TestTermQuery, TestBooleanQuery, TestLRUQueryCache, TestTotalHits: 14 classes, 134 tests, 0 failures, 5 skipped. Most use newSearcher, which picks partitions at random, so they can exercise the changed collector's success path (no coverage was measured).
• Patched, facet suites that call FacetsCollectorManager.search with n == 0: 6 classes, 67 tests, 0 failures.
• :lucene:core:check -x test passes with the fix and the test (google-java-format, ecjLint, forbiddenApis).
• Standalone harness: the test 3 of 3 (-ea, -da); the hook-free test 0 hangs in 200 calls per test; the study's 7-test class 6 of 7, the failing one being LFV-053's reuse test.

The fix the 2026-10-22 study model-checked: key the map by the search's Weight first (Map<Weight, Map<Object, Future<Boolean>>>, chosen in setWeight, which IndexSearcher calls before any leaf), complete the future exceptionally as in A, and let a waiter whose first partition failed throw CollectionTerminatedException, so in the tested paths the search fails with the original exception. +23/-6 lines, same file. Also fixes LFV-053.

private final Map<Weight, Map<Object, Future<Boolean>>> earlyTerminatedMap = new ConcurrentHashMap<>();

@Override
public void setWeight(Weight weight) {   // in LeafPartitionAwareTotalHitCountCollector
  super.setWeight(weight);
  earlyTerminatedMap = perWeight.computeIfAbsent(weight, _ -> new ConcurrentHashMap<>());
}
// first partition: catch (Throwable t) { firstEarlyTerminated.completeExceptionally(t); throw t; }
// waiter: catch (ExecutionException _) { throw new CollectionTerminatedException(); }

Trade-offs

• Larger, and changes the map's lifetime (LFV-053's fix): a reused manager keeps a failed search's per-Weight map until the next successful reduce() clears the whole map.
• The skipping waiter relies on the first partition's exception failing the search, which holds in IndexSearcher; an IndexSearcher subclass whose searchLeaf swallowed it would undercount instead of failing.
• getLeafCollector without a prior setWeight would hit a null map (IndexSearcher always calls setWeight, IndexSearcher.java:811). Uses _ (Java 22+): a 10.x backport needs named variables. The map is keyed by the Weight object, not by a per-search token, and one manager shared by concurrent searches was not tested; TaskExecutor still reports whichever task exception comes first in its list.

What the checks validated

• The study's TLC model of this protocol passes every count-barrier configuration, with throwing counts and manager reuse.
• Gradle: the same 14 core classes, 134 tests, 0 failures, assertions on and off; facets 67 tests, 0 failures; :lucene:core:check -x test passes (a first try failed ecjLint on an unused catch variable, now _).
• Harness: the test 3 of 3 (-ea, -da); the hook-free test 0 hangs in 200 calls per test, the original exception at top level in 200 of 200 per partitioned trigger; the study's 7-test class 7 of 7, reuse test included.

Removes the wrapped exception with a 2-line change to the waiter.

Trade-offs

On a reused manager, a stale, exceptionally completed entry would make every partition of that segment skip silently, turning LFV-053's hang into a silent undercount. Only safe with C's scoping. Untested.

Make the "count from statistics or collect" decision without a blocking barrier, for example before the tasks fork, or deterministically in each partition.

Trade-offs

Removes the class of bug, but the barrier exists so that the query cache cannot answer some partitions with a whole-segment count, and getLeafCollector is the first point where leaf, weight and collector meet. A TaskExecutor-level change such as the stale PR #12756 (stop running siblings when one throws) would not release a thread parked in CompletableFuture.get().

Method: formal (TLA+ model checking with TLC).

• The 2026-10-22 formal-verification study wrote four design-level TLA+ models of Lucene's search-side concurrency, each with a file:line mapping table to the Java code. CountBarrier.tla models TotalHitCountCollectorManager with intra-segment partitions on top of TaskExecutor: the shared map, registration, the count outcomes (count, no count, throw), the blocked waiter, submission, the caller claiming tasks, cancellation of unstarted tasks and result collection.
• With throwing counts allowed, TLC reported a deadlock in a 6-state trace (one segment, two partitions, one worker): the worker's partition registers the future, its count throws, the caller's partition waits forever. Larger configurations (3 slices, 2 workers; 3 segments, 2 searches) deadlock the same way. The same model without the throw is deadlock-free and returns the right count.
• The trace was translated into a deterministic JUnit test on the stock jars; the study's independent verifier wrote its own 7-test reproduction (4 fail, both interleavings, the stock timeout variant, the reuse variant) and classified it as a new bug.
• On 2026-10-26 the hunt re-ran it on ea57360912ed, an independent reviewer from another model family (OpenAI GPT-5.6) tried to refute it in a read-only session, a tie-break judge wrote a hook-free test, and the history, prior-report and security checks followed.