ConfusionMatrixGenerator never stops a classification that exceeds its 5 s limit, so the documents after it time out too

ConfusionMatrixGenerator.getConfusionMatrix evaluates a Classifier by classifying every labelled document of an index on a one-thread executor, waiting at most 5 s for each ("fail if classification takes more than 5s"). On a timeout it adds 5,000 ms to the running time and moves on, but it never cancels or abandons the timed-out task. That task keeps the only worker, so the next documents queue behind it, use up their own 5 s in the queue and are counted as timeouts too, although their own classification takes no time. On main 645d4fe14a20, 3 documents with a 12 s first classification give 1 of 2 instant documents counted and an average classification time of 11,999 ms; a hung first classification gives 0 of 2, an empty matrix and an average of Infinity. There is no error; only a lower getNumberOfEvaluatedDocs() hints at the loss, and when the stuck classification outlives the call, it and the documents queued behind it are still classified afterwards, with the results thrown away. Every release with the class, 6.0.0 to 10.6.0, and main is affected. It is a low-priority robustness and accounting bug in an opt-in evaluation utility: no index, search or classifier answer is affected.

1. One worker. getConfusionMatrix creates Executors.newFixedThreadPool(1, new NamedThreadFactory("confusion-matrix-gen-")) (ConfusionMatrixGenerator.java:71-72); the thread is non-daemon (NamedThreadFactory.java:73).
2. A 5 s wait per document, the Future dropped. For every labelled document: // fail if classification takes more than 5s, then executorService.submit(() -> classifier.assignClass(text)).get(5, TimeUnit.SECONDS), and time += end - start (:100-107).
3. Timeout = skip and continue, on the same executor. catch (TimeoutException _) { time += 5000; } (:140-142). Nothing cancels the task or replaces the executor.
4. So the next task queues. The timed-out task still holds the only worker; the next document's task is queued behind it, and its get(5, SECONDS) counts queue time. If the stuck task needs about 5 s more, that document times out without ever starting, and so on.
5. The overall budget acts only when positive. if (timeoutMilliseconds > 0 && time >= timeoutMilliseconds) break; (:87-89). With no budget every document is visited; with a budget of at most 5,000 ms the loop stops after the first timeout anyway.
6. Result and afterwards. new ConfusionMatrix(counts, time / counter, counter) (:149), so the average is Infinity when timeouts were charged but nothing was counted (NaN if nothing added time at all) and includes queue waits otherwise. finally shutdown() (:150-151) does not wait, so when the stuck classification outlives the call, it and the tasks still queued run afterwards, and their results go nowhere.

At 0 s doc 0 starts and blocks. At 5 s its wait expires (time = 5000) and doc 1 is queued behind it. At 10 s doc 1's wait expires while it is still queued (time = 10000). The call returns 0 evaluated documents, an empty matrix and 10000 / 0 = Infinity.

In the 12 s case b queues at 5 s and times out at 10 s; c is queued at 10 s; at 12 s the slow task ends, b runs (its result is discarded) and c returns inside its own 5 s, so only c is counted, and the running time of about 12,000 ms is divided by 1.

Reach rule (from the code, with no overall budget): while the slow classification runs, each later labelled document with text is submitted when the previous wait times out, so the k-th is submitted about 5k s after the slow one started and times out if the slow one is still running 5 s later. A classification of d seconds therefore loses about floor(d/5) - 1 later documents when d > 10 s and none when d < 10 s (near an exact multiple of 5 s, scheduling decides); verifiers ran it at 7 s (none lost) and 12 s (one lost). One that never returns loses every later labelled document with text.

• Who calls it. ConfusionMatrixGenerator is the classification module's evaluation utility ("Utilities for evaluation, data preparation, etc."). In Lucene only 8 test classes call it; a read-only code search found no use in Solr, Elasticsearch or OpenSearch.
• What it takes. One classification slower than about 10 s, or one that hangs (a slow custom or remote classifier, a blocked lock), and no overall budget (timeoutMilliseconds <= 0, as 13 of the module's 15 test calls use) or one above 5 s. The only timings on record are dev@ CI failure mails of 2017-2018 for a classifier performance test that #9270 later relaxed: on slow CI machines the whole evaluation of a 100-document test index took 61-153 s, about 0.6-1.5 s per document. That is weak evidence about classifiers in general, but nothing near 10 s per document. Likelihood: rare.
• What goes wrong. A smaller evaluation sample, biased by hit order (the documents right after a slow one drop out), with no error; an average classification time that includes queue waits or is Infinity; classification work already counted as timed out keeps running after the call returns.
• What does not. Nothing is written (the index is only read), the counted cells are correct, getNumberOfEvaluatedDocs() reports the lower count (since 6.1.0), and the overall budget still stops the loop when set, as before (it is checked before each document, so a call can run up to one 5 s wait past it, bug or not). With a budget of at most 5,000 ms there is no extra loss.
• Priority. Low: a robustness and accounting bug in an evaluation helper.

The class sits in org.apache.lucene.classification.utils, next to TestConfusionMatrixGenerator, and uses nothing package-private. Smallest input: two documents in a ByteBuffersDirectory with the default codec and StringFields, so the input is fixed (the test framework still picks its usual randomized test settings, such as codec=Asserting(Lucene104) and a random similarity; they do not change the outcome, and the test failed with every seed tried). Doc 0's text stuck makes the test's classifier wait on a latch (this is the slow classification itself, not test synchronization); doc 1's text b is classified at once and correctly. timeoutMilliseconds = -1, so only the per-document 5 s limit acts. The latch is released in finally, so the worker ends and the suite reports no thread leak.

Regression test: this same test with one change for Lucene's build: catch (InterruptedException e) becomes catch (InterruptedException _), because ecjLintTest rejects an unused exception parameter. That form fails on stock 645d4fe14a20 (assertions on and off) and passes with the recommended fix, together with the module's lint checks.

cp $REPRO/TestConfusionMatrixTimedOutClassification.java $LUCENE/lucene/classification/src/test/org/apache/lucene/classification/utils/
cd $LUCENE && ./gradlew :lucene:classification:test --tests org.apache.lucene.classification.utils.TestConfusionMatrixTimedOutClassification -Ptests.seed=DEADBEEF
# about 10 s to fail; assertions off fails the same way: -Ptests.asserts=false (the recorded runs added --offline --no-daemon)

# gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF, assertions on
TestConfusionMatrixTimedOutClassification > testStuckClassificationDoesNotTimeOutNextDocument FAILED
    java.lang.AssertionError: evaluated documents expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:662B61BD54B6EC4A]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.classification.utils.TestConfusionMatrixTimedOutClassification.testStuckClassificationDoesNotTimeOutNextDocument(TestConfusionMatrixTimedOutClassification.java:77)

# a 3-document main() check (not the repro), 645d4fe14a20, -ea, selected fields
control(all instant): elapsedMs=62 matrix={a={a=1}, b={b=1}, c={c=1}} instantCounted=2/2 getNumberOfEvaluatedDocs=3
scenario HUNG:   elapsedMs=15002 matrix={} instantCounted=0/2 getNumberOfEvaluatedDocs=0 getAvgClassificationTime=Infinity callsAfterRelease=[slow, b, c]
scenario SLOW12: elapsedMs=12001 matrix={c={c=1}} instantCounted=1/2 getNumberOfEvaluatedDocs=1 getAvgClassificationTime=11999.0
RESULT bug-reproduced: instant documents counted: HUNG 0/2, SLOW12 1/2

• Code intent. The comment // fail if classification takes more than 5s (ConfusionMatrixGenerator.java:100) and the catch branch that adds 5,000 ms and continues with the next document (:140-142) only make sense if a timeout costs that one document.
• The dropped documents take no time. They time out only because they waited behind another document's task; no broken classifier is needed (12 s and then a normal return is enough).
• The skipped work still runs. In the hung case the calls after release are [slow, b, c]: b and c, already counted as timed out, are classified after the call returned, and the results are discarded.
• Nothing endorses it. No test covers a timed-out classification (the module's tests use fast classifiers and no budget), there is no CHANGES or MIGRATE entry, and the issue that added the timeout (#7538) has only commit notices and one thank-you comment. This shows only that nothing documents the behaviour as intended, not what the author meant.
• Code intent, not contract: the per-document 5 s limit is in the code only; the javadoc documents only the overall timeoutMilliseconds and the number of evaluated documents. The 12 s case shows the effect most clearly.

The strongest case for "intended" (a devil's-advocate verifier and the cross-model review lenses): an experimental, best-effort evaluation utility whose javadoc promises only the overall budget (@param timeoutMilliseconds timeout to wait before stopping creating the confusion matrix) and "the no. of documents evaluated", so a partial matrix is a visible, allowed outcome; Future.get(timeout) only bounds the wait; the single worker serialises calls into a classifier with no thread-safety contract (an effect of the code; nothing records it as a goal); Lucene's NIOFSDirectory and MMapDirectory javadocs warn that Thread#interrupt() or Future#cancel(boolean) during I/O can close the channel, so interrupting the stuck task is risky; and a classifier that hangs is arguably broken.

Why it fails: "fail if classification takes more than 5s" cannot mean that documents taking no time fail; a finite 12 s classification triggers it; and the interrupt concern limits how to fix it, not whether it is a bug (the recommended fix does not interrupt). All verifiers returned "real"; the adjudication confirmed with high confidence.

One fix was validated through gradle in a scratch worktree of 645d4fe14a20. Every fix that keeps the per-document limit either interrupts the stuck classification or lets it overlap with the next ones; the recommended one takes the second route. Two non-overlapping alternatives and a complement were not built.

In the TimeoutException branch, shutdown() the executor whose worker is stuck (it accepts no more work and its thread ends when the stuck classification returns; nothing is interrupted) and continue with a new single-thread executor. Calls without a timeout are unchanged. This stops the executor queue from turning one timeout into several; a classifier that serialises internally (for example, behind a lock the stuck call holds) would still lose the next documents.

} catch (TimeoutException _) {
  // add classification timeout
  time += 5000;
  // the timed-out classification still holds the only worker: leave it to finish on
  // its own (no interrupt, it may be reading the index) and give the next documents a
  // fresh worker, so that they do not wait behind it and time out as well
  executorService.shutdown();
  executorService = newExecutor();
}
...
private static ExecutorService newExecutor() {
  return Executors.newFixedThreadPool(1, new NamedThreadFactory("confusion-matrix-gen-"));
}

Trade-offs

• Overlap. The stuck classification can now run at the same time as the next ones on the same Classifier instance; today the single worker serialises all calls. This is an open API-contract question: the Classifier interface says nothing about thread safety. The reader is safe to share (IndexReader instances "are completely thread safe", IndexReader.java:58-61), but a classifier's own state may not be. LUCENE-6045 (#7107) refactored the classifiers to be shared across threads, but some keep per-call mutable state: KNearestNeighborClassifier sets the boost settings of its shared MoreLikeThis inside each call (KNearestNeighborClassifier.java:160-172; read, no overlap failure was run). A javadoc note or a cap may be wanted.
• Threads. One non-daemon thread per timed-out classification lives until that classification returns; one that never returns keeps its thread, as today; from the code (not run), such a non-daemon thread can keep the JVM from exiting normally, today and with the fix. A cap (stop after a few timeouts) would bound them.
• Accounting unchanged. The 5,000 ms charge is still divided by the evaluated count (average 2,500 ms in the fixed 3-document runs), and the average is still Infinity when timeouts were charged but nothing was evaluated.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31). Stock: the repro fails. With the fix: it passes with assertions on (about 5.2 s instead of about 10 s) and off.
• With the regression-test form: checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest and :lucene:classification:test pass: 44 tests (the module's 12 suites with 43 tests, plus the repro), 1 skipped (Test20NewsgroupsClassification, @Monster), 0 failures.
• The follow-up's 3-document check with the fix shadowed over the 645d4fe14a20 jars: the hung and the 12 s cases both count 2 of 2 and return after 5,002 ms.

A verifier's root-cause sketch: future.cancel(true), shutdownNow() and a new executor on a timeout, shutdownNow() in finally. It also makes the repro pass (6.4 s) and counts 2 documents in 5,080 ms when the first hangs.

Trade-offs

It interrupts a thread that is usually reading the caller's index; the NIOFSDirectory and MMapDirectory javadocs warn that this can close the channel for good (ClosedChannelException afterwards). The verifiers and the adjudication advise against proposing it.

Keep one worker, and measure the 5 s from the moment the task starts (or wait for the busy worker before submitting the next document).

Trade-offs

No overlap and no interrupt, but with no overall budget a classification that never returns would make the whole call wait forever (today it returns after 5 s per document); it needs a bound such as the overall budget.

break (or fail) instead of submitting more work to a busy worker.

Trade-offs

No overlap and no interrupt, and the call returns sooner, but no later document is evaluated, so the repro's expected count is not met; it makes the cut-off explicit only if documented.

Javadoc the hard-coded 5 s per-document limit, and return a defined, documented value from getAvgClassificationTime() when no document was evaluated (today Infinity after timeouts, NaN when nothing added time).

Trade-offs

Accounting only; it does not stop the collateral timeouts.

Method: non-formal (code reading, then a public-API JUnit).

• A hunt unit swept Lucene APIs that accept or create executors against differently behaving executors and tasks. ConfusionMatrixGenerator creates its own executor, so it was not one of the sweep's surfaces; reading its timeout path showed that a timed-out task is never cancelled and keeps the only worker.
• A JUnit with one latch-blocked and two instant documents failed on the stock main jar with assertions on and off, and on 10.5.2 and 9.12.3.
• Then an independent reproducer, a devil's-advocate verifier, cross-model reviews (OpenAI GPT-5.6, executed mode), an adjudication and a credibility audit checked it; a follow-up ran the version history, a prior-report search and a security screen; the repro was minimized to one 2-document test.