AutomatonToTokenStream.toTokenStream silently requires every state to be reachable from state 0, which its javadoc does not say, and rejects an acyclic automaton with an unreachable state with a false 'Cycle found in automaton' / 'Start node has incoming edges, creating cycle' message (documentation and error-message defect; the restriction itself was stated at design time)

Finding LFV-868Area: core / analysis: AutomatonToTokenStream (automaton to token-graph conversion)Priority: lowImpact: wrong errorNovelty: new

AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.

What it shows

You build a finite acyclic automaton with one extra state that nothing leads to, for example 0 -a-> 2 and 1 -b-> 2 with state 2 accepting, or you take TokenStreamToAutomaton's output for the text 'the' after a stop filter removed it, and convert it with AutomatonToTokenStream.toTokenStream, whose javadoc asks only for a finite DAG. You expect the token graph of the part reachable from the start (here the single token 'a'), or at least an error that says what is wrong. Instead it throws IllegalArgumentException 'Cycle found in automaton' ('Start node has incoming edges, creating cycle' when the unreachable state points at state 0), because it counts incoming edges over all states but walks only from state 0; the designers knew such input is unsupported (PR #157), but the javadoc never says so and the message is false.

Expected vs actual

• Expected: A TokenStream with the one token a, or an IllegalArgumentException that does not claim a cycle (ideally one that names the unreachable state).
• Actual: IllegalArgumentException: Cycle found in automaton at AutomatonToTokenStream.java:89, with assertions on and off.

Files

• src/org/apache/lucene/analysis/TestAutomatonToTokenStreamUnreachableState.java (JUnit test)

Run it

• Tested on Lucene main e1faeba08bde (2026-11-02): still reproduces.
• Tested on Lucene main a9779ef3c193 (2026-10-26): reproduced.
• How to read the outcome: a failing test is the finding reproducing (the failure message states expected against actual).
• Seeds: a rerun with the same seed replays the same random choices. Assertions off (-da -dsa -Dtests.asserts=false) is the production-like setting.

# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed 3430DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=3430DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.analysis.TestAutomatonToTokenStreamUnreachableState

Gradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).

Feedback

Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.

Snapshot of 2026-11-04. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.