Loading some well-formed hyphenation grammars (e.g. the current TeX Afrikaans patterns) throws ArrayIndexOutOfBoundsException, because TernaryTree.insert(char[], int, char) sizes its node arrays with strlen(key) instead of strlen(key, start)

Finding LFV-894Area: analysis-common / compound hyphenation (TernaryTree.compact via HyphenationTree.loadPatterns and HyphenationCompoundWordTokenFilter)Priority: lowImpact: crashNovelty: new

Finding page for LFV-894

AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.

What it shows

You load a well-formed FOP-format hyphenation grammar with HyphenationCompoundWordTokenFilter.getHyphenationTree (or HyphenationTree.loadPatterns, or the factory's hyphenator file) and expect it to load, as the TernaryTree javadoc's 'room for about 65000 nodes' suggests. For some grammars, for example the current TeX Afrikaans patterns or the repro's 1,310 random 4-8-letter patterns, loading instead throws ArrayIndexOutOfBoundsException: Index 2048 out of bounds for length 2048 from TernaryTree.insert, called by compact() during trimToSize(). The cause: insert(char[] key, int start, char val) reserves room for strlen(key) + 1 nodes counted from index 0 instead of from start, and its only caller with start > 0, compact(), passes a key array whose index 0 is an empty string, so it reserves one node and a multi-node insert that crosses the 2048-node block boundary writes past the arrays; with strlen(key, start), the bound the String overload already uses, every failing grammar loads with all values right.

The test shows that loading a well-formed 2-pattern hyphenation grammar through HyphenationCompoundWordTokenFilter.getHyphenationTree(InputSource) throws, because TernaryTree.insert(char[] key, int start, char val) (called by trimToSize() -> compact()) sizes its node arrays with strlen(key) instead of strlen(key, start).

Expected vs actual

Expected: the grammar loads. Actual: ArrayIndexOutOfBoundsException: Index 2048 out of bounds for length 2048 from TernaryTree.insert (with -ea and -da); with strlen(key, start) + 1 it loads.

Files

Run it

# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.analysis.compound.TestHyphenationGrammarLongPatternsLoad

Gradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).

Feedback

Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.

Snapshot of 2026-11-04. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.