ICUTokenizerFactory keeps the UTF-8 byte order mark at the start of a custom per-script rule file (rulefiles) as rule text, so the file either fails with a misleading ICU syntax error or loads without error and the tokenizer then emits no tokens for that script (every release since 4.1.0)

Finding LFV-719Area: analysis / ICU (ICUTokenizerFactory per-script rule files: rulefiles, parseRules)Priority: lowImpact: silent wrong resultsNovelty: new

Finding page for LFV-719

AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.

What it shows

You configure ICUTokenizerFactory with rulefiles="Latn:latin.rbbi" and a rule file saved as UTF-8 with a byte order mark (BOM), as editors such as Windows Notepad often write and as ICU's own rule-file compiler genbrk accepts (it strips the BOM, and its man page recommends one), and expect the same tokens as with the same file without the BOM. Instead ICUTokenizerFactory.parseRules decodes the BOM to U+FEFF and passes it to ICU's RuleBasedBreakIterator(String) as rule text: a file that starts with the rule '.+ {200};' (alone or after '#' comment lines) compiles with U+FEFF as a literal prefix of that rule, so 'ab cd' gives no tokens at all instead of ['ab cd'], and a file that starts with a comment and '!!forward;' (the shape of Lucene's Latin-break-only-on-whitespace.rbbi) fails with 'IllegalArgumentException: Error 66052 at line 2 column 1'. Nothing on disk is corrupted, but in the silent case all text of the tailored script disappears from the analyzed output with no error.

The test shows that ICUTokenizerFactory keeps a UTF-8 byte order mark at the start of a rulefiles rule file as rule text: the file <BOM>.+ {200}; loads without error, then the Latin text is dropped.

Expected vs actual

Expected [ab cd] for ab cd (as without the BOM); actual [] (expected:<[ab cd]> but was:<[]>), with -ea and -da; passes when parseRules skips a leading U+FEFF.

Files

Run it

# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:$LUCENE/lucene/analysis/icu/build/libs/lucene-analysis-icu-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.analysis.icu.segmentation.TestICUTokenizerFactoryRuleFileBom

The module(s) analysis-icu also need their third-party runtime jars; ./gradlew -p lucene/<module> dependencies lists them.

Gradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).

Feedback

Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.

Snapshot of 2026-11-04. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.