ICUTokenizer with myanmarAsWords=false silently drops the Myanmar modifier letters U+A9E6 and U+AA70 (Shan and Khamti reduplication, as in 'ကꧦ'), and Common-script letters such as MICRO SIGN that fall into a Myanmar run, because MyanmarSyllable.rbbi gives a token status only to syllables and numbers

Finding LFV-720Area: analysis / ICU (DefaultICUTokenizerConfig syllable rules MyanmarSyllable.rbbi / MyanmarSyllable.brk, used for Myanmar text when myanmarAsWords=false; ICUTokenizerFactory 'icu' with myanmarAsWords="false")Priority: lowImpact: silent wrong resultsNovelty: new

Finding page for LFV-720

AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.

What it shows

You tokenize Myanmar-script text with ICUTokenizer in syllable mode (new DefaultICUTokenizerConfig(true, false), or ICUTokenizerFactory with myanmarAsWords="false"), whose only documented effect is that Myanmar text 'will be tokenized as syllables', and expect every letter to end up in some token, as in the default mode, where 'ကꧦ' (KA + U+A9E6 MYANMAR MODIFIER LETTER SHAN REDUPLICATION) gives [ကꧦ] and 'ကꩰ' (KA + U+AA70 MYANMAR MODIFIER LETTER KHAMTI REDUPLICATION) gives [ကꩰ]. Instead syllable mode gives [က] for both, a mark on its own gives no token, and 'ကောင်း µm' gives [ကောင်း, m] (the space and µ are Common script, so the script iterator keeps them in the Myanmar run), all without an error. The cause is the rule file: MyanmarSyllable.rbbi gives a non-zero rule status only to syllables built on a consonant, consonant placeholder or independent vowel ({200}) and to numbers ({100}), and ICUTokenizer skips every segment with status 0.

The test shows that ICUTokenizer in Myanmar syllable mode (DefaultICUTokenizerConfig(true, false)) silently drops U+A9E6 MYANMAR MODIFIER LETTER SHAN REDUPLICATION, which word mode keeps.

Expected vs actual

Expected the tokens of KA + U+A9E6 (\u1000\uA9E6) to contain both letters; actual only KA (expected:<က[ꧦ]> but was:<က[]>), with -ea and -da.

Files

Run it

# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:$LUCENE/lucene/analysis/icu/build/libs/lucene-analysis-icu-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.analysis.icu.segmentation.TestMyanmarSyllableModifierLetter

The module(s) analysis-icu also need their third-party runtime jars; ./gradlew -p lucene/<module> dependencies lists them.

Gradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).

Feedback

Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.

Snapshot of 2026-11-04. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.