ICUTokenizer with myanmarAsWords=false silently drops the Shan and Khamti reduplication marks U+A9E6 and U+AA70

ICUTokenizer splits text into script runs and cuts each run with a break iterator; every piece gets a rule status, and pieces with status 0 (spaces, punctuation) are skipped (ICUTokenizer.java:209-214). With myanmarAsWords=false (the constructor DefaultICUTokenizerConfig(cjkAsWords, false) or the icu factory option; default true), Myanmar runs go to Lucene's rule file MyanmarSyllable.rbbi (DefaultICUTokenizerConfig.java:115-120), documented only as "it will be tokenized as syllables" (:97-98). That file has two status rules: syllables built on a consonant, consonant placeholder or independent vowel {200} (MyanmarSyllable.rbbi:21-27, :44-50) and numbers {100} (:53).

U+A9E6 MYANMAR MODIFIER LETTER SHAN REDUPLICATION and U+AA70 MYANMAR MODIFIER LETTER KHAMTI REDUPLICATION (gc=Lm, Indic_Syllabic_Category Other) are the only two Script=Myanmar letters that are not syllable bases, so they get status 0 and vanish: ကꧦ → [က], ကꩰ → [က], ꧦ → []. The default mode gives [ကꧦ], [ကꩰ], [ꧦ], and so do StandardTokenizer and ICU's root word iterator. In an index, ကꧦ and က become one term, so a same-analyzer search for ကꧦ matches both documents and a search for the mark alone has no terms; the token offsets leave it out. Secondary: a Common-script letter written after Myanmar text is dropped the same way (ကောင်း µm → [ကောင်း, m]).

It arrived with the option in 6.2.0 (LUCENE-7393, which restored unchanged the 3.1-4.10.4 default rules, which already dropped U+AA70) and reproduces on every release run from 6.2.0 to 10.5.2 and on main. A rule change validated in a scratch worktree (rules regenerated with regenerateRbbi, :lucene:analysis:icu:check and all module tests pass) keeps every letter; the recommended variant attaches the marks to their syllable, as the default mode does.

• Syllable mode: myanmarAsWords=false, through new DefaultICUTokenizerConfig(cjkAsWords, false) or the icu factory (ICUTokenizerFactory.java:97, default true). getBreakIterator(UScript.MYANMAR) then returns the iterator compiled from MyanmarSyllable.rbbi instead of Default.brk (DefaultICUTokenizerConfig.java:83-84, :115-120).
• Rule status: the number a break-iterator rule assigns to the piece it matches. ICUTokenizer skips every piece with status 0, "skipping over non-tokens (rule status 0)" (ICUTokenizer.java:209-214).
• ꧦ U+A9E6 MYANMAR MODIFIER LETTER SHAN REDUPLICATION and ꩰ U+AA70 MYANMAR MODIFIER LETTER KHAMTI REDUPLICATION: Script=Myanmar, gc=Lm (modifier letter), Indic_Syllabic_Category Other, Word_Break Other. By their names, they mark reduplication in Shan and Khamti writing, as ๆ does in Thai.

1. MyanmarSyllable.rbbi:27 defines the syllable base $Cons as Indic_Syllabic_Category Consonant, Consonant_Placeholder or Vowel_Independent; :44-47 build a syllable from a base plus marks; :50 gives it {200}, and :53 gives numbers {100}. No other rule has a status.
2. ꧦ and ꩰ are not bases, and not $Extend or $Format, so they cannot attach to the syllable before them. Their piece gets status 0.
3. ICUTokenizer skips it, like a space: ကꧦ → [က].
4. They are the only two such Myanmar letters: an independent sweep of all 120 Script=Myanmar letters (alone and after KA) finds exactly these two lost in syllable mode and kept in the default mode.
5. Secondary: the script iterator keeps a Common-script character after Myanmar text (even after a space) in the Myanmar run, so µ in ကောင်း µm reaches the syllable rules and gets status 0 too.
6. Not letters, but lost the same way: the five Myanmar-script symbols U+109E, U+109F (Shan), U+AA77, U+AA78 and U+AA79 (Aiton) (gc=So) are kept by the default mode and dropped by syllable mode: က႞ → [က႞] and [က]. Neither fix below keeps them (their catch-all covers letters only).

Search, documents ကꧦ and က, same analyzer at query time: the default mode finds 1 of 2 for ကꧦ and 1 of 2 for က; syllable mode on main finds 2 of 2 for both (the query becomes body:က) and builds no query for ꧦ; the recommended fix gives the default mode's counts.

• Who is exposed. Java code that builds ICUTokenizer with DefaultICUTokenizerConfig(cjkAsWords, false), and schemas that set myanmarAsWords="false" on ICUTokenizerFactory (SPI name icu; Solr's reference guide documents the option). A factory rulefiles entry for Myanmar (Mymr:...) replaces the syllable rules, so applications with their own Myanmar rule file get what their rules say.
• Not exposed. The default ICUTokenizer() and the factory default (myanmarAsWords=true). Elasticsearch and OpenSearch hard-code DefaultICUTokenizerConfig(true, true) in their icu_tokenizer.
• What triggers it. Any text with U+A9E6 or U+AA70, every time, with assertions on or off. Ordinary Burmese prose seldom has them; how often Shan and Khamti text uses them was not measured, so the registry rates it rare. Secondary: a Common-script letter such as µ written after Myanmar text.
• What the damage looks like. Silent: no exception, valid index. The marks are never indexed; a text made only of marks gives no token; ကꧦ and က share one term, so a same-analyzer search for ကꧦ also matches every plain က document (2 of 2 instead of 1 of 2), a precision loss; a query for the mark alone analyzes to nothing, so it builds no query clause; the token offsets leave the mark out, so a highlighter that uses them cannot cover it (no highlighter was run).
• Not affected. Other tokens, other values on the same reused tokenizer, cost: 1,200,000-character values tokenize in 53-135 ms, about the same as a control.
• Reindex. Fields analyzed before a fix lack the mark-bearing terms. After fix B a query for ကꧦ analyzes to the term ကꧦ (stock: က), so it no longer matches documents indexed before the fix as က, and an index with old and new segments answers inconsistently: reindex the affected fields.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<က[ꧦ]> but was:<က[]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.icu.segmentation.TestMyanmarSyllableModifierLetter.testShanReduplicationMarkIsKept(TestMyanmarSyllableModifierLetter.java:38)

• The contract. @param myanmarAsWords "true if Myanmar text should undergo dictionary-based segmentation, otherwise it will be tokenized as syllables" (DefaultICUTokenizerConfig.java:97-98). CHANGES 6.2.0: "Add ICUTokenizer option to parse Myanmar text as syllables instead of words ... This allows for the previous tokenization used before Lucene 5" (lucene/CHANGES.txt:9052-9054). The rule file: "Parses Myanmar text, with syllable as token." (MyanmarSyllable.rbbi:18). All describe how the text is split. That every letter stays in some token is implied, not stated.
• Status 0 means "not a token". ICUTokenizer skips "non-tokens (rule status 0)" (ICUTokenizer.java:209), the status that spaces and punctuation get; ꧦ and ꩰ are letters.
• In scope. The rules do tokenize Shan and Khamti consonants (U+1075 SHAN KA, U+A9E0, U+AA60 KHAMTI GA are ISC Consonant), so Shan and Khamti text is text the mode is meant for.
• References. The default mode, ICU's root word iterator (status 200) and StandardTokenizer keep ကꧦ as one token. The Thai, Lao and Khmer repetition marks ๆ, ໆ, ៗ (gc=Lm, ISC Other, Line_Break SA, the same profile) are kept by the default ICUTokenizer, StandardTokenizer and ThaiTokenizer.
• Tests. TestMyanmarSyllable (21 tests) is the only dedicated syllable-mode test and covers Burmese syllable categories only. TestRandomChains also picks myanmarAsWords at random (TestRandomChains.java:496-499), but checks only consistency, so it cannot see a dropped letter. No test in lucene/analysis contains U+A9E6 or U+AA70.
• History. The rules came from LUCENE-2414 (3.1, the default until 4.10.4), were removed by LUCENE-5995 (5.0) and restored unchanged by LUCENE-7393 (6.2.0). #13239 (9.11.0) rewrote $Cons without discussing coverage, so the rules are not frozen. U+A9E6 was added in Unicode 7.0 (2014), after the rules were written. In 2010 (#3243) a different tokenizer dropped Lao and Myanmar letters; committers treated it as a defect and fixed it.

The strongest "intended" case (the devil's-advocate verifier and the cross-model lenses rate it moderate; the independent reproducer weak to moderate): the option exists to restore the pre-5.0 tokenization, and 4.10.4 drops U+AA70 and µ the same way (U+A9E6 is not assigned in its ICU version), so it restores the old behaviour faithfully, drop included; the rules' author called them informal and "not perfect", with "no rules for rarer cases", and TestMyanmarSyllable says the algorithm cannot handle some special cases; status 0 is skipped by design; both classes are @lucene.experimental; and for µ, attaching a Common character to the preceding script's run is the script iterator's intended heuristic (#10793).

Why it fails for the two marks: nothing documents a drop; the admitted imperfections concern syllable structure ("syllable chaining, great sa"), which mis-splits text but does not lose it; the legacy match shows the gap is old, not wanted; the rules were rewritten in 9.11.0; and every other Lucene path keeps these marks and the analogous Thai, Lao and Khmer marks. Verdict: a defect (adjudication: confirmed, medium). What the counter-case gets right: the µ sub-case rests on run assignment upstream defends, so it stays a secondary remark, and a maintainer may call the whole thing a known limitation of best-effort legacy rules.

MyanmarSyllable.brk is generated: edit MyanmarSyllable.rbbi, run ./gradlew :lucene:analysis:icu:regenerateRbbi (it rewrites the .brk files and src/generated/checksums/regenerateRbbi.json; check verifies the checksums). Default.brk comes out byte-identical. Both fixes keep the Burmese syllables of TestMyanmarSyllable unchanged. Neither keeps the five Myanmar symbols U+109E, U+109F, U+AA77, U+AA78 and U+AA79 (gc=So), which syllable mode also drops (their catch-all covers letters only). Not covered by either (a side observation, not claimed): since #13239 the syllable rules emit -, NBSP, – and × as tokens (they are ISC Consonant_Placeholder), for example က-က → [က, -, က].

In MyanmarSyllable.rbbi, one new letter class, one changed line and one new rule (+5/-1 with comments, +3/-1 without), then regenerateRbbi. A syllable may end with U+A9E6 or U+AA70 ($ModifierLetter, derived from Unicode properties: exactly these two in ICU4J 78.3), and a new rule gives any other letter of the run its own {200} token. Syllable mode then gives the default mode's tokens for every input with the marks: ကꧦ → [ကꧦ], ကꧦꧦ → [ကꧦꧦ], ꧦ → [ꧦ]; ကောင်း µm → [ကောင်း, µ, m]. Search counts match the default mode (ကꧦ 1 of 2, က 1 of 2).

+# U+A9E6 and U+AA70, the Shan and Khamti reduplication marks
+$ModifierLetter = [[:Script=Myanmar:] & [:General_Category=Modifier_Letter:]];
 ...
-$MyanmarSyllableEx = $ConsEx ($Virama $ConsEx)? ($AsatEx)*;
+$MyanmarSyllableEx = $ConsEx ($Virama $ConsEx)? ($AsatEx)* ($ModifierLetter ($Extend | $Format)*)*;
 ...
 !!forward;
 $MyanmarJoinedSyllableEx {200};
+# any other letter of a Myanmar run is a token of its own
+[\p{L}] ($Extend | $Format)* {200};

Trade-offs

• Same tokens as the default mode, ICU's root iterator and StandardTokenizer for the marks; ကꧦ becomes a term distinct from က, so a syllable query for က no longer finds ကꧦ (as in the default mode).
• The catch-all makes each other letter a single-letter token: ကောင်း µµ gives [ကောင်း, µ, µ] where the default mode gives [ကောင်း, µµ]. It is not limited to Myanmar letters: [\p{L}] applies to every letter the script iterator puts in a Myanmar run (Common and Inherited letters such as µ), a wider change than the two marks that maintainers may want to scope.
• The marks attach only after a recognised syllable. After a number or a Common-script letter they fall to the catch-all and become their own token: ၁ꧦ (MYANMAR DIGIT ONE + U+A9E6) → [၁, ꧦ], 1ꧦ → [1, ꧦ], µꧦ → [µ, ꧦ], where the default mode gives [၁ꧦ], [1ꧦ], [µꧦ] (stock syllable mode: [၁], [1], no token).
• Changes analysis output for syllable-mode users: a CHANGES line and a reindex note (old and new terms do not match each other). The compiled rules grow by 5,600 bytes, loaded once per JVM; tokenizing speed with the fix was not measured.

What the checks validated

• Gradle, JDK 25, scratch worktree at a9779ef3c193: on stock the new test fails 4 of 6; regenerateRbbi rewrote MyanmarSyllable.brk (13,896 to 19,496 bytes) and the checksum file, and left Default.brk unchanged.
• :lucene:analysis:icu:check passes: the checksum check, checkGoogleJavaFormat, ecjLint, forbiddenApis, checkLicenses and all module tests: 17 suites, 164 tests, 0 failed (the 6 new tests and 148 upstream tests, TestMyanmarSyllable 21 of 21; the 10 tests of TestICUTokenizerCJK are @AwaitsFix).
• The module tests pass with assertions off (164 tests, 0 failed).
• Nightly mode: the finder's and the independent reproducer's tests, the new test, TestMyanmarSyllable and a random-data check of syllable mode (checkRandomData plus 2,000 × the test multiplier, so at least 2,000, random Myanmar-heavy strings through checkAnalysisConsistency): 40 tests, 0 failed.
• The regenerated .brk gives the same tokens as the rules compiled at runtime (12 inputs), and the standalone runner passes the new test 6 of 6 with it, assertions on and off.

Add [\p{L}] ($Extend | $Format)* {200}; after the syllable rule (MyanmarSyllable.rbbi:50), then regenerateRbbi (+2/-0 with its comment). The finder's fix; its regenerated .brk (19,136 bytes) is byte-identical to the finder's compiled file. ကꧦ → [က, ꧦ], ꧦ → [ꧦ], ကောင်း µm → [ကောင်း, µ, m]. Search: ကꧦ as a phrase matches 1 of 2, as a boolean query 2 of 2 (through the shared က), ꧦ 1 of 2.

 !!forward;
 $MyanmarJoinedSyllableEx {200};
+# any other letter of a Myanmar run is a token of its own
+[\p{L}] ($Extend | $Format)* {200};

Trade-offs

The smallest change, and syllable-level recall for က is kept. But the mark after a syllable becomes a separate token, unlike every other Lucene path, so text with marks gets a frequent one-letter term ꧦ that the default mode produces only for a lone mark. Same reindex note.

What the checks validated

Gradle, scratch worktree: :lucene:analysis:icu:check passes (164 tests, 0 failed, the same checks), and in nightly mode the finder's, the independent reproducer's and the new test, TestMyanmarSyllable and the random-data check: 40 tests, 0 failed. The module tests with assertions off were run with option B only.

myanmarAsWords="false" rulefiles="Mymr:MyanmarSyllableFixed.rbbi" on the icu factory, with a copy of MyanmarSyllable.rbbi plus fix B (or A); the factory compiles it at runtime (ICUTokenizerFactory.java:119, :137-151). Executed in the probe with both rule files: the fixed tokens. Or use the default myanmarAsWords=true, which keeps the marks.

Trade-offs

No Lucene change, but each application must carry its own rule file and keep it in sync with the shipped rules.

Not run.

Trade-offs

Changes no indexed terms, but leaves a mode that silently loses letters of the script it is for.

Method: bounded-exhaustive content-coverage sweep on the stock main jars (the study's "semi-formal" tier), hunt unit "tokenizer content-coverage sweep", 2026-10-27. No model checker, SMT solver or proof produced the signal.

• The harness ran the tokenizer configurations, among them ICUTokenizer with myanmarAsWords=false, over 823 letter-class representatives alone, in runs and in pairs with 107 context characters (3 orders), and over every string of length 1-3 on the 107 characters. Oracle: every letter or digit of the input lies inside some token.
• The class is the misses that occur only with myanmarAsWords=false, for example U+A9E6 alone.
• An AI finder then probed the tokens in both modes, recompiled MyanmarSyllable.rbbi with ICU4J (the stock rules compile byte-identically to the shipped .brk) with one added rule, which isolated the root cause, and wrote the public-API JUnit; independent AI tools reproduced, reviewed and judged it (see the checks).