Kuromoji completion: `romaji_map.txt` has no rows for チェ, イェ, ツァ-ツォ, ディ, デュ or ウォ, so typing `chekku`, `ye` or `tsa` suggests nothing

Kuromoji's completion analyzer (JapaneseCompletionAnalyzer, JapaneseCompletionFilter) indexes each Japanese word together with its romaji spellings, so an autocomplete can match Latin-letter input. Its bundled table romaji_map.txt has rows for loanword sounds such as シェ (she), ジェ (je), ティ (thi), ウィ (wi) and even the doubled ッイェ (yye), but none for チェ, イェ, ツァ/ツィ/ツェ/ツォ, ディ, デュ or ウォ. Those kana are spelled out as base kana plus a small vowel (チェック → tixekku, chixekku; イェール → ixeーru; ツァー → tuxaー, tsuxaー), so typing the Hepburn spellings chekku, ye or tsa, or the input-method keystrokes dhi, dhu or who, suggests nothing, with no error. Kana typed through an input method still match. It affects every release from 9.0.0 to 10.6.0 and main; the fix is data only.

• Extended katakana: two-kana units for sounds of loanwords, written with a small vowel: チェ (che, as in チェック "check"), イェ (ye, イェール "Yale"), ツァ (tsa, ツァー "tsar"), ディ (ディスク "disk"), デュ (デュエット "duet"), ウォ (ウォッカ "vodka").
• Romanizations: modified Hepburn writes チェ, イェ, ツァ/ツィ/ツェ/ツォ as che, ye, tsa/tsi/tse/tso. The open-source Mozc input-method table accepts the same keystrokes and maps dhi, dhu, who to でぃ, でゅ, うぉ; a small vowel alone can be typed with x (xe → ぇ).
• JapaneseCompletionFilter: for each word it emits the word itself and every romaji spelling of its katakana reading at the same position (generateOutputs, JapaneseCompletionFilter.java:217-233). The spellings come from romaji_map.txt, where each row maps a key to its spellings (for example シェ,sye,she); KatakanaRomanizer matches the reading longest key first and combines the spellings of the matched rows (KatakanaRomanizer.java:103-169). The javadoc lists "(modified) Hepburn-shiki, Kunrei-shiki (Nihon-shiki) and Wāpuro shiki" as supported (:41-42).
• Completion query: PrefixCompletionQuery suggests a word when the typed text, after the QUERY-mode analyzer, is a prefix of one of the word's indexed forms. Raw romaji such as chekku comes out as the single token chekku.

Index-side tokens on main: チェック → [チェック, tixekku, chixekku], イェール → [イェール, ixeーru], ツァー → [ツァー, tuxaー, tsuxaー], ディスク → [ディスク, dexisuku], デュエット → [デュエット, dexyuetto], ウォッカ → [ウォッカ, uxokka]; compare シェア → [シェア, syea, shea]. Typed chekku, ye, tsa, dhisuku, dhuetto, whokka: no suggestion. Typed shea, thi, wiki, chixekku, and the kana ちぇ, ちぇっk, いぇ, つぁ, でぃ, うぉ: found. Typing raw romaji, ch and ts still find チェック and ツァー, but che, tsa, y, dh, wh, ty and cy find nothing, and w finds ウィキ but not ウォッカ.

The code follows the table: romanize() matched a separately coded longest-match model of the same table, bounded-exhaustively, on all 1,830,854 inputs of length up to 3 and on 57,289,761 inputs of length 4 (0 mismatches; a shortest-match mutant was caught). That shows implementation and model agree over the tested domain; the gap itself is in the data rows.

• Who. Applications that build Japanese autocomplete with JapaneseCompletionAnalyzer or JapaneseCompletionFilter (also through the analysis SPI as japaneseCompletion; search servers built on Lucene expose it as a kuromoji_completion token filter). Opt-in; it reproduces with the default constructors, the standard INDEX/QUERY configuration and no user dictionary.
• What triggers it. Loanwords whose reading has チェ, イェ, ツァ/ツィ/ツェ/ツォ, ディ, デュ or ウォ, typed as raw romaji: チェック chekku, イェール ye..., ツァー tsa..., and with the input-method keystrokes ディスク dhi..., デュエット dhu..., ウォッカ who.... No special input, no scale. How often such words occur was not measured.
• Two groups of users. People who type romaji directly lose the suggestion for these words; raw romaji is a designed input (the introducing issue wants 桜 suggested for "さ", "サ" or "sa"). Input-method users lose it only while a romaji prefix is pending, before the kana appear: y (before いぇ), dh (before でぃ/でゅ), w and then wh (before うぉ, two keystrokes), and ty/cy if they type ちぇ as tye/cye (the Mozc table maps both).
• What still works. Kana committed or composed through an input method (ちぇ, ちぇっk, いぇ, つぁ, でぃ, うぉ) and the explicit small-vowel spellings (chixe, ixe, tsuxa, dexi, uxo).
• Silent. No exception, no log line: the suggestion list is just shorter.
• After a fix the index-side tokens change, so existing suggest fields must be reindexed. The query side needs no change; QUERY mode loads the same table, so kana queries gain the new spellings too (ちぇ → [ちぇ, tye, che, cye, tixe, chixe]).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected che in romanize(チェ), but was [tixe, chixe]
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.ja.completion.TestRomajiMapChe.testCheRomanizesToChe(TestRomajiMapChe.java:28)

• The contract names modified Hepburn. JapaneseCompletionFilter.java:41-42 and romaji_map.txt:3 list "(modified) Hepburn-shiki" as supported. The module's Hepburn romanizer ToStringUtil maps チェ → che (:481), イェ → ye (:304) and ツァ/ツィ/ツェ/ツォ → tsa/tsi/tse/tso (:489-498), and TestToStringUtil asserts these (:154, :187-191). That romanizer has a different contract: it gives one modified-Hepburn reading per token for JapaneseReadingFormFilter (ToStringUtil.java:240, JapaneseReadingFormFilter.java:84-87), not a list of accepted keystrokes, so it corroborates the spellings but does not bind the completion table.
• The disclaimer's example does not explain it. JapaneseCompletionFilter.java:44-48: "This does not strictly comply with the romanization systems listed above, but tries to cover possible keystroke supported by various Input Methods. e.g.: Circumflex / Macron representing Chōonpu (長音符) are not supported." Its one example of a deviation is the long-vowel mark; it is introduced with "e.g.", so it need not be the full list. che, ye, tsa, dhi, dhu and who are keystrokes the open-source Mozc romaji table accepts (it maps them to ちぇ, いぇ, つぁ, でぃ, でゅ, うぉ), so the stated aim of covering input-method keystrokes argues for them.
• The table is inconsistent with itself. ッイェ,yye but no イェ; シェ, ジェ and チャ/チュ/チョ but no チェ; ティ/テュ but no ディ/デュ; ウィ/ウェ but no ウォ. It does have クァ, グァ, トゥ, ドゥ and ヴァ (:121-154), which the 1991 government notice on loanword spelling puts in its second, less common table, while its first table has チェ, ツァ/ツェ/ツォ, ディ and デュ (the intended-behaviour checker's reading of the notice).
• The author's own reference has the rows. In #11140 the author named a comparison of romaji tables as a reference; its input-method loanword row lists sye she, zye je, tye che, tsa tsi tse tso, thi, thu, dhi, dhu, ye, who. Lucene's table has sye/she, zye/je, thi and thu but none of the others; nothing shows how the author picked entries from it (the intended-behaviour checker's reading of the page).
• The same kind of gap was fixed on request. In the review of PR #297, a reviewer (GitHub johtani) asked on the ッジ,zzi row "Should we add jji? I usually use jajji for ジャッジ"; the author (GitHub mocobeta) answered "yes, of course" and added it.
• Nothing pins the current spellings. No completion test covers these kana, no comment, CHANGES or MIGRATE entry mentions them, and an anchored exact-key git log --all -G search finds no commit that ever added or removed such a row.
• Not proof of intent either way. No javadoc, test, CHANGES entry, commit message, saved PR text or code comment explains the absence of these rows; the points above support an undocumented omission, which maintainers may also label an alias-coverage improvement.

The strongest case (the cross-model review and the intended-behaviour checker): the header says the table "does not strictly comply" and only "tries to cover" keystrokes, and every extended katakana already has some spelling through the small-vowel rows (chixe, ixe, tsuxa, dexi, uxo all work), so this is missing coverage, an enhancement, not a broken contract. ToStringUtil serves another contract, and every extra alias multiplies the indexed forms (a separate unbounded-expansion issue).

Why it is weaker than it looks: the one deviation the documentation names is the long-vowel mark (as an example, not a full list); the module's own Hepburn romanizer and its test use che, ye and tsa; the table covers the siblings and even the doubled ッイェ while missing イェ, which no coverage policy explains; and in the original review a spelling that a reviewer said they usually type (jji) was added as an omission. The best-effort wording, the separate ToStringUtil contract and the token cost lower the priority and may decide the label (bug or improvement); none of them shows that the gap was intended. Not claimed: Hepburn di, dyu, wo for ディ, デュ, ウォ, which this table already uses for ヂ, ヂュ and ヲ (:55, :60, :118).

All options are edits of the bundled table only; no Java changes. The recommended option was validated in a scratch worktree at a9779ef3c193 with the whole :lucene:analysis:kuromoji:test task plus the paste-ready test, assertions on and off, and with the standalone runner on the end-to-end and reproducer tests. The row without the fallback spellings was run for チェ only (the finder's first sketch) and loses matches that work today.

Each row sits next to its siblings and follows the table's conventions: the y form first where the sibling rows have one (tye, built like チャ,tya; an input-method spelling the open-source Mozc table accepts, not shown to be an official Kunrei spelling), then the Hepburn and input-method forms, then the spellings the reading gets today (as クァ,kwa,kuxa and ヴォ,vo,vuxo keep them). A word's romaji after the fix are a superset of today's, so nothing that matches now stops matching.

 チュ,tyu,chu,cyu
+チェ,tye,che,cye,tixe,chixe
 チョ,tyo,cho,cyo
 ユ,yu
+イェ,ye,ixe
 ヨ,yo
 ヱ,we
+ウォ,who,uxo
 ヲ,wo,o
 スォ,swo,suxo
+ツァ,tsa,tuxa,tsuxa
+ツィ,tsi,tuxi,tsuxi
+ツェ,tse,tuxe,tsuxe
+ツォ,tso,tuxo,tsuxo
 テョ,tho,texyo
+ディ,dhi,dexi
+デュ,dhu,dexyu

Trade-offs

• Every extra spelling multiplies a word's romaji tokens (the product in KatakanaRomanizer.java:138-156): チェ goes from 2 to 5 spellings, ツァ-ツォ from 2 to 3, the others from 1 to 2, so チェック goes from 2 romaji tokens to 5. That slightly enlarges a separate unbounded-expansion issue (different fix site); the module's random and huge-string tests still pass.
• The ッ-prefixed forms (ッチェ, ッツァ, ...) still use the fallback; the table has no ッシェ or ッジェ row either, so adding them is a separate consistency step.
• Existing suggest fields must be reindexed to gain the new tokens. The query side needs no code or configuration change (kana queries gain the new spellings too).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the paste-ready test fails 2 of 3 on stock main with assertions on and off; with this fix the whole :lucene:analysis:kuromoji:test task passes, 31 classes, 198 tests, 0 failures (7 skipped, all in pre-existing module tests: TestExtendedMode 1, TestFactories 1, TestJapaneseNumberFilter 2, TestJapaneseTokenizer 3), with assertions on and off. That includes the completion suites (TestJapaneseCompletionAnalyzer 4, TestJapaneseCompletionFilter 5, TestJapaneseCompletionFilterFactory 2, TestKatakanaRomanizer 2) and TestToStringUtil (3).
• With the paste-ready test, ecjLintTest and checkGoogleJavaFormat pass.
• The repro runner with the fixed table first on the class path: the paste-ready test passes (assertions on and off), the end-to-end suggester test passes (off), the independent reproducer's test passes 3 of 3 (on). A token probe shows ch, che, chekku, chixekku, y, ye, ts, tsa, dh, dhi, wh and who all completing their words.

Like シェ,sye,she and ウィ,wi, which keep none, for example チェ,tye,che,cye (the finder's first sketch).

Trade-offs

Fewer tokens, but what matches today stops matching. Run for チェ only: chixekku no longer completes チェック, and neither does the composition state ちx (chi then x on the way to ちぇ), which completes it on stock main and with the recommended fix. The paste-ready test's control fails: "'chixekku' does not complete チェック [チェック, tyekku, chekku, cyekku]".

What the checks validated

Run as a negative check only: the paste-ready test fails its control with this チェ row, and the token probe loses chixekku and ちx.

The finder's table-law check lists them as the next gap once the 9 rows exist.

Trade-offs

Consistency only; the table has no ッシェ or ッジェ row either. Untested.

Lucene's Hepburn romanizer gives di, dyu, wo for these kana.

Trade-offs

Not suggested without a maintainer's view: the table already uses them for ヂ (ヂ,di,zi,ji), ヂュ and ヲ (ヲ,wo,o), and the open-source Mozc table turns them into ぢ, ぢゅ and を, so a raw di would then complete both ヂ and ディ words. Untested.

• First signal. A planning probe compared the stock KatakanaRomanizer with the Latin spelling that ICU4J's Katakana-Latin transliterator gives (Hepburn che for チェ, but ie, uo and tsua for イェ, ウォ and ツァ) for a hand-listed chart of 132 gojūon, yōon and extended-katakana units, and flagged 23 units whose ICU spelling the romanizer does not produce, among them チェ, イェ, ウォ, ディ, デュ and ツァ-ツォ; an end-to-end probe showed chekku, disuku and wo suggesting nothing on the stock main jars.
• Table laws. A model of the table's own regularities: law L4 (sibling-family closure) finds シェ/ジェ present but チェ absent, ウィ/ウェ but not ウォ, ッイェ but not イェ, no ツ+small-vowel row, and テァ-テョ without voiced デ rows.
• Translation validation (bounded-exhaustive). Part A: romanize() equals an independent longest-match model of the table on all 1,830,854 inputs of length up to 3 and 57,289,761 inputs of length 4 (0 mismatches; a shortest-match mutant: 33,518 mismatches), so the defect is in the data. Part B: the Hepburn images of these extended katakana are missing (che, ye, tsa, ...; the colliding di, dyu, wo are not claimed). Part C: typing ye through a model input method (the open-source Mozc romaji table) already fails at the state y.
• Reduction. A public-API JUnit with the document suggester on the stock main jars (2026-10-28), then this doc's kuromoji-only test.
• Tier. Non-formal first signal: a check against another implementation's output; the table-law model and the bounded-exhaustive checks came after it.