Kuromoji completion: a small tsu before shi, chi or tsu gets no doubled Hepburn spelling, so typing `zasshi` never suggests 雑誌

Kuromoji's completion analyzer (JapaneseCompletionAnalyzer, JapaneseCompletionFilter) indexes each Japanese word together with its romaji spellings, so an autocomplete can match Latin-letter input. Its bundled table romaji_map.txt gives the small-tsu rows ッシ, ッチ and ッツ a single Kunrei-style spelling (ssi, tti, ttu), although the base rows list shi, chi, tsu and the neighbouring small-tsu rows double the consonant (ssha, ccha, jji). So 雑誌, 決して, マッチ and 三つ are indexed only as zassi, kessite, matti and mittu, and typing the modified-Hepburn or input-method spellings zasshi, kesshite, macchi or mittsu suggests nothing, with no error. It affects every release from 9.0.0 to 10.6.0 and main; the fix is data only.

• Small tsu (sokuon, ッ / っ): doubles the next consonant. 雑誌 is read ザッシ, za-sshi.
• Romanizations: modified Hepburn writes シ, チ, ツ as shi, chi, tsu and a small tsu before them as sshi, tchi, ttsu (雑誌 zasshi, 三つ mittsu, 抹茶 matcha); Kunrei-shiki writes si, ti, tu (zassi, mittu). Japanese input methods accept both and type a small tsu by doubling the consonant (ss, cc, tt) or with tch; macchi is such an input-method spelling.
• JapaneseCompletionFilter: for each word it emits the word itself and every romaji spelling of its katakana reading at the same position. The spellings come from romaji_map.txt, where each row maps a key, mostly a kana unit, to its spellings (for example シ,si,shi); the reading is matched longest key first, and every spelling of every matched row is combined (KatakanaRomanizer.java:103-169). The javadoc lists "(modified) Hepburn-shiki, Kunrei-shiki (Nihon-shiki) and Wāpuro shiki" as supported (JapaneseCompletionFilter.java:41-42).
• Completion query: PrefixCompletionQuery matches a suggestion when the typed text, after the QUERY-mode analyzer, is a prefix of one of the suggestion's indexed forms. Raw romaji such as zasshi comes out as the single token zasshi; no romaji token is added.

Smaller gaps of the same kind: ッチャ, ッチュ, ッチョ have no Hepburn tch form (matcha), and the rare ッヂ, ッヅ, ッヂャ, ッヂュ, ッヂョ rows (:209-210, :213-215) have no zz/jj form.

Index-side tokens on main: 雑誌 → [雑誌, zassi], 決して → [決して, kessite], マッチ → [マッチ, matti], 三つ → [三つ, mittu], こっち → [こっち, kotti]; compare 一緒 → [一緒, issyo, issho] and 抹茶 → [抹茶, mattya, maccha, maccya]. Typed zasshi, kesshite, macchi, matchi, mittsu, matcha: no suggestion, and kocchi is not a prefix of any token of こっち. Typed zassi, kessite, issho, maccha: found. Lucene's Hepburn reading filter (JapaneseReadingFormFilter, useRomaji=true) gives zasshi, kesshite, matchi, mittsu, matcha for the same words.

The code follows the table: romanize() matched a separately coded longest-match model of the same table, bounded-exhaustively, on all 1,830,854 inputs of length up to 3 and on 57,289,761 inputs of length 4 (0 mismatches; a shortest-match mutant was caught). That shows implementation and model agree over the tested domain; the omission itself is in the data rows.

• Who. Applications that build Japanese autocomplete with JapaneseCompletionAnalyzer or JapaneseCompletionFilter (also through the analysis SPI as japaneseCompletion; search servers built on Lucene expose it as a kuromoji_completion token filter). Opt-in; it reproduces with the standard INDEX/QUERY configuration, no user dictionary and no custom table.
• What triggers it. Ordinary words whose reading has ッシ, ッチ or ッツ, typed in modified Hepburn or the way input methods type them: 雑誌 zasshi, 決して kesshite, マッチ macchi / matchi, 三つ mittsu, こっち kocchi, and 抹茶 matcha (the tch gap). No special input, no scale.
• Two groups of users. People who type romaji directly lose the suggestion completely. Input-method users lose it between the small tsu and the completed kana (ざっｓｈ → zassh, みっｔｓ → mitts); handling such mid-composition strings is a stated goal of the feature (JapaneseCompletionFilter.java:184-188).
• What still works. Kana committed through an input method (ざっし → zassi) and the Kunrei spellings (zassi, mittu).
• Silent. No exception, no log line: the suggestion list is just shorter.
• After a fix the index-side tokens change, so existing suggest fields must be reindexed. The query side needs no code or configuration change; QUERY mode loads the same table, so kana queries gain the new spellings too (with fix C, ざっし → [ざっし, zassi, zasshi]).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected zasshi among the index-side tokens, actual: [雑誌, zassi]
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.ja.TestCompletionSokuonHepburn.testZasshiIsIndexed(TestCompletionSokuonHepburn.java:41)

• The contract names modified Hepburn. JapaneseCompletionFilter.java:41-42 and romaji_map.txt:3 list "(modified) Hepburn-shiki" as supported. zasshi, kesshite, mittsu and matcha are the modified-Hepburn forms, and JapaneseReadingFormFilter with useRomaji=true emits them.
• The feature exists to cover more than Hepburn. The introducing issue #11140 says JapaneseReadingFormFilter "supports only modified Hepburn-shiki, so it isn't sufficient to cover all possible romanized forms". For these words the completion filter covers fewer forms than that filter.
• The disclaimer's example does not explain it. JapaneseCompletionFilter.java:44-48: "This does not strictly comply with the romanization systems listed above, but tries to cover possible keystroke supported by various Input Methods. e.g.: Circumflex / Macron representing Chōonpu (長音符) are not supported." Doubled consonants (ss, cc, tt) and tch are ordinary input-method keystrokes for a small tsu (the open-source Mozc input method maps each to っ), so the disclaimer's stated aim argues for them. Its example ("e.g.") of what is not supported is long-vowel marks; no comment, test or doc names the small-tsu rows. The first wording, replaced a day later by 5dfbef313cf (before 9.0.0), excluded "romanji forms which are official but not commonly used with Japanese Input Methods".
• The neighbouring rows follow the rule. The table doubles sh, ch, j and f on the neighbouring small-tsu rows (:189-191, :193, :205-207, :218); the base rows シ, チ, ツ list two spellings each. (It is not consistent everywhere: a sibling bug covers two copy errors on other small-tsu rows, :281 ッスォ and :289 ッドゥ.)
• The same gap was fixed on request. In the review of PR #297, a reviewer (GitHub johtani) asked on the ッジ,zzi row "Should we add jji? I usually use jajji for ジャッジ"; the author (GitHub mocobeta) answered "yes, of course" and added jji, jja, jju, jjo (PR commit fc90e452474), whose diff shows ッチ,tti and ッツ,ttu as unchanged context.
• Nothing pins the current spellings. No upstream test covers a small tsu before シ, チ or ツ (they cover サッカ, ヒャッカテン, コダッk and サッｋ), and no comment, CHANGES or MIGRATE entry mentions it.

The strongest case (the cross-model review and the intended-behaviour checker): the table is a curated, best-effort set ("does not strictly comply"), every extra spelling multiplies the completion paths (the expansion behind a separate registry bug), and the document-suggester classes are @lucene.experimental.

Why it fails: the table gives two to four spellings to 123 of its 329 rows and to 56 of the 158 rows whose key starts with ッ (ッファ, ッフィ, ッフェ, ッフォ list four), so bounding the product cannot explain dropping exactly these rows; the review that added jji treated the same gap as an omission; and the defect is in the tokens of the kuromoji filter, which is not marked experimental and feeds any suggester. What remains is a labelling question (bug fix or improvement), not intent.

All options are edits of the bundled table only; no Java changes. Fix C and fix A were validated in a scratch worktree at a9779ef3c193 with the whole :lucene:analysis:kuromoji:test task plus the paste-ready test; fix A also with assertions off in the independent fact check; fix B was validated by the independent reproducer with the standalone runner. A fixes the four headline misses, B adds the Hepburn tch forms, C also adds the rare ッヂ/ッヅ rows for consistency, which are the least supported part. None touches the separate copy-error rows :281 ッスォ and :289 ッドゥ.

Each affected row gets its base row's spellings with the first consonant doubled (sh → ssh, ch → cch, ts → tts, z/j → zz/jj), plus the Hepburn tch form on ッチ, ッチャ, ッチュ and ッチョ.

-ッシ,ssi
+ッシ,ssi,sshi
-ッチ,tti
-ッツ,ttu
+ッチ,tti,cchi,tchi
+ッツ,ttu,ttsu
-ッチャ,ttya,ccha,ccya
+ッチャ,ttya,ccha,ccya,tcha
   (ッチュ, ッチョ: + tchu, tcho)
-ッヂ,ddi,
-ッヅ,ddu
+ッヂ,ddi,zzi,jji
+ッヅ,ddu,zzu
   (ッヂャ, ッヂュ, ッヂョ: + zzya,jja / zzyu,jju / zzyo,jjo)

Trade-offs

• Every extra spelling multiplies a word's romaji tokens: each ッシ, ッツ or ッヅ in a reading doubles the count, each ッチ, ッヂ, ッヂャ, ッヂュ or ッヂョ triples it, and ッチャ/ッチュ/ッチョ go from 3 to 4 spellings. マッチ goes from [matti] to [matti, macchi, matchi]. That slightly enlarges a separate unbounded-expansion issue (different fix site); the module's random and huge-string tests still pass.
• The ッヂ/ッヅ part (5 rows) concerns rare kana and is not directly asserted by any test run. For it: the table's own closure check flags these rows, and the finder's standards check finds their Hepburn and Kunrei images missing. Against it: in the saved input-method table the new zzi/jji compose to っじ (not っぢ), zzu to っず (not っづ), and the yōon forms likewise to ジ forms, although the base rows already alias ヂ with ジ and ヅ with ズ. Keep it for consistency, not as a headline, or drop it (the 6-row option).
• Existing suggest fields must be reindexed to gain the new tokens. The query side needs no code or configuration change (kana queries gain the new spellings too).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the paste-ready test fails on stock main with assertions on and off; with this fix the whole :lucene:analysis:kuromoji:test task passes, 31 classes, 196 tests, 0 failures (7 skipped, all in pre-existing module tests: TestExtendedMode 1, TestFactories 1, TestJapaneseNumberFilter 2, TestJapaneseTokenizer 3; no stock whole-module run to compare), with assertions on and off.
• With the paste-ready test, ecjLintTest and checkGoogleJavaFormat pass.
• The repro runner with the fixed table first on the class path: the paste-ready test and the end-to-end suggester test pass; earlier, the 13 upstream completion tests passed with the combined table of all four sibling table fixes under 4 seeds, all with assertions on.

ッシ,ssi,sshi, ッチ,tti,cchi,tchi, ッツ,ttu,ttsu (+3/-3).

Trade-offs

Smallest change; leaves matcha and the ッヂ/ッヅ rows as they are.

What the checks validated

The whole kuromoji test task: 31 classes, 196 tests, 0 failures, with assertions on (this doc) and off (the independent fact check); the end-to-end test passes too.

Adds matcha-style Hepburn spellings, without the rare ッヂ/ッヅ rows.

Trade-offs

Middle ground between the two options above.

What the checks validated

By the independent reproducer only, with the standalone runner: its 4 tests pass with this table.

• First signal. A planning probe checked romaji_map.txt against its own small-tsu rule and flagged ッシ, ッチ, ッツ and the ッヂ/ッヅ rows; its end-to-end probe showed zasshi, kesshite, macchi and mittsu returning no suggestion on the stock main jars.
• Table laws. A model of the table's rules: the gemination-closure law found 10 violations, 8 of them this bug's rows (the other 2 belong to a sibling copy-error bug).
• Translation validation (bounded-exhaustive). Part A: romanize() equals an independent longest-match model of the table on all 1,830,854 inputs of length up to 3 and 57,289,761 inputs of length 4 (0 mismatches; a shortest-match mutant: 33,518 mismatches); the Part A counters completed, then that process ran out of memory entering Part B, so Parts B and C ran separately. Part B: the modified-Hepburn, Kunrei and Nihon-shiki images of every sequence of up to 4 kana units, from an independent standards table, isolate the failing small-tsu units. Part C: typing the words through a model input method (the open-source Mozc romaji table) finds the failing composition states っsh, っts and tc.
• Reduction. A public-API JUnit with the document suggester on the stock main jars (2026-10-28), then this doc's kuromoji-only test.
• Tier. Semi-formal: the first signal came from a table self-consistency script, not from a solver's or model checker's own output.