Kuromoji completion: while an input method shows a small tsu plus a pending c, j, f or v (まっｃ while typing maccha), nothing is suggested

Kuromoji's completion analyzer (JapaneseCompletionAnalyzer, JapaneseCompletionFilter) adds romaji tokens to each Japanese word so an autocomplete can match Latin-letter input. Its QUERY mode is documented as "Input Method aware romanization": when the input method still shows a small tsu plus a pending letter, as in さっｋ while typing sakkaa, it joins the two tokens and romanizes them (sakk), so サッカー stays suggested. The bundled table romaji_map.txt has such rows only for k, g, s, z, t, d, h, b, p, m, y, r and w, although its own full small-tsu rows also double c, j, f and v. So まっｃ (typing maccha), ばっｊ (bajji), すたっｆ (sutaffu) are romanized maxtuc, baxtuj, sutaxtuf and suggest nothing for one or two keystrokes. Every release from 9.0.0 to 10.6.0 and main; the fix is four table rows.

• Small tsu (っ / ッ): doubles the next consonant. 抹茶 is read マッチャ, ma-ccha.
• Composition text: a Japanese input method shows what you type before you commit it. Doubling a consonant shows a small tsu plus the still-pending letter: s-a-k-k shows さっｋ, m-a-c-c shows まっｃ. The open-source Mozc input method's romaji table maps kk, cc, jj, ff, vv (and Hepburn tch) this way.
• QUERY mode: JapaneseCompletionFilter.Mode.QUERY, "Input Method aware romanization. Expected to be used when querying." When a token with a reading (here the kana まっ) is followed by a lowercase-letter token it joins them and romanizes the result: the code comment says "words that are in mid-IME composition are split into two tokens by JapaneseTokenizer; should be recovered when querying ... e.g.: "サッ" + "k" => "サッk"" (JapaneseCompletionFilter.java:180-193). A test asserts サッｋ → sakk.
• romaji_map.txt: each row maps a key to its romaji; KatakanaRomanizer matches the longest key first; at the first position no key matches it stops and appends the rest of the reading, unchanged, to what it has built so far. Rows :326-338 cover a small tsu plus one pending letter: ッk,kk ッg,gg ッs,ss ッz,zz ッt,tt ッd,dd ッh,hh ッb,bb ッp,pp ッm,mm ッy,yy ッr,rr ッw,ww.

The full small-tsu rows double 17 letters; the partial rows cover 13. The missing four:

With no ッc row the longest match at ッ is the bare :321 ッ,xtu, and c matches no key, so it is appended (KatakanaRomanizer.java:162-167). On the index side the full ッチ, ッジ, ッフ and ッヴ rows take a single small tsu, so a kana reading yields xtu before c, j, f or v only after a doubled small tsu (ッッジ → xtujji; checked over all katakana strings up to length 4). No indexed spelling of 抹茶 starts with maxtuc, so the prefix query matches nothing.

Typing maccha: まｃ → mac suggests 抹茶; まっｃ → maxtuc and まっｃｈ → maxtuch suggest nothing; まっちゃ → [mattya, maccha, maccya] suggests 抹茶 again. Likewise ばっｊ → baxtuj (バッジ is [bazzi, bajji]), すたっｆ → sutaxtuf (スタッフ is [sutahhu, sutaffu]), じゃっｊ → [zyaxtuj, jaxtuj] (ジャッジ has jajji), きゃっｃ → kyaxtuc (キャッチャー). Kunrei typing keeps it: まっｔ → matt. Half-width まっc behaves the same.

The code follows the table: romanize() equals an independently coded longest-match model of the same table on 59,120,615 inputs (0 mismatches; a shortest-match mutant was caught). A sweep that checked 367,780 simulated composition states of 43,775 synthetic one- and two-unit words found failing tails only after っc, っf, っj, っv (and っh, which belongs to a sibling bug's malformed ッへ row).

• Who. Applications that build Japanese autocomplete with JapaneseCompletionAnalyzer or JapaneseCompletionFilter in QUERY mode (also through the analysis SPI as japaneseCompletion; search servers built on Lucene expose it as a kuromoji_completion token filter) and whose front end sends the text the input method is still composing, the use the feature was built for (#11140). Opt-in.
• What triggers it. Ordinary typing of words with っち/っちゃ, っじ, っふ/っふぁ or っゔ the Hepburn or wapuro way: 抹茶 maccha, キャッチャー kyacchaa, バッジ bajji, ブリッジ burijji, スタッフ sutaffu, バッファ baffa. Hepburn match... also passes through まっｃｈ. How often users type this way was not measured.
• How long. One keystroke (っｊ, っｆ, っｖ) or two (っｃ, then っｃｈ); the states before and after suggest normally.
• Not hit. Kunrei typists (tt, zz, hh), kana typed or committed in full, raw romaji queries (not joined to kana) and INDEX-mode tokens of kana readings. Nothing throws and the index is unchanged.
• After a fix the new rows match only a small tsu directly followed by a Latin letter. QUERY mode builds that by joining tokens; INDEX mode romanizes each token's reading as it is, and a kana-only reading can never match the new keys. The index-side tokens of the 9 probed words are identical with and without the fix. One exception: a user-dictionary reading may contain Latin letters (readings are not checked), and an entry 試験 read アッc is indexed as axtuc on main but acc with the fix. So existing suggest fields need no reindexing unless a user dictionary has readings with a Latin letter after ッ.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected macc among the query-side tokens, actual: [まっc, maxtuc]
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.ja.TestCompletionPendingLetterAfterSmallTsu.testMidCompositionMacc(TestCompletionPendingLetterAfterSmallTsu.java:41)

• The table contradicts itself. Its full small-tsu rows double 17 letters, its partial-input rows cover 13, and the four missing (c, j, f, v) are the letters that no Kunrei-shiki or Nihon-shiki spelling uses (per a reference table of the three systems; modified Hepburn uses all four; the table itself does not label its spellings by system). The javadoc and the table header list "(modified) Hepburn-shiki, Kunrei-shiki (Nihon-shiki) and Wāpuro shiki" as supported (JapaneseCompletionFilter.java:41-42, romaji_map.txt:3).
• The state is handled on purpose. Mode.QUERY is "Input Method aware romanization" (:77); the join exists for exactly this state (:184-188); TestJapaneseCompletionFilter asserts サッｋ → sakk and TestKatakanaRomanizer asserts コダッk → kodakk.
• The design goal names it. #11140: "When querying, mid-IME composition strings will be sent to the search systems, and auto-complete systems should handle them (or, it may just ignore such inputs, but it hurts users' experience)."
• The jj gap arose in review. In PR #297 a reviewer (GitHub johtani) asked: "Should we add jji? I usually use jajji for ジャッジ. What do you think?"; the author (GitHub mocobeta) answered "yes, of course" and added jji/jja/jju/jjo to the full rows (review commit fc90e452474), but no ッj,jj row. The reviewer's own jajji passes through じゃっｊ, which romanizes to [zyaxtuj, jaxtuj] and matches nothing. The c, f and v gaps were already in the PR's first commit, next to full rows with ccha, ffu and vvu.
• No documented or tested behaviour depends on it. For kana readings, maxtuc, baxtuj, sutaxtuf start an indexed spelling only after a doubled small tsu. The javadoc's only example of an unsupported case is circumflex/macron long vowels; no comment, test, CHANGES or MIGRATE entry mentions the partial rows.

The strongest case (the intended-behaviour checker and the cross-model review): the javadoc says the filter "does not strictly comply ... but tries to cover possible keystroke supported by various Input Methods" (best effort); the 13 partial rows are all Kunrei-shiki consonants (every one except n, which no full small-tsu row doubles), so they could be a deliberate Kunrei-only set; #11140 allows that a system "may just ignore such inputs"; only k is tested; the suggester classes are @lucene.experimental.

Why it fails: the same table accepts cc, jj, ff, vv in its full rows, so a Kunrei-only rule is not what the table does; jji was added on request without the matching partial row, which reads as an oversight; the "may just ignore" clause describes systems that ignore composition text altogether, while this filter handles it for 13 of the 17 letters; and the defect is in the kuromoji filter's tokens, not in the experimental suggester API. What remains is a labelling question (bug fix or improvement), not intent.

The recommended option is a four-line edit of the bundled table, validated in a scratch worktree at a9779ef3c193 with the whole :lucene:analysis:kuromoji:test task plus the paste-ready test, assertions on and off. The two other options were not run.

With it, まっｃ → macc, まっｃｈ → macch, ばっｊ → bajj, ばっｆ → baff, すたっｆ → sutaff, じゃっｊ → [zyajj, jajj], きゃっｃ → kyacc, ぶりっｊ → burijj: each a prefix of an indexed spelling. さっｋ and まっｔ are unchanged.

--- a/lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/completion/romaji_map.txt
+++ b/lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/completion/romaji_map.txt
@@ -336,6 +336,10 @@
 ッy,yy
 ッr,rr
 ッw,ww
+ッc,cc
+ッj,jj
+ッf,ff
+ッv,vv
 
 # below are characters that should be kept but have no explicit romanization rules.
 # Chōonpu (Katakana-Hiragana Prolonged Sound Mark)

Trade-offs

• Each new row has one spelling, so it adds no romaji combinations (a separate expansion issue is untouched).
• For kana readings only query-side tokens change (the new keys contain a Latin letter); the index-side tokens of the 9 probed words are identical with and without the fix, so existing suggest fields should not need reindexing. A user-dictionary reading with a Latin letter after ッ does change (アッc: axtuc before, acc after), so such fields would need reindexing.
• Where a full row lacks the doubled spelling, this fix alone does not bring the suggestion back: ッチ has only tti (a sibling bug), so マッチ typed macchi still misses at まっｃ (macc against matti). The examples here fail for this root cause alone.
• A PR could add one assertion per letter to TestKatakanaRomanizer.testRomanizeWithAlphabets (like コダッk → kodakk) and a まっｃ case to TestJapaneseCompletionFilter.testCompletionQuery (like サッｋ).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the paste-ready test fails on stock main with assertions on and off (2 of 2); with this fix the whole :lucene:analysis:kuromoji:test task passes, 31 classes, 197 tests, 0 failures (7 skipped, all in pre-existing module tests: TestExtendedMode 1, TestFactories 1, TestJapaneseNumberFilter 2, TestJapaneseTokenizer 3), with assertions on and off, including the completion suites TestJapaneseCompletionAnalyzer, TestJapaneseCompletionFilter, TestJapaneseCompletionFilterFactory and TestKatakanaRomanizer.
• With the paste-ready test, ecjLintTest and checkGoogleJavaFormat of the module pass.
• The repro runner with the fixed table first on the class path: the paste-ready test passes with assertions on and off, and the end-to-end suggester test passes with assertions off. Earlier, the independent reproducer's and the intended-behaviour checker's tests passed with the same four rows.

In the static initializer of KatakanaRomanizer (:44-84), add ッX → XX for every letter X that some full ッ row doubles.

Trade-offs

Self-maintaining if rows are added later, but it moves a table rule into code and changes behaviour that is now plain data; more than the defect needs. Not run.

Add "a small tsu followed by a pending c, j, f or v is not supported" to the javadoc's limitation (JapaneseCompletionFilter.java:44-48).

Trade-offs

No behaviour change; contradicts the table's own full rows. Not recommended.

• Table laws. A model of the table's rules; law L2p (partial-input rows ッ<letter> must cover every letter the full ッ rows double) flagged exactly ッc, ッf, ッj, ッv.
• Translation validation (bounded-exhaustive). Part A: romanize() equals an independent longest-match model of the table on all 1,830,854 inputs of length up to 3 and all 57,289,761 inputs of length 4 over the key katakana (0 mismatches; a shortest-match mutant was caught), so the code agrees with the table on those inputs and the fault is in the data. The same log then ends with an out-of-memory error in a later part that is not used here.
• Composition sweep. Part C2 took all 43,775 one- and two-unit katakana words (up to 4 kana, no ン) and typed each index-side spelling that a model input method (the open-source Mozc romaji table) composes back to the word (5,643 word/spelling pairs that do not were skipped). Each composition state, romanized by the stock QUERY path, had to stay a prefix of an indexed spelling, checked up to the first failing state of each spelling: 367,780 states. The only failing tails: っc 918, っf 1,993, っj 1,075, っv 1,687 word/spelling pairs, and っh 463 (a sibling bug's ッへ row). With all table fixes: 0 of 432,288 checked states fail (8,344 pairs skipped). The two totals are not comparable: the fixed table has more spellings and the stock run stops each failing spelling early.
• Reduction. A public-API JUnit with the document suggester on the stock main jars (2026-10-28), then this doc's kuromoji-only test.
• Tier. Semi-formal: the first signal came from a table self-consistency script, not from a solver's or model checker's own output.