The completion filter adds romaji tokens to each Japanese word, read from the bundled table romaji_map.txt, so a suggester can match Latin-letter input. The row for ッヘ has a look-alike hiragana character in its key, so it never matches.
![Summary card of the bug in four panels. Panel 1, what you do: index ヨッヘン with JapaneseCompletionAnalyzer and type yohhen, expecting [ヨッヘン]; the javadoc lists Hepburn, Kunrei and Wāpuro romanization as supported. Panel 2: row 219 of romaji_map.txt, among the rows ッハ, ッヒ, ッフ, ッホ, uses the hiragana へ (U+3078, bytes e3 81 b8) instead of the katakana ヘ (U+30D8, bytes e3 83 98); inside the analyzer romanize() only receives katakana or a-z, so the row never matches; it is the only one of the 329 keys outside that range. Panel 3, what Lucene does: ヨッヘン is indexed as yoxtuhen; typed yohhen differs at the 3rd letter, so the suggester returns nothing. Panel 4, executed on main with assertions on and off: yohhen and the composing state よっh miss ヨッヘン, while yohhan and kana typed in full are still found; two more typos, ッスォ suxo and ッドゥ ddoxi, have no user impact claimed; with the 3-character fix the six queries shown are found, and after reindexing yoxtuhe and よっ find nothing. Footnote: called directly with the hiragana key, romanize fails its assertion with -ea and returns [hhe] with -da; no analyzer path passes hiragana to it.](../assets/LFV-848/bug.png)
yoxtuhen), why yohhen is not a prefix of it, and the executed result. Select the still to open it at full size. Generated with AI, checked against the code.- You index the names ヨッハン and ヨッヘン as suggestions with
JapaneseCompletionAnalyzerand type their romaji into a completion query. - You expect
yohhento suggest ヨッヘン, asyohhansuggests ヨッハン: Hepburn, Kunrei and Nihon all spell ッヘhhe, and the table has a row for it,ッへ,hhe. - Instead
yohhensuggests nothing, with no error. Typing through an input method, the stateよっh(query termyohh) finds ヨッハン but not ヨッヘン. - Why: in that row the へ is hiragana (U+3078), not katakana ヘ (U+30D8). Inside the analyzer the romanizer only ever sees katakana or a-z, so the row never matches: ッヘ becomes ッ
xtu+ ヘhe, and ヨッヘン is indexed as[ヨッヘン, yoxtuhen, yoxtuhenn]. - Kana typed in full (
よっへ,ヨッヘ) still find it, because both sides share the wrong spelling. The rows ッスォ (suxo) and ッドゥ (ddoxi) carry other rows' spellings; no user impact is claimed for them.
Applications using the opt-in JapaneseCompletionAnalyzer or JapaneseCompletionFilter, with default settings, in every release since 9.0.0. Only words with katakana ッヘ, mostly transliterated names such as ヨッヘン, typed as raw romaji or caught in the input-method state よっh; kana typed in full still match. Rare and silent.
Mechanism, terms and numbers605 words
Kuromoji's completion analyzer (JapaneseCompletionAnalyzer, JapaneseCompletionFilter) indexes each Japanese word together with its romaji spellings, read from the bundled table romaji_map.txt, so an autocomplete can match Latin-letter input. Row 219 of that table, meant for katakana ッヘ, is written ッへ,hhe with the look-alike HIRAGANA LETTER HE (U+3078). Inside the analyzer romanize() only ever receives katakana or a-z, so the row can never match: ッヘ is spelled ッ → xtu plus ヘ → he, ヨッヘン is indexed as yoxtuhen, and typing yohhen (the Hepburn, Kunrei and Nihon spelling) or the input-method state よっh suggests nothing, while kana typed in full still match. Two more rows carry other rows' spellings (ッスォ suxo, ッドゥ ddoxi). All three are typos from the table's creation, in every release from 9.0.0 to 10.6.0 and on main; the fix is three characters of data.
- Sokuon ッ: the small tsu doubles the next consonant. In romaji it is written by doubling: ッハ is
hha, ッヘ ishhein Hepburn, Kunrei and Nihon alike; input methods also acceptxtufor a lone small tsu. JapaneseCompletionFilter: for each word it emits the word itself and every romaji spelling of its katakana reading at the same position (generateOutputs,JapaneseCompletionFilter.java:217-233). The spellings come fromromaji_map.txt;KatakanaRomanizerloads it (KatakanaRomanizer.java:44-84), splits a reading by longest key match and combines the matched rows' spellings (:103-169).- Completion query:
PrefixCompletionQuerysuggests a word when the typed text, after the QUERY-mode analyzer, is a prefix of one of the word's indexed forms. While an input method is still composing, the pending letters are appended to the reading (よっh→ヨッh→yohh, via the partial-input rowッh,hh,romaji_map.txt:332).
romanize() asserts that its input is katakana or a-z (KatakanaRomanizer.java:104), and its only production caller, JapaneseCompletionFilter.generateOutputs (JapaneseCompletionFilter.java:228), converts kana surfaces to katakana (:125-128) and skips every other reading (:221-226). A key with a hiragana character can therefore never be matched on the analyzer path. (KatakanaRomanizer is public; the only other calls in the tree are the module's own tests, with katakana and a-z input. Called directly with the hiragana key and assertions off, romanize returns [hhe]; with assertions on it fails.)
| line | row | what is wrong | siblings / base | should be |
|---|---|---|---|---|
| 219 | ッへ,hhe | the key's へ is HIRAGANA U+3078 (bytes e3 81 b8), the only key in the file outside romanize()'s input domain | ッハ,hha, ッヒ,hhi, ッホ,hho | ッヘ,hhe (katakana U+30D8) |
| 281 | ッスォ,sswo,suxo | suxo is スォ's own spelling (:135), without the doubling | ッスァ..ッスェ = ssuxa..ssuxe | ssuxo |
| 289 | ッドゥ,ddwu,ddoxi | ddoxi is also the spelling of ッドィ (:288), the row above | base ドゥ,dwu,doxu (:143); ddoxa, ddoxe, ddoxo | ddoxu |
On main: romanize(ッヘ) = [xtuhe] (control ッハ → [hha]); romanize(ッスォ) = [sswo, suxo]; romanize(ッドゥ) = [ddwu, ddoxi]. ヨッヘン is indexed as [ヨッヘン, yoxtuhen, yoxtuhenn], ヨッハン as [ヨッハン, yohhan, yohhann]. Typed yohhen and yohhe: no suggestion; よっh: ヨッハン only; よっへ, よっへん, ヨッヘ: ヨッヘン, because the query side makes the same mistake (よっへ and ヨッヘ → yoxtuhe, よっへん → yoxtuhen; ヨッヘ is also a prefix of the surface form).
The code follows the table: romanize() matched a separately coded longest-match model of the same table on all 1,830,854 inputs of length up to 3 and on 57,289,761 inputs of length 4 (0 mismatches; a shortest-match mutant was caught). The defects are in the data rows.
Exposure and likelihood165 words
- Who. Applications that build Japanese autocomplete with
JapaneseCompletionAnalyzerorJapaneseCompletionFilter(also through the analysis SPI asjapaneseCompletion; search servers built on Lucene expose it as akuromoji_completiontoken filter). Opt-in; it reproduces with the default constructors and the standard INDEX/QUERY configuration. - What triggers it. Ordinary katakana words with ッヘ, mostly transliterated names such as ヨッヘン, typed as raw romaji (
yohhe...) or caught in the input-method stateよっh. No special input, no scale. How often such words occur was not measured. - Raw romaji is a designed input. The introducing issue wants 桜 suggested for "さ", "サ" or "sa", and the mid-composition "会ｓｙ" to suggest 会社.
- What still works. Kana typed or composed in full (
よっへ,よっへん,ヨッヘ). - Silent. No exception, no log line: the suggestion list is just shorter.
- Rows 281 and 289. No user impact claimed: words with ッスォ or ッドゥ are rare (an assessment; no frequency was measured), and the first spellings
sswoandddwuwork.
The test shows that Kuromoji's romaji_map.txt row 'ッへ,hhe' has a hiragana へ key, so katakana ッヘ never gets the row's romaji.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.analysis.ja.completion;
import java.util.List;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.util.CharsRef;
public class TestRomajiMapSokuonHe extends LuceneTestCase {
public void testSokuonHeRomanizesToHhe() {
List<CharsRef> actual = KatakanaRomanizer.getInstance().romanize(new CharsRef("ッヘ"));
// expected hhe (row "ッへ,hhe", whose key has a hiragana へ); actual [xtuhe]
assertTrue(
"expected hhe in romanize(ッヘ), but was " + actual, actual.contains(new CharsRef("hhe")));
}
}
# in a copy of this finding's repro folder (repro/LFV-848/, which holds src/)
# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:$LUCENE/lucene/analysis/kuromoji/build/libs/lucene-analysis-kuromoji-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar # JUnit Platform console launcher (Maven Central)
javac -d classes -cp "$CP" $(find src -name '*.java')
# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
-cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
execute --select-class org.apache.lucene.analysis.ja.completion.TestRomajiMapSokuonHe
# the module(s) analysis-kuromoji also need their third-party runtime jars
- Expected vs actual
- Expected: KatakanaRomanizer.romanize(ッヘ) contains hhe (as ッハ -> hha). Actual on main: [xtuhe], with -ea and -da.
More files, environment, full output36 words
| lucene | Lucene main e1faeba08bde (2026-11-02) |
|---|---|
| jdk | 25.0.3+9-LTS |
# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
=> java.lang.AssertionError: expected hhe in romanize(ッヘ), but was [xtuhe]
org.junit.Assert.fail(Assert.java:89)
org.junit.Assert.assertTrue(Assert.java:42)
org.apache.lucene.analysis.ja.completion.TestRomajiMapSokuonHe.testSokuonHeRomanizesToHhe(TestRomajiMapSokuonHe.java:28)- The row states its own mapping,
ッへ,hhe, betweenッヒ,hhiandッホ,hho; its key differs from katakana ッヘ only by a look-alike character. romanize()documents and asserts katakana input (KatakanaRomanizer.java:100-104), so the row can only match input that breaks that precondition.suxoandddoxiare verbatim spellings of other rows (スォ, ッドィ); nothing tests or discusses them.
The full argument224 words
- The row states its own mapping.
ッへ,hhesits betweenッヒ,hhiandッホ,hhoand differs from the katakana key only by a look-alike character. Every other key in the file is katakana, or ッ plus a letter a-z (the partial-input rows,:326-338). - The key is outside the asserted input domain.
romanize()"Translates a sequence of katakana to romaji" (:100) and asserts it (:104); the filter enforces it (JapaneseCompletionFilter.java:125-128,:221-226). - The contract's three systems agree. The javadoc lists "(modified) Hepburn-shiki, Kunrei-shiki (Nihon-shiki) and Wāpuro shiki" (
JapaneseCompletionFilter.java:41-42); all spell ッヘhhe, with the same doubling as the sibling rows and the partial-input rows (ッh,hh; the upstream test assertsサッｋ→sakk,TestJapaneseCompletionFilter.java:213-219). suxoandddoxiare other rows' spellings.suxois スォ's (:135),ddoxiis ッドィ's (:288); an input method composes them to すぉ and っどぃ (the open-source Mozc romaji table).- Nothing pins the current behaviour. No test asserts
hhe,suxo,ssuxoorddoxi; no CHANGES or MIGRATE entry or review comment mentions these rows; the only later commit on the file (#12047) fixed comment typos. - Precedent. In the #297 review a missing
jjispelling was added on request; #12885, a romanization gap in the module's other romanizer, is listed under "Bug Fixes" in 9.10.0.
The devil's-advocate check128 words
The strongest case (the intended-behaviour checker and the cross-model review): the javadoc and the table header say the table "does not strictly comply with the romanization systems listed above, but tries to cover possible keystroke supported by various Input Methods", so coverage is best effort; KatakanaRomanizer is public, and with assertions off a caller passing mixed-script ッへ does get hhe; kana queries still work; the words are rare.
Why it fails: the caveat excuses deviations from the romanization systems (its example is the long-vowel mark), not a row that cannot deliver the mapping it states; mixed-script input breaks romanize()'s documented and asserted precondition; the partial-input design needs the doubled spelling; and suxo and ddoxi are verbatim spellings of other rows. The rest only limits the impact.
Recommended Correct the three rows (recommended)
Correct three characters of romaji_map.txt: the key ッへ becomes katakana ッヘ (U+30D8), suxo becomes ssuxo, ddoxi becomes ddoxu. The repro passes and all 198 kuromoji tests pass, assertions on and off. Existing suggest fields need reindexing.
ッフ,hhu,ffu
-ッへ,hhe
+ッヘ,hhe
ッホ,hho
ッスェ,sswe,ssuxe
-ッスォ,sswo,suxo
+ッスォ,sswo,ssuxo
ッドィ,ddwi,ddoxi
-ッドゥ,ddwu,ddoxi
+ッドゥ,ddwu,ddoxuAll options and trade-offs3 options · 524 words
All options are edits of the bundled table or tests; no Java code changes. The recommended option was validated in a scratch worktree at a9779ef3c193 with the whole :lucene:analysis:kuromoji:test task plus the paste-ready test, assertions on and off, and with the standalone runner on the end-to-end and reproducer tests. A diff viewer shows the old and new line 219 as identical; the new key's bytes are e3 83 83 e3 83 98. Separate observation, not part of this bug: line 142 ドィ,dwi, ends with an empty field, so ドィ lacks doxi (the independent reproducer's extra test fails on main).
The finder's patch (+3/-3); the independent reproducer's own edit is identical.
ッフ,hhu,ffu
-ッへ,hhe
+ッヘ,hhe
ッホ,hho
ッスェ,sswe,ssuxe
-ッスォ,sswo,suxo
+ッスォ,sswo,ssuxo
ッドィ,ddwi,ddoxi
-ッドゥ,ddwu,ddoxi
+ッドゥ,ddwu,ddoxuTrade-offs
- Existing suggest fields must be reindexed. Until then, with a fixed Lucene and an old index, the hiragana query
よっへ(today[よっへ, yoxtuhe], after the fix[よっへ, yohhe]) no longer matches ヨッヘン; the katakanaヨッヘstill does through the surface form (from the two token lists; the mixed case was not run). - After reindexing, the accidental spelling
yoxtuheand the composing stateよっ(query tokens[よっ, yoxtu]before and after the fix), which find ヨッヘン today only through the wrongxtu, stop finding it: with the fixed table both return[](executed). ヨッハン behaves that way today:yoxtuhaandよっdo not find it. - Token counts do not change: ッヘ has one spelling before and after; ッスォ and ッドゥ keep two.
What the checks validated
- Scratch worktree at
a9779ef3c193, gradle, seed DEADBEEF: the paste-ready test fails 3 of 3 on stock main with assertions on and off; with this fix the whole:lucene:analysis:kuromoji:testtask passes, 31 classes, 198 tests, 0 failures (7 skipped, all in pre-existing module tests: TestExtendedMode 1, TestFactories 1, TestJapaneseNumberFilter 2, TestJapaneseTokenizer 3), with assertions on and off. That includes the completion suites (TestJapaneseCompletionAnalyzer 4, TestJapaneseCompletionFilter 5, TestJapaneseCompletionFilterFactory 2, TestKatakanaRomanizer 2). - With the paste-ready test,
ecjLintTestandcheckGoogleJavaFormatpass. - The repro runner with the fixed table first on the class path: the paste-ready test, the finder's end-to-end test and the independent reproducer's 7 tests pass. Token probe: ヨッヘン →
[ヨッヘン, yohhen, yohhenn], ッスォ →[ッスォ, sswo, ssuxo], ッドゥ →[ッドゥ, ddwu, ddoxu].
Read romaji_map.txt as a class resource and assert CharSequenceUtils.isKatakanaOrHWAlphabets for every key, as the independent reproducer's testEveryTableKeyIsInRomanizerInputDomain does.
Trade-offs
Catches the next look-alike typo with no code change. Making the loader reject such a key instead would turn a future typo into a class-load error for every user; a test is the gentler guard. The loader variant was not run.
What the checks validated
The reproducer's scan fails on stock main ("line 219 'ッへ,hhe' key code points: U+30C3 U+3078") and passes with the recommended fix.
Apply toKatakana to the first column in KatakanaRomanizer's loader.
Trade-offs
Hides this typo instead of fixing it, leaves rows 281 and 289 wrong, and would hide the next one. Untested.
Search servers that expose this filter as kuromoji_completion load the same table from the Lucene jar, and ports that copied romaji_map.txt carry the same rows, so their Japanese autocomplete should miss the same spellings (inferred from the code; not run there).
A self-check script over every table row flagged a key with a hiragana character and two rows with other rows' spellings. Checks of the table's own rules confirmed them, and a bounded-exhaustive check showed KatakanaRomanizer implements the table as written (0 mismatches on 59 million inputs), so the defects are in the data.
The method in full204 words
- First signal. A planning probe's self-check of
romaji_map.txt(a script over every row) reported HIRAGANA-IN-KEY for line 219 (ッへ, U+3078) and GEMINATION-MISMATCH and SAME-SPELLING-FOR-DIFFERENT-KANA for lines 281 and 289. - Table laws. A checker of the table's own rules: L1 (every key lies in
romanize()'s katakana/a-z input domain) flags only line 219; L2s (each spelling of a small-tsu row doubles a spelling of its base) flags lines 281 and 289, among its four hits (the other two, line 249 ッイyyiand line 263 ッウwwu, are not part of this bug); L3 (input-method round trip against the open-source Mozc romaji table) showssuxocomposes to すぉ andddoxito っどぃ. - Translation validation (bounded-exhaustive). Part A:
romanize()equals an independent longest-match model of the table on all 1,830,854 inputs of length up to 3 and 57,289,761 inputs of length 4 (0 mismatches; a shortest-match mutant: 33,518 mismatches). Part B: the standard spellinghheof ッヘ is missing in Hepburn, Kunrei and Nihon alike. - Reduction. A public-API JUnit with the document suggester on the stock main jars, then this doc's kuromoji-only test.
- Tier. Semi-formal: the first signal came from a script and a custom checker, not from a formal tool's own output.
No matching prior report found in the sources searched (as of 2026-10-28): GitHub issues, PRs, commits and code, JIRA, the mailing lists, open PRs' changed files, downstream projects and the web. Code search finds only verbatim copies of the bad row; nobody has published a corrected one.
Searches4 searches
No report of these rows was found. The closest upstream items are the introducing PR #297 (whose review threads on the file are about comment lines and a missing jji) and #12885, a fixed romanization gap in a different class. The table's data rows have not changed since 9.0.0, and no open PR changes its rows (#16911 only deletes the whole branch_10x tree).
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits (apache/lucene), downstream repositories, mailing lists (dev@, java-user@, issues@, commits@), JIRA archive | 2026-10-28 | the scripted prior-report search, two runs (98 queries) | #11140 and #297 (the introduction), #12885 (another romanizer), #16416 (a refactor); nothing about these rows |
| GitHub code search; the commits API for the table | 2026-10-28 | the exact rows ッへ,hhe, ッヘ,hhe, ssuxo, ddoxu; commits on main, branch_10_6 and branch_9x | only verbatim copies of the bad row (forks, a Rust and a Kotlin port); 0 files with the corrected row; only 4e86df96c03 and #12047 touch the table |
| Open and recent PRs | 2026-10-28 | all 411 open apache/lucene PRs (files of the 27 analysis-labelled or recent ones read); the 34 PRs closed or updated since 2026-10-20; the 15 created since 2026-10-26 | none fixes the rows (#16911 deletes all of branch_10x as a decommission) |
| Web (English and Japanese) | 2026-10-28 | 7 searches or fetches | usage articles; a 2022 post testing the analyzer's romaji variants fails on a reading ambiguity (にほん/にっぽん), a different issue |
In the table since it was created with the feature (4e86df96c03, #297, 2021-09-17; first released in 9.0.0); the rows were never correct and never changed. The history harness fails on all 33 releases 9.0.0-10.5.2, on 10.6.0 (branch_10_6 4ae946e5a9b) and main.
Versions tested7 rows
Introduced with the feature: 4e86df96c03 "LUCENE-10102: Add JapaneseCompletionFilter for Input Method-aware auto-completion (#297)", 2021-09-17 (issue #11140), which created romaji_map.txt; git blame on main assigns lines 219, 281 and 289 to it, and the rows already have their wrong form in the PR's first commit dff3942eed5. The only later commit to the file, 4676a735c18 (#12047, 2023-01-02), fixed header comments.
Executed: the history harness (a plain main() with the same four checks) fails on every release from 9.0.0 to 10.5.2 (33), on the introducing commit (its completion classes and table on the 9.0.0 jars), on the branch_10_6 head 4ae946e5a9b and on main; 8.11.4 has no completion filter. The table blob is 4a803a469ad1 in 9.x and 8136668e7085 in 10.x, branch_10_6 and main, with identical data rows. No fix upstream.
| Version or commit | Verdict | Result |
|---|---|---|
main a9779ef3c193 | affected | paste-ready test fails 3 of 3 (gradle and runner, assertions on and off); the end-to-end and reproducer tests fail |
| 10.5.2, 9.12.3 | affected | paste-ready test fails with the same messages (published jars; 10.5.2 assertions on and off, 9.12.3 on) |
| 9.0.0-10.5.2 (all 33 releases) | affected | history harness: all four defects present on every release |
branch_10_6 4ae946e5a9b | affected | history harness and the finder's JUnit fail (kuromoji compiled from that commit's sources) |
4e86df96c03 (introducing commit) | intro | its completion classes and table on the 9.0.0 jars: the same failure |
| 8.11.4 | no API | no completion package: the harness does not compile |
| main + the 3-character fix | not affected | the paste-ready, end-to-end and reproducer tests pass; the whole kuromoji test task passes (198 tests); 9.0.0 with the fixed introducing table passes too |
The finder, an independent reproducer, an intended-behaviour checker, the adjudication, a follow-up re-run and a credibility audit all ran code and said real. A reviewer from another model family (OpenAI GPT-5.6) could not run code; its "not real" is procedural and it agreed on the root cause. This doc's gradle runs confirm the repro and the fix.
Checks and verdicts325 words
Every verdict except the cross-model review's rests on executed runs on main a9779ef3c193. The cross-model review read the checkout at ea57360912ed, where the table is identical.
| Check | Verdict | What it found |
|---|---|---|
| finder | candidate | Table laws L1-L3 and the bounded-exhaustive check of the real romanizer; the end-to-end JUnit fails on main with assertions on and off and on 10.5.2, and passes with the fixed table. |
| independent reproducer | real, low | Own 7-test JUnit with a second suggester path and a key scan: 6 of 7 fail on main (on and off), 10.5.2 and 9.12.3; its own identical fix passes 7 of 7 and 13 upstream completion tests; found the line 142 side lead. |
| refute review (OpenAI GPT-5.6) | not real, low (procedural, reading only) | Could not run code; said it is "not a substantive refutation", agreed on the root cause and that the inputs are legal, and rated the best-effort argument moderate. Separate re-runs of the finder's test failed on main and 10.5.2. |
| intended-behaviour checker | real, low | Built the best-effort and mixed-script case and rejected it; its probe shows yohhe and よっh miss ヨッヘン while kana in full find it; re-ran on main and 10.5.2. |
| adjudication | confirmed, high | Re-ran the reproducer's test with a fresh seed (6 of 7 fail, on and off) and its fix (7 of 7 pass); byte check of line 219. Asked to state the impact narrowly and claim no impact for rows 281 and 289. |
| credibility audit | keep | Re-ran the finder's test on main with a new seed: the same four report lines; re-read the table and the katakana-only gate; re-checked prior art. |
| this doc | reproduces; fix validated | Gradle in a scratch worktree: the paste-ready test fails 3 of 3 on stock main with assertions on and off; the fix passes the whole kuromoji test task, 198 tests, on and off; the runner reproduces on 10.5.2 and 9.12.3. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 547 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,439 words
- Registry title
- Kuromoji's completion table romaji_map.txt has three typo rows: the ッヘ row uses a hiragana へ (U+3078) in its key, so katakana ッヘ is romanized 'xtuhe' instead of 'hhe' and a word like ヨッヘン is not suggested for 'yohhen', and the ッスォ and ッドゥ rows carry the copied spellings 'suxo' and 'ddoxi'
- Area
- analysis / kuromoji (JapaneseCompletionFilter and JapaneseCompletionAnalyzer romanization table romaji_map.txt, read by KatakanaRomanizer)
- Priority
- Low, rank 508: The typo in the ッヘ key and the two rows for words that almost never occur only affect rare names like ヨッヘン typed in raw romaji, which is narrower than its sibling table-gap row LFV-850.
- Impact class
- silent wrong results
- Impact note
- Only applications that use the opt-in kuromoji JapaneseCompletionAnalyzer or JapaneseCompletionFilter (default constructors, INDEX and QUERY mode) are hit, and only for words that contain katakana ッヘ (for example the name ヨッヘン, indexed as 'yoxtuhen'): a raw-romaji query 'yohhen' (the Hepburn, Kunrei/Nihon and IME spelling) suggests nothing and the mid-composition query 'よっh' finds ヨッハン but not ヨッヘン, while queries typed fully in kana ('よっへ', 'ヨッヘ') still match because index and query side share the same wrong romanization; the ッスォ and ッドゥ rows are data typos with practically no reachable words, each keeping a working first spelling; no scale is needed, and every release from 9.0.0 to 10.6.0 and main are affected.
- Summary
- You index the katakana name ヨッヘン with JapaneseCompletionAnalyzer and type its romaji 'yohhen' into a suggester, and expect the name to be suggested, just as ヨッハン is suggested for 'yohhan'. Instead nothing is suggested: the table row meant for ッヘ, 'ッへ,hhe' (romaji_map.txt line 219), has the look-alike HIRAGANA LETTER HE (U+3078) in its key, and romanize() only ever receives katakana or a-z, so the row can never match and ッヘ falls back to 'ッ'->'xtu' plus 'ヘ'->'he' (indexed as 'yoxtuhen'). The same table gives ッスォ the undoubled spelling 'suxo' (line 281, the spelling of スォ) and ッドゥ the spelling 'ddoxi' (line 289, copied from ッドィ) instead of 'ssuxo' and 'ddoxu'; a three-character table edit fixes all three.
- Introduced
- 4e86df96c0385a7b552434d7e2a9bee69f18bd77 (apache/lucene#297, LUCENE-10102 / GitHub issue #11140 'Add JapaneseCompletionFilter for Input Method-aware auto-completion', by GitHub mocobeta, co-authored by rmuir) [read: the commit creates romaji_map.txt (344 lines) with KatakanaRomanizer, JapaneseCompletionFilter and JapaneseCompletionAnalyzer; git blame attributes lines 219 ('ッへ,hhe', U+30C3 U+3078), 281 ('ッスォ,sswo,suxo') and 289 ('ッドゥ,ddwu,ddoxi') to it; inside the squashed PR the three rows already have their wrong form in the first PR commit dff3942eed5 (2021-09-13) and no later PR commit touches them; the only later change to the file, 4676a735c18 (#12047), fixed header-comment typos and the final newline] [exec: the introducing commit's completion classes and table on the 9.0.0 jars fail the history harness]. Present since the file was created, so not a regression (2021-09-17 (merged 2021-09-17T13:37:13Z; the rows were first written in PR commit dff3942eed5, 2021-09-13))
- First affected version
- 9.0.0 [exec] (the first release with JapaneseCompletionFilter; CHANGES.txt lists LUCENE-10102 under 9.0.0 New Features; no 8.x release has the completion package: 8.11.4 executed, the history harness does not compile; the 8.10.1, 8.11.0 and 8.11.4 kuromoji jars contain no completion classes and no romaji_map.txt; the parent commit de45b68c909 has no table)
- Affected versions
- every published release that has the filter: 10.6.0 (revision 4ae946e5a9b, below) and all 33 from 9.0.0 to 10.5.2 [exec: history harness RomajiMapMalformedRowsHistory (main(), -ea), each 'bug-present (4 defects)'; the candidate JUnit also fails on 9.1.0, 9.12.3, 10.0.0 and 10.5.2 [exec] (it does not compile on 9.0.0, whose test framework predates the org.apache.lucene.tests package; the history harness covers 9.0.0 with -ea and -da); the independent reproduction's JUnit fails on 10.5.2 and 9.12.3 [exec]; the introducing commit 4e86df96c03 [exec]; the branch_10_6 head 4ae946e5a9b (10.6.0, released 2026-10-29) [exec; its kuromoji jar compiled with javac from that commit's sources, because the cached snapshot jar set has none]; main a9779ef3c193 [exec, -ea and -da]; upstream main 20ced71c1e6 and origin/branch_10_6 carry the same table blob 8136668e708 [read]. The 9.x releases ship blob 4a803a469ad1 and the 10.x releases blob 8136668e708; they differ only in comment lines and the final newline (#12047), the data rows are identical [read, confirmed by execution]. Not applicable to 8.x and older (API absent) [exec: 8.11.4]. Negative controls: 9.0.0 with the introducing table plus the 3-line fix, and main with the fixed table, pass [exec]. All runs: OpenJDK 25.0.3, Linux aarch64
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its low-confidence “not real” is procedural), an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal; also reached by exhaustive check
- Upstream and prior reports
- none found (2026-10-28, the prior-report record verdict new, read-only, 120 queries: two scripted prior-report searches (98 queries over apache/lucene issues and PRs, commit messages, downstream Elasticsearch, OpenSearch and Solr, the dev@, java-user@, issues@ and commits@ lists and the JIRA archive); GitHub code search for the exact rows (only verbatim copies of the bad rows, in forks and ports such as turbopuffer/alyze and lucene-kmp; no repository carries a corrected 'ッヘ,hhe' or 'ッスォ,sswo,ssuxo'); the commits API for romaji_map.txt (no commit after #12047 on main, branch_10_6 or branch_9x); all 411 open apache/lucene PRs listed, with the files of the 27 analysis-labelled or recently created ones read, and the 34 PRs closed or updated since 2026-10-20 (none edits the table; #16911 deletes the whole branch_10x tree as a branch decommission, not a fix); web searches in English and Japanese, including a 2022 blog post that tests the analyzer's romaji variants (its failing case is a reading ambiguity, にほん/にっぽん, not these rows)). The finder (45 queries), the independent reproduction, the intended-behaviour check (35 queries) and the adjudication found nothing either. Re-checked for this row: the 15 apache/lucene PRs created since 2026-10-26 (#16907-#16926) touch no kuromoji file, and the commits API still lists only 4e86df96c03 and 4676a735c18 for the table. Context, not reports: #11140 / #297 (introduced the table; the review threads on the file are about comment lines and adding 'jji'), #12047 (comment typos only), #12885 and its java-user thread of 2023-12-07 (a romanization gap in JapaneseReadingFormFilter / ToStringUtil, a different table and class, merged as a bug fix in 9.10.0), #16416 (output buffering of the same filter), #11399 / LUCENE-10363 (offsets, open). Not fixed on upstream main 20ced71c1e6 or origin/branch_10_6 (same table blob). Gaps: GitHub issue search does not reliably index the table's Unicode spellings (covered by code search and by reading the plausible items); the web-search service returned rate-limit errors, every query was retried and completed; private reports to security@lucene.apache.org cannot be searched; searches cannot prove absence)
- Lucene commit tested
- main
a9779ef3c193(gradle in a scratch worktree, assertions on and off, and the standalone JUnit runner); the release jars 10.5.2 (assertions on and off) and 9.12.3 (this doc's paste-ready test); all 33 releases 9.0.0-10.5.2, the introducing commit4e86df96c03on the 9.0.0 jars and the branch_10_6 head4ae946e5a9b(the history harness); the 3-character table fix on main (whole kuromoji test task, assertions on and off).
- Finding record
lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/completion/romaji_map.txt:73 (ヘ,he), :135 (スォ,swo,suxo), :143 (ドゥ,dwu,doxu), :219 (ッへ,hhe; bytes e3 83 83 e3 81 b8, U+30C3 U+3078), :281 (ッスォ,sswo,suxo), :288 (ッドィ,ddwi,ddoxi), :289 (ッドゥ,ddwu,ddoxi), :321 (ッ,xtu), :332 (ッh,hh)lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/completion/KatakanaRomanizer.java:100-104 (romanize javadoc 'Translates a sequence of katakana to romaji'; assert CharSequenceUtils.isKatakanaOrHWAlphabets(input))lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseCompletionFilter.java:36-58 (class javadoc: supported systems, 'does not strictly comply ... tries to cover possible keystroke'), :125-128 (kana surface -> toKatakana as reading), :221-228 (skip readings outside katakana/a-z; the only romanize() call)
- The finding in 30 seconds
lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/completion/romaji_map.txt:73, (at a9779ef3c193) :135, :143, :216-220 (ッハ..ッホ; :219 bytes e3 83 83 e3 81 b8), :277-281, :287-291, :321, :326-338 (partial-input rows), :332lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/completion/KatakanaRomanizer.java:44-84 (at a9779ef3c193) (table load), :100-104 (romanize javadoc and assert), :103-169 (longest match and product)lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/completion/CharSequenceUtils.java:45-68 (at a9779ef3c193) (input domain), :76-88 (toKatakana)lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseCompletionFilter.java:125-128, (at a9779ef3c193) :180-193 (mid-composition), :217-233 (generateOutputs)
- Who is affected
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseCompletionAnalyzer.java:36-54 (at a9779ef3c193) (default constructor, INDEX/QUERY chain)lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseCompletionFilterFactory.java:38 (at a9779ef3c193) (SPI name japaneseCompletion)
- Why it looks unintended
lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/completion/romaji_map.txt:4-5 (at a9779ef3c193) (header caveat), :216-220, :135, :143, :281, :288, :289, :326-338lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/completion/KatakanaRomanizer.java:100-104 (at a9779ef3c193)lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseCompletionFilter.java:41-48 (at a9779ef3c193) (javadoc), :125-128, :221-226lucene/analysis/kuromoji/src/test/org/apache/lucene/analysis/ja/TestJapaneseCompletionFilter.java:213-219 (at a9779ef3c193) (サッｋ -> sakk)lucene/CHANGES.txt:3032-3042 (at a9779ef3c193) (9.10.0 Bug Fixes: GITHUB#12885)
- Suggested fix
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/completion/KatakanaRomanizer.java:56-64 (at a9779ef3c193) (loader)lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/completion/romaji_map.txt:142 (at a9779ef3c193) (ドィ,dwi)
- TestRomajiMapSokuonHe.java (JUnit test):
repro/LFV-848/src/org/apache/lucene/analysis/ja/completion/TestRomajiMapSokuonHe.java(31 lines)
The PNG is the summary card. Made in parallel with this doc.
assets/LFV-848/bug.png(547 KB)