In romaji the small tsu ッ is written by doubling the next consonant (キット kitto, バッグ baggu). Kuromoji's romaji mode does this for only four kana rows.
ッ switch with only four rows, バッグ and バグ both becoming bagu, and the fix. Select the still to open it at full size. Generated with AI, checked against the code.- You index Japanese text with
JapaneseTokenizerandJapaneseReadingFormFilter(tokenizer, true)(romaji mode,useRomaji=true), whose romanizer is documented as "Romanize katakana with modified hepburn". - You expect the small tsu ッ to double the next consonant, as it does for キット →
kittoand マッチ →matchi: バッグ (bag) →baggu, ベッド (bed) →beddo. - Instead バッグ →
bagu, the same term as バグ (bug), ベッド →bedo, and キッズ (kids) →kizu, the same as キズ (scratch). In the repro 7 of 7 such loanwords lose the ッ; the 4 controls (ッ before the s, t and p rows) pass. - Why: the
case 'ッ'branch has inner cases only for the カ, サ, タ and パ rows. Any other next kana falls through to a barebreak, so nothing is written for the ッ. - No error: bag and bug match each other, and a query typed
baggustaysbagguand matches no document that contains バッグ.
Applications that index or query romaji readings with JapaneseReadingFormFilter and useRomaji=true (opt-in; the default katakana mode is not affected), or call ToStringUtil.getRomanization directly, in every release since 3.6.0. Ordinary loanwords trigger it: バッグ, ベッド, ビッグ, キッズ, スタッフ, ブリッジ. A fix changes indexed terms, so romaji fields need reindexing.
Mechanism, terms and numbers489 words
Kuromoji's JapaneseReadingFormFilter can replace each token with its reading in romaji (useRomaji=true, opt-in). Its romanizer, ToStringUtil.getRomanization, is documented as "Romanize katakana with modified hepburn", and Hepburn writes the small tsu ッ by doubling the next consonant. The method does that only before the カ, サ, タ and パ rows (キット kitto, マッチ matchi); before any other kana its case 'ッ' falls through and writes nothing. So common loanwords lose the ッ: バッグ (bag) becomes bagu, the same term as バグ (bug), ベッド becomes bedo, キッズ becomes kizu like キズ. Distinct words then match each other, and a romaji query typed baggu finds no document with バッグ. There is no error and no index damage. It has been so since the method was written in 2012, in every release from 3.6.0 to 10.5.2 and on main; a fix changes indexed romaji terms, so such fields need reindexing.
- Small tsu (sokuon, ッ): a doubled consonant. キット (kit) is ki-tto, バッグ (bag) is ba-ggu. Hepburn writes it by doubling the consonant that follows,
tchbeforech(マッチmatchi). JapaneseReadingFormFilter: replaces each token with its reading, in katakana (the default) or, withuseRomaji=true, in romaji. For a dictionary word it romanizes the reading, for an unknown word the term text, by callingToStringUtil.getRomanization(JapaneseReadingFormFilter.java:80-88).ToStringUtil.getRomanization: a public static method, documented "Romanize katakana with modified hepburn" (ToStringUtil.java:240,:251). It walks the string once with a lookahead of two characters and aswitchon the current character.
- For ッ the method looks at the next character in an inner switch (
ToStringUtil.java:264-295). It has cases for カキクケコ (k), サシスセソ (s), タチツテト (t) and パピプペポ (p), each appending the letter and leaving withbreak main. - Any other next kana matches no case, leaves the inner switch and reaches the outer
break, which appends nothing. The next iteration romanizes the following kana as usual. - So バッグ is バ
ba+ ッ (nothing) + グgu=bagu, while キット is キki+ ッt+ トto=kitto. - The filter writes that string into the term. Positions, offsets and the token count do not change; only the term text does.
On main: バッグ → bagu (expected baggu), ベッド → bedo, ビッグ → bigu, キッズ → kizu, ブリッジ → buriji, スタッフ → sutafu, バッハ → baha: 7 of 7 wrong, while キット kitto, キップ kippu, ザッシ zasshi and マッチ matchi are right. Through the filter, バッグ and バグ both give the term bagu, and the typed query baggu stays baggu. In the security screen's 3,022-document probe index (random sokuon-heavy text), the term query baggu matched 0 documents on stock main and 116 with the generic fix shadowed in.
Exposure and likelihood215 words
- Who. Users of
JapaneseReadingFormFilter(input, true)or of the SPI factoryjapaneseReadingFormwithuseRomaji=true(default false,JapaneseReadingFormFilterFactory.java:44-50; its javadoc shows the Solr schema form,:26-34), and direct callers ofToStringUtil.getRomanization(packageja.dictis exported,module-info.java:25). - What triggers it. Ordinary loanwords with ッ before g, z/j, d, b or h/f: バッグ, ベッド, ビッグ, キッズ, グッズ, ブリッジ, スタッフ, ヘッド, ドッグ, バッハ. No special input and no scale. How often such words occur in real text was not measured.
- Two symptoms. Minimal pairs collide (バッグ/バグ →
bagu, キッズ/キズ →kizu), so each matches the other; and romaji queries spelled with the doubled consonant (baggu,beddo) analyze to themselves, so they miss the indexedbaguandbedo(run as an index query forbaggu: 0 hits). - Inconsistent output. The same method writes
kittoandmatchi, so users cannot predict when the doubling applies. - Silent. No exception, no log line, no index damage (CheckIndex clean in the security probe).
- How likely. Romaji mode is opt-in and rarely used, and the romanizer is already lossy by design (it drops the long-vowel mark ー, so ビール and ビル share a term); nobody has reported this since 2012.
- After a fix these words get new romaji terms, so existing romaji fields must be reindexed.
The test shows that Kuromoji's ToStringUtil.getRomanization drops the small tsu ッ before the ガ row instead of doubling the consonant, as it does before k/s/t/p (キット -> kitto).
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.analysis.ja.dict;
import org.apache.lucene.tests.util.LuceneTestCase;
public class TestToStringUtilSokuon extends LuceneTestCase {
public void testSokuonBeforeVoicedConsonant() {
// expected: the small tsu doubles g (as it doubles k in キット -> kitto); actual: "bagu"
assertEquals("baggu", ToStringUtil.getRomanization("バッグ"));
}
}
# in a copy of this finding's repro folder (repro/LFV-605/, which holds src/)
# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:$LUCENE/lucene/analysis/kuromoji/build/libs/lucene-analysis-kuromoji-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar # JUnit Platform console launcher (Maven Central)
javac -d classes -cp "$CP" $(find src -name '*.java')
# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
-cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
execute --select-class org.apache.lucene.analysis.ja.dict.TestToStringUtilSokuon
# the module(s) analysis-kuromoji also need their third-party runtime jars
- Expected vs actual
- Expected
getRomanization("バッグ")to be "baggu"; actual "bagu" (the romaji of バグ), with-eaand-da.
More files, environment, full output32 words
| lucene | Lucene main e1faeba08bde (2026-11-02) |
|---|---|
| jdk | 25.0.3+9-LTS |
# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
=> org.junit.ComparisonFailure: expected:<bag[g]u> but was:<bag[]u>
org.junit.Assert.assertEquals(Assert.java:117)
org.junit.Assert.assertEquals(Assert.java:146)
org.apache.lucene.analysis.ja.dict.TestToStringUtilSokuon.testSokuonBeforeVoicedConsonant(TestToStringUtilSokuon.java:25)- Both
getRomanizationoverloads say "Romanize katakana with modified hepburn"; Hepburn doubles the consonant after a small tsu, and the method itself does so for k, s, t and p. - The other rows are simply missing: no case, test, comment or CHANGES entry covers them.
- Lucene's own completion table maps ッグ, ッド, ッズ and ッハ to
ggu,ddo,zzuandhha.
The full argument287 words
- The contract. Both overloads say "Romanize katakana with modified hepburn" (
ToStringUtil.java:240,:251). Hepburn writes the sokuon by doubling the consonant that follows (tchforch), with no exception for voiced consonants (the Hepburn page thatTestToStringUtil.java:36points to). - The method applies the rule itself for four rows (
rokku,zasshi,kittoandmatchi,kippu); the other rows get no handling at all, the inner switch just falls through. - Lucene's own later romaji table doubles them. The completion filter's
romaji_map.txt("covers romanization systems: modified Hepburn-shiki, Kunrei-shiki (Nihon-shiki), and Wāpuro shiki", line 3) hasッグ,ggu(:178),ッジ,zzi,jji(:193),ッズ,zzu(:194),ッド,ddo(:212),ッハ,hha(:216),ッフ,hhu,ffu(:218),ッバ,bba(:224),ッブ,bbu(:226). It is supporting precedent, not an oracle for this method: lines 4-5 say the table "does not strictly comply with the romanization systems listed above" and covers input-method keystrokes, and some rows differ from what fix B writes (ッヂ,ddiat :209, where fix B givesjjibecause this method romanizes ヂ asji). - "Used only for debugging" is stale. The class javadoc says so (
ToStringUtil.java:24), but the TODO in the same method says "now that this is used by readingsfilter and not just for debugging" (:252-253), and in #5499 (LUCENE-4433, メ romanized asmi) a committer wrote "Now its used by at least a tokenfilter"; that table error was fixed as a bug in 2012. - Nothing pins the current output. No upstream test has a ッ input (
TestToStringUtil,TestJapaneseReadingFormFilter), and no comment, CHANGES or MIGRATE entry mentions the sokuon. - Precedent. Romaji output defects of this filter were filed under "Bug Fixes": GITHUB#12885 (OOV hiragana, 9.10.0) and SOLR-3906 (OOV half-width kana).
The devil's-advocate check191 words
The strongest case for "intended" (the independent reproducer, the intended-behaviour checker and the cross-model lenses rate it moderate): the romanizer is approximate on purpose. It drops the long-vowel mark (マージャン → majan, ロバート → robato, both asserted by upstream tests), so ビール and ビル already share a term; TestToStringUtil says its Hepburn table "isnt even thorough or really probably what we want!"; native and Sino-Japanese words double only k, s, t and p, which may be why only those rows exist; and romaji mode is opt-in.
Why it fails: the approximations cited here are explicit code (case 'ー': break;, ン written m before b/p/m) with tests, while this is an unhandled fall-through with no comment or test; the output is inconsistent (kitto but bagu), so it is not a deliberate "ignore doubling" search key; the native-word pattern explains why the cases were missed, not that dropping was chosen, since the dictionary's readings of common loanwords contain exactly these sequences; and Lucene's own completion table doubles these rows. What remains is a labelling question (bug fix or improvement), not intent. Verdict: not intended, a missing-cases defect.
Recommended Add the missing rows to the inner switch (fix B)
Add the missing rows to the inner ッ switch in ToStringUtil.getRomanization: ガ row → g, ザ/ズ/ゼ/ゾ/ヅ → z, ジ/ヂ → j, ダ/デ/ド → d, バ row → b, ハ/ヒ/ヘ/ホ → h, フ → f, ヴ → v (+42 lines). The repro and the whole kuromoji test task pass with assertions on and off. Romaji fields need reindexing.
All options and trade-offs2 options · 851 words
Both options change only the case 'ッ' block, and neither changes the output of any string without ッ (differential over all 761,943 katakana strings of length 1-3: fix B changes 4,758 outputs, fix A 9,984, every one containing ッ). For a ッ directly before a kana of the g, z/j, d, b or h/f rows or ヴ they agree. They differ in 5,252 strings. In 5,226 of them fix B keeps today's output while fix A also doubles before kana that romanize to n, m, y, r or w, before ヷ-ヺ (ッヷ vva), ン, a second ッ or ー (ッナ nna, ッン nn, ッッカ kkka, ッーカ kka). In the other 26, a double ッ before a row that fix B adds (ッッガ), both change today's output and still differ: stock ga, fix B gga, fix A ggga.
In the style of the existing k/s/t/p cases: the ガ row appends g; ザ, ズ, ゼ, ゾ and ヅ (zu) append z; ジ and ヂ (both ji) append j; ダ, デ and ド append d; the バ row appends b; ハ, ヒ, ヘ and ホ append h; フ (fu) appends f; ヴ appends v. Result: バッグ baggu, ベッド beddo, ビッグ biggu, キッズ kizzu, グッズ guzzu, ブリッジ burijji, スタッフ sutaffu, バッハ bahha, ホッブズ hobbuzu; the k/s/t/p results are unchanged.
case 'ガ':
case 'ギ':
case 'グ':
case 'ゲ':
case 'ゴ':
builder.append('g');
break main;
case 'ザ':
case 'ズ':
case 'ゼ':
case 'ゾ':
case 'ヅ':
builder.append('z');
break main;
case 'ジ':
case 'ヂ':
builder.append('j');
break main;
case 'ダ':
case 'デ':
case 'ド':
builder.append('d');
break main;
case 'バ':
case 'ビ':
case 'ブ':
case 'ベ':
case 'ボ':
builder.append('b');
break main;
case 'ハ':
case 'ヒ':
case 'ヘ':
case 'ホ':
builder.append('h');
break main;
case 'フ':
builder.append('f');
break main;
case 'ヴ':
builder.append('v');
break main;Trade-offs
- Explicit and allocation-free, and it touches only rows that occur in loanwords.
- It leaves the rare n, m, y, r, w rows and ヷ-ヺ as today: the ッ still vanishes there (these sequences are phonotactically rare, and the checks advised against claiming them).
- Changes indexed romaji terms of these words: existing romaji fields need reindexing.
What the checks validated
- Whole
:lucene:analysis:kuromoji:testtask plus the paste-ready test, gradle, seed DEADBEEF, scratch worktree ata9779ef3c193: assertions on: 31 classes, 198 tests, 0 failures, 0 errors, 7 skipped; assertions off: 31 classes, 198 tests, 0 failures, 0 errors, 7 skipped. The 7 skips are pre-existing module tests (TestExtendedMode1,TestFactories1,TestJapaneseNumberFilter2,TestJapaneseTokenizer3); the run includesTestToStringUtil(3),TestJapaneseReadingFormFilter(8, withtestRandomData) andTestJapaneseReadingFormFilterFactory(2). ecjLintMain,ecjLintTestandcheckGoogleJavaFormatof the module with fix B and the test: pass.- Earlier, in the hunt: the finder's JUnit passes with this variant shadowed in.
In case 'ッ', romanize the next up to 3 characters into a scratch StringBuilder and append its first letter if it is a consonant (t before ch). It replaces the four existing rows (+11/-28) and gives the same k/s/t/p results.
// the sokuon doubles the first consonant of the following syllable (ch becomes tch)
if (i < len - 1) {
StringBuilder next = new StringBuilder();
getRomanization(next, s.subSequence(i + 1, Math.min(len, i + 4)));
if (next.length() > 1 && next.charAt(0) == 'c' && next.charAt(1) == 'h') {
builder.append('t');
} else if (next.length() > 0
&& next.charAt(0) >= 'a'
&& next.charAt(0) <= 'z'
&& "aiueo".indexOf(next.charAt(0)) < 0) {
builder.append(next.charAt(0));
}
}Trade-offs
- With less code, covers every following sequence whose romanization starts with an ASCII consonant, including the rare n/m/y/r/w rows; for a trailing ッ, ッー and ッ before the unconverted small ヵ/ヶ it still writes nothing for the ッ.
- Allocates one
StringBuilderand recurses on up to 3 characters per ッ: at 4M characters the security screen measured about 101 ns per character on all-ッ input versus 3.9 stock, and 51 versus 10.7 on alternating ッグ input (assertions on, best of 3 on a shared host; the numbers are noisy, and there was no superlinear growth up to 4M characters). - Its extra outputs are debatable: ッッカ becomes
kkka, ッーカ becomeskka. It also doubles non-katakana text after a ッ in a direct call (ッb →bb; stock and fix Bb), which the katakana-only differential does not cover. - Changes indexed romaji terms too.
What the checks validated
- Whole
:lucene:analysis:kuromoji:testtask plus the paste-ready test, gradle, assertions on: 31 classes, 198 tests, 0 failures, 0 errors, 7 skipped. ecjLintMain,ecjLintTestandcheckGoogleJavaFormatof the module with fix A and the test: pass.- Earlier, in the hunt and the registration check: the finder's JUnit, the independent reproduction's JUnits (6/6, assertions off) and the 11 upstream
TestToStringUtil+TestJapaneseReadingFormFiltertests pass with it shadowed in; the finder's exhaustive check then reports no sokuon mismatch except ッヵ/ッヶ (the separate small ヵ/ヶ pass-through).
Search servers that offer this filter's useRomaji option through the factory (its javadoc shows the Solr schema form) run the same code, so their romaji fields should drop the same ッ (inferred from the code; not run on those servers).
Semi-formal. A bounded-exhaustive property check ran the stock romanizer on all 761,943 katakana strings of length 1 to 3 and checked the Hepburn rule "ッ doubles the next syllable's first consonant" for every syllable; it flagged the rows other than k, s, t and p. A public-API JUnit then reproduced the common loanword cases end to end.
The method in full135 words
- Unit. A hunt unit that checked kuromoji and nori token filters against their javadoc, with a bounded-exhaustive harness over the stock kuromoji jar.
- Section C of the harness. All 761,943 strings of length 1-3 over U+30A1..U+30FA plus ー were romanized twice (determinism: 0 mismatches) and checked for romaji-only output; then, for every one- or two-kana syllable, ッ + syllable was checked against "double the syllable's first consonant,
ch→tch". 145 of 200 consonant-initial syllables lost the ッ. That count includes rare sequences (ッ before n, m, r, y, w), so it is not the headline. - End to end. The realistic subset, loanwords with ッ before g, z/j, d, b and h/f, was then reproduced through
JapaneseTokenizer+JapaneseReadingFormFilter(useRomaji=true)with a public-API JUnit, and checked on releases back to 3.6.0 with a version-independent probe.
No prior report found (searched 2026-10-26: 115 tool queries, 61 manual GitHub, JIRA, list and web searches, all 407 open PRs listed, the file lists of the 17 analysis or Japanese ones read). Closest: #5499 (メ → mi, fixed 2012), #12885 (OOV hiragana), and a downstream plugin that replaced this romanizer in 2018 without naming the sokuon.
Searches5 searches
No report of this defect was found. The upstream items about this method are different defects, and the closest downstream item criticized the romaji format for other reasons.
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits (apache/lucene), downstream repositories, mailing lists (dev@, java-user@, issues@, commits@), JIRA, web | 2026-10-26 | the scripted prior-report search, 115 queries in two runs, plus the finder's 45-query run | 280 distinct hits, 1 high and 16 medium, none about the sokuon |
| GitHub, JIRA, lists and web, by hand | 2026-10-26 | 61 searches, e.g. getRomanization, kuromoji sokuon, kuromoji romaji ッ, useRomaji, hepburn; JIRA LUCENE and SOLR romaji / useRomaji / hepburn / sokuon / romanization | #5499 (LUCENE-4433, メ → mi, fixed 2012), #12885 (OOV hiragana, merged 2024), SOLR-3906 (OOV half-width kana), #11140 / #297 (the separate completion romanizer), #4988 (the filter's introduction): all different |
| Open PRs | 2026-10-26 | all 407 open apache/lucene PRs listed; the changed files of the 17 labelled analysis or naming kuromoji/Japanese read | none of the 17 changes the ッ handling (#16911 deletes all of branch_10x; not a fix) |
| Downstream | 2026-10-26 | Japanese analysis plugins and other Lucene-based projects | elasticsearch-sudachi #29 / PR #32 (2018) called this romaji format odd (macrons, ロ゜ → lo) and replaced it with a romanizer that doubles every consonant row except the ラ row (k, s, t, n, h, m, y, w, g, z, d, b, p); it never names the sokuon and nothing was reported upstream |
| Git history | 2026-10-26 | log --follow, -S "case 'ッ':", --grep romaji / hepburn / sokuon | the ッ block is unchanged since the original contribution |
In the method since it was written (cd372bdc838, LUCENE-3305, 2012-01-12); the ッ block never changed. First released in 3.6.0; fails on all 12 releases run from 3.6.0 to 10.5.2 and on main.
Versions tested5 rows
Introduced with the method in the original kuromoji contribution: cd372bdc838 "LUCENE-3305: add Kuromoji Japanese morphological analyzer" (2012-01-12; issue now #4378, no PR) and its branch_3x copy 8c9485fb664 (2012-01-14). git log -S on the block's lines, git log -L and git blame -C -C -M name only that commit. The output reached indexed terms with c3ddb9dc672 "Added KuromojiReadingFormFilter (LUCENE-3915)" (#4988, 2012-03-25). The only later fix to the table, fc84112ece8 "add hepburn test and fix some corner cases" (2012-05-17), left ッ alone. 3.5.0 has no kuromoji module. Not fixed on upstream main a95c754ee00; none of the 17 open analysis or Japanese PRs whose file lists were read changes it.
| Version or commit | Verdict | Result |
|---|---|---|
main a9779ef3c193 | affected | paste-ready test fails 2 of 3 (gradle and runner, assertions on and off); the finder's, the independent reproduction's and the intended-behaviour check's tests fail too |
| 10.5.2, 10.0.0 | affected | paste-ready test fails with the same messages (published jars, assertions on) |
| 3.6.0, 3.6.2, 4.0.0, 4.10.4, 5.5.5, 6.6.6, 7.7.3, 8.11.4, 9.0.0, 9.12.3, 10.0.0, 10.5.2 | affected | version-independent probe: 7 of 7 loanwords lose the ッ, the filter gives bagu for バッグ, the probe's controls pass (ッ before the t and p rows: キット, マッチ, キップ; plus バグ) |
| 3.5.0 | no API | no kuromoji module (the probe reports the API absent) |
| main + fix B, main + fix A | not affected | paste-ready test passes; the whole kuromoji test task passes |
The finder, an independent reproducer, an intended-behaviour checker, the adjudication and a credibility audit all ran code and said real. Reviews by another model family (OpenAI GPT-5.6) could not run code; on reading they agree with the root cause. This doc's gradle runs confirm the repro and both fixes.
Checks and verdicts287 words
Every verdict except the cross-model reviews' rests on executed runs on main a9779ef3c193. The audit kept the bug and lowered its likelihood to rare (opt-in, already lossy romanizer).
| Check | Verdict | What it found |
|---|---|---|
| finder | candidate | Bounded-exhaustive harness (section C); JUnit fails 2/3 on main (assertions on and off), 10.5.2 and 10.0.0; the fix passes the repro and 11 upstream tests. |
| independent reproducer | real | Own JUnits from the claim only: 4/6 fail on main (assertions on and off) and 10.5.2, 2/3 on 9.12.3; 9 of 9 loanwords wrong; k/s/t/p controls pass. Rated the intended-behaviour case moderate. |
| intended-behaviour checker | real | Built the strongest 'approximate by design' case and rejected it; factory-path probe gets 10 of 10 loanwords wrong on main and 10.5.2 while native words come out right. |
| refute review (OpenAI GPT-5.6) | not real, low (procedural) | Could not execute; agrees with the root cause and that the inputs are legal; its own static case (ビッグ → bigu vs キック → kikku) supports real. It caught an overbroad first title, since narrowed. |
| intended review (OpenAI GPT-5.6) | not real, low (procedural) | Could not execute; on substance leans real (stale 'debugging only' javadoc; the completion table doubles these consonants). A separate run re-ran the finder's JUnit: 2/3 fail. |
| adjudication | confirmed, high | Re-ran the reproducer's JUnits with a new seed on main: 4/6 fail, controls pass. |
| credibility audit | kept, likelihood rare | Re-ran the finder's JUnit (assertions on and off): 2/3 fail; lowered the likelihood because romaji mode is opt-in and the romanizer is lossy by design. |
| security screen | none | No superlinear growth in cost up to 4M characters, CheckIndex clean, term queries match the analyzer's output exactly. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 469 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,300 words
- Registry title
- Kuromoji's romaji reading (ToStringUtil.getRomanization, used by JapaneseReadingFormFilter with useRomaji=true) writes the small tsu ッ as a doubled consonant only before the k, s, t and p rows and silently drops it before any other kana, so loanwords with ッ before g, z/j, d, b or h/f lose it: バッグ 'bag' and バグ 'bug' both become 'bagu', ベッド becomes 'bedo'
- Area
- analysis / kuromoji romanization (ToStringUtil.getRomanization; JapaneseReadingFormFilter romaji mode)
- Priority
- Low, rank 506: Kuromoji romaji mode silently drops the small tsu before voiced consonants, so loanword pairs such as bag and bug collide, which is deterministic but sits in an opt-in, rarely used and already lossy romanizer, so it heads the CJK rare-input rows after the factory rows and just above LFV-459.
- Impact class
- silent wrong results
- Impact note
- Applications that index or query Japanese text through JapaneseReadingFormFilter in romaji mode (useRomaji=true or the factory parameter of that name; the default is katakana) or call ToStringUtil.getRomanization directly, in every release from 3.6.0 to 10.5.2 and main: every occurrence of a common loanword with ッ before g, z/j, d, b or h/f (バッグ, ベッド, ビッグ, キッズ, グッズ, スタッフ, ブリッジ, バッハ) is indexed without the doubled consonant, so it shares a term with a different word (バッグ/バグ -> 'bagu', キッズ/キズ -> 'kizu') and a query typed in Hepburn or keyboard romaji ('baggu', 'beddo') does not match it; no scale is needed, there is no error and no index damage, and a fix changes indexed romaji terms, so such fields would need reindexing.
- Summary
- You analyze Japanese text with JapaneseTokenizer and JapaneseReadingFormFilter(useRomaji=true), whose romanizer is documented as 'Romanize katakana with modified hepburn', and expect the small tsu ッ to double the next consonant, as the method already does for キット -> 'kitto' and マッチ -> 'matchi'. Instead the 'ッ' case appends a consonant only for the カ, サ, タ and パ rows and appends nothing otherwise, so バッグ 'bag' becomes 'bagu', the same term as バグ 'bug', ベッド becomes 'bedo' and スタッフ becomes 'sutafu'. Distinct words then match each other, and a romaji query 'baggu' finds no document that contains バッグ.
- Introduced
- cd372bdc83836148a7ae2e2958407ff0a37b7fc2 (trunk, svn r1230748, 2012-01-12, committed by GitHub rmuir: LUCENE-3305 'add Kuromoji Japanese morphological analyzer', now apache/lucene#4378, no PR) and its branch_3x copy 8c9485fb66427e54ff0dac9708fdcbe72ba62e94 (svn r1231536, 2012-01-14): they created ToStringUtil.getRomanization with the 'ッ' switch that appends only k/s/t/p [read: git log -L on the block, git log -S "case 'ッ':" and git blame -C -C -M on main all name only cd372bdc838]. The output reached indexed terms with c3ddb9dc67242b9dacbea753e28a27787d6d3305 (LUCENE-3915 / #4988, 2012-03-25, the reading form filter with useRomaji; branch_3x dd51f252c55) [read]. Not a regression: the method was born with the gap [exec at release granularity: 3.5.0 has no API, 3.6.0 has the bug] (2012-01-12 (romaji filter path added 2012-03-25))
- First affected version
- 3.6.0 [exec], the first release with kuromoji (contrib lucene-kuromoji; there the useRomaji factory parameter is in the Solr factory, Lucene's own factory exists from 4.0.0 [read]); 3.5.0 has no kuromoji module, so no release is unaffected in the strict sense [exec: API absent]; on the trunk line 4.0.0-alpha [read] and 4.0.0 [exec]
- Affected versions
- every release from 3.6.0 through 10.5.2 (the last release before 10.6.0) and main a9779ef3c193. [exec, the follow-up's version-independent main() probe RomajiSokuonHistoryProbe, JDK 25] 7 of 7 loanwords lose the ッ, the filter turns バッグ into 'bagu' and the k/s/t/p controls pass on 3.6.0, 3.6.2, 4.0.0, 4.10.4, 5.5.5, 6.6.6, 7.7.3, 8.11.4, 9.0.0, 9.12.3, 10.0.0, 10.5.2 and main (-ea and -da); with the candidate fix shadowed in, the probe passes on main. [exec, JUnits] the finder's JUnit fails 2/3 on main (-ea, -da), 10.5.2 and 10.0.0; the independent reproduction's JUnits fail 4/6 on main (-ea, -da) and 10.5.2 and 2/3 (filter test only) on 9.12.3. [read] releases in between were not executed; the 'ッ' block is unchanged since cd372bdc838 (git log -L) and tag containment covers them. Not fixed upstream: ToStringUtil.java is identical on ea57360912ed and a9779ef3c193, no commit after a9779ef touches lucene/analysis/kuromoji (upstream head a95c754ee00, 2026-10-26), and no open PR changes the ッ handling.
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its low-confidence “not real” is procedural), an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal: exhaustive check
- Upstream and prior reports
- none found (2026-10-26, read-only; the prior-report record, verdict new plus the finder's run: 115 scripted prior-report search queries over apache/lucene issues, PRs and commits, downstream repositories, the Lucene mailing lists, the JIRA archive and the web, 61 manual GitHub, JIRA, list and web searches, the finder's 45 scripted prior-report search queries, and the changed files of all 407 open apache/lucene PRs): no issue, PR, JIRA entry, list thread or downstream report says the romanizer drops ッ. Related, different defects: #5499 (LUCENE-4433, メ romanized as 'mi', fixed 2012 by fc84112ece8, which left the ッ switch alone), #12885 (out-of-vocabulary hiragana not romanized, merged 2024) and its java-user@ thread, SOLR-3906 (OOV half-width kana, 2012), #11140 / #297 (LUCENE-10102, the separate KatakanaRomanizer; a reviewer asked for 'jji' for ジャッジ there). Closest downstream item: WorksApplications/elasticsearch-sudachi#29 (2018) called ToStringUtil's romaji format odd (macrons, ロ゜ -> lo) and PR #32 replaced it with a romanizer that doubles every row; neither mentions the sokuon (issue #29 re-read via the GitHub API on 2026-10-26) and nothing was reported upstream. Not fixed on upstream main a95c754ee00. Gaps: one Stack Overflow page and one blog could not be read (bot check); GitHub search does not stem.
- Lucene commit tested
- main
a9779ef3c193(gradle in a scratch worktree with assertions on and off, and the standalone JUnit runner); the release jars 10.5.2 and 10.0.0 (this doc's paste-ready test); 3.6.0, 3.6.2, 4.0.0, 4.10.4, 5.5.5, 6.6.6, 7.7.3, 8.11.4, 9.0.0, 9.12.3, 10.0.0 and 10.5.2 (the history step's version-independent probe); fix B and fix A on main (gradle, the whole kuromoji test task).
- Finding record
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/ToStringUtil.java:24 (at a9779ef3c193) (class javadoc 'used only for debugging'), :240 and :251 (javadoc 'Romanize katakana with modified hepburn'), :252-253 (TODO 'now that this is used by readingsfilter and not just for debugging'), :264-295 (case 'ッ': inner switch appends 'k', 's', 't', 'p' for the カ/サ/タ/パ rows, then a bare 'break' that appends nothing)lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilter.java:27-28 (at a9779ef3c193) (javadoc 'reading of a token in either katakana or romaji form'), :62-99 (romaji mode calls ToStringUtil.getRomanization on the reading or the OOV term text)lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilterFactory.java:44-50 (at a9779ef3c193) (useRomaji, default false)
- The finding in 30 seconds
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/ToStringUtil.java:240, (at a9779ef3c193) :251-295 (javadoc; lookahead; case 'ッ')lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilter.java:26-29, (at a9779ef3c193) :62-99 (javadoc; romaji mode)
- Who is affected
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilterFactory.java:26-34, (at a9779ef3c193) :44-50 (Solr schema example; useRomaji default false)lucene/analysis/kuromoji/src/java/module-info.java:25 (at a9779ef3c193) (exports ja.dict)
- Why it looks unintended
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/ToStringUtil.java:24, (at a9779ef3c193) :240, :251-253, :264-295, :768-800, :1089-1090lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/completion/romaji_map.txt:3, (at a9779ef3c193) :4-5, :178, :193, :194, :209, :212, :216, :218, :224, :226lucene/analysis/kuromoji/src/test/org/apache/lucene/analysis/ja/dict/TestToStringUtil.java:28-37 (at a9779ef3c193) (majan; Hepburn page; 'isnt even thorough')lucene/analysis/kuromoji/src/test/org/apache/lucene/analysis/ja/TestJapaneseReadingFormFilter.java:96-101 (at a9779ef3c193) (robato)lucene/CHANGES.txt:3042 (at a9779ef3c193) (GITHUB#12885, 9.10.0 Bug Fixes), :14729 (SOLR-3906, Bug Fixes)
- TestToStringUtilSokuon.java (JUnit test):
repro/LFV-605/src/org/apache/lucene/analysis/ja/dict/TestToStringUtilSokuon.java(27 lines)
Made in parallel with this doc: every code row is the source at a9779ef3c193 and every romaji string, term and hit list is read from an executed probe on the stock main jars and with the hunt's fix shadowed in (assertions on and off). Its fix step is based on the generic variant (fix A); fix B, the recommended option, gives the same romaji for every word drawn.
assets/LFV-605/bug.png(469 KB)
