All findings · Lucene finding · LFV-605 · romanization-drops-sokuon-before-voiced-consonants

Kuromoji romaji readings drop the small tsu before g, z/j, d, b and h/f, so バッグ (bag) and バグ (bug) both become "bagu"

In romaji mode, JapaneseReadingFormFilter doubles the consonant after a small tsu ッ only before k, s, t and p: キット is kitto, but バッグ (bag) becomes bagu, the same term as バグ (bug), and a query typed baggu does not find it. Every release since 3.6.0.

Summary

Priority
Low, rank 506Kuromoji romaji mode silently drops the small tsu before voiced consonants, so loanword pairs such as bag and bug collide, which is deterministic but sits in an opt-in, rarely used and already lossy romanizer, so it heads the CJK rare-input rows after the factory rows and just above LFV-459.
Impact
silent wrong results
Since
3.6.0 (2012), the first release with kuromojiintroduced 2012-01-12 · over 14 years
Affected
3.6.0 to 10.5.2 and main; 12 releases run
Status
documented
Found by
bounded-exhaustive property check of the romanizer

AI-assisted page: this finding was first signalled by a semi-formal check (an exact check of the real jar, or a modelling hypothesis that a tool then witnessed) and then checked with AI tools; the checks it went through, and what each found, are under Prior reports, history and checks. Not yet triaged by a human. Snapshot of 2026-11-06.

The finding in 30 seconds

In romaji the small tsu ッ is written by doubling the next consonant (キット kitto, バッグ baggu). Kuromoji's romaji mode does this for only four kana rows.

Diagram of the bug: the small tsu ッ is doubled before the k, s, t and p rows (キット kitto, マッチ matchi) but dropped before other rows, so バッグ (bag) and バグ (bug) both become the term bagu and a typed query baggu matches neither.
Step by step: what you index and query, the Hepburn rule, the ッ switch with only four rows, バッグ and バグ both becoming bagu, and the fix. Select the still to open it at full size. Generated with AI, checked against the code.
  1. You index Japanese text with JapaneseTokenizer and JapaneseReadingFormFilter(tokenizer, true) (romaji mode, useRomaji=true), whose romanizer is documented as "Romanize katakana with modified hepburn".
  2. You expect the small tsu ッ to double the next consonant, as it does for キット → kitto and マッチ → matchi: バッグ (bag) → baggu, ベッド (bed) → beddo.
  3. Instead バッグ → bagu, the same term as バグ (bug), ベッド → bedo, and キッズ (kids) → kizu, the same as キズ (scratch). In the repro 7 of 7 such loanwords lose the ッ; the 4 controls (ッ before the s, t and p rows) pass.
  4. Why: the case 'ッ' branch has inner cases only for the カ, サ, タ and パ rows. Any other next kana falls through to a bare break, so nothing is written for the ッ.
  5. No error: bag and bug match each other, and a query typed baggu stays baggu and matches no document that contains バッグ.

Who is affected

Applications that index or query romaji readings with JapaneseReadingFormFilter and useRomaji=true (opt-in; the default katakana mode is not affected), or call ToStringUtil.getRomanization directly, in every release since 3.6.0. Ordinary loanwords trigger it: バッグ, ベッド, ビッグ, キッズ, スタッフ, ブリッジ. A fix changes indexed terms, so romaji fields need reindexing.

Mechanism, terms and numbers489 words

Kuromoji's JapaneseReadingFormFilter can replace each token with its reading in romaji (useRomaji=true, opt-in). Its romanizer, ToStringUtil.getRomanization, is documented as "Romanize katakana with modified hepburn", and Hepburn writes the small tsu ッ by doubling the next consonant. The method does that only before the カ, サ, タ and パ rows (キット kitto, マッチ matchi); before any other kana its case 'ッ' falls through and writes nothing. So common loanwords lose the ッ: バッグ (bag) becomes bagu, the same term as バグ (bug), ベッド becomes bedo, キッズ becomes kizu like キズ. Distinct words then match each other, and a romaji query typed baggu finds no document with バッグ. There is no error and no index damage. It has been so since the method was written in 2012, in every release from 3.6.0 to 10.5.2 and on main; a fix changes indexed romaji terms, so such fields need reindexing.

Terms

  • Small tsu (sokuon, ッ): a doubled consonant. キット (kit) is ki-tto, バッグ (bag) is ba-ggu. Hepburn writes it by doubling the consonant that follows, tch before ch (マッチ matchi).
  • JapaneseReadingFormFilter: replaces each token with its reading, in katakana (the default) or, with useRomaji=true, in romaji. For a dictionary word it romanizes the reading, for an unknown word the term text, by calling ToStringUtil.getRomanization (JapaneseReadingFormFilter.java:80-88).
  • ToStringUtil.getRomanization: a public static method, documented "Romanize katakana with modified hepburn" (ToStringUtil.java:240, :251). It walks the string once with a lookahead of two characters and a switch on the current character.

The bug

  1. For ッ the method looks at the next character in an inner switch (ToStringUtil.java:264-295). It has cases for カキクケコ (k), サシスセソ (s), タチツテト (t) and パピプペポ (p), each appending the letter and leaving with break main.
  2. Any other next kana matches no case, leaves the inner switch and reaches the outer break, which appends nothing. The next iteration romanizes the following kana as usual.
  3. So バッグ is バ ba + ッ (nothing) + グ gu = bagu, while キット is キ ki + ッ t + ト to = kitto.
  4. The filter writes that string into the term. Positions, offsets and the token count do not change; only the term text does.

The numbers

On main: バッグ → bagu (expected baggu), ベッド → bedo, ビッグ → bigu, キッズ → kizu, ブリッジ → buriji, スタッフ → sutafu, バッハ → baha: 7 of 7 wrong, while キット kitto, キップ kippu, ザッシ zasshi and マッチ matchi are right. Through the filter, バッグ and バグ both give the term bagu, and the typed query baggu stays baggu. In the security screen's 3,022-document probe index (random sokuon-heavy text), the term query baggu matched 0 documents on stock main and 116 with the generic fix shadowed in.

Exposure and likelihood215 words
  • Who. Users of JapaneseReadingFormFilter(input, true) or of the SPI factory japaneseReadingForm with useRomaji=true (default false, JapaneseReadingFormFilterFactory.java:44-50; its javadoc shows the Solr schema form, :26-34), and direct callers of ToStringUtil.getRomanization (package ja.dict is exported, module-info.java:25).
  • What triggers it. Ordinary loanwords with ッ before g, z/j, d, b or h/f: バッグ, ベッド, ビッグ, キッズ, グッズ, ブリッジ, スタッフ, ヘッド, ドッグ, バッハ. No special input and no scale. How often such words occur in real text was not measured.
  • Two symptoms. Minimal pairs collide (バッグ/バグ → bagu, キッズ/キズ → kizu), so each matches the other; and romaji queries spelled with the doubled consonant (baggu, beddo) analyze to themselves, so they miss the indexed bagu and bedo (run as an index query for baggu: 0 hits).
  • Inconsistent output. The same method writes kitto and matchi, so users cannot predict when the doubling applies.
  • Silent. No exception, no log line, no index damage (CheckIndex clean in the security probe).
  • How likely. Romaji mode is opt-in and rarely used, and the romanizer is already lossy by design (it drops the long-vowel mark ー, so ビール and ビル share a term); nobody has reported this since 2012.
  • After a fix these words get new romaji terms, so existing romaji fields must be reindexed.

Minimal repro

The test shows that Kuromoji's ToStringUtil.getRomanization drops the small tsu ッ before the ガ row instead of doubling the consonant, as it does before k/s/t/p (キット -> kitto).

TestToStringUtilSokuon.java (JUnit test)27 lines
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.analysis.ja.dict;

import org.apache.lucene.tests.util.LuceneTestCase;

public class TestToStringUtilSokuon extends LuceneTestCase {

  public void testSokuonBeforeVoicedConsonant() {
    // expected: the small tsu doubles g (as it doubles k in キット -> kitto); actual: "bagu"
    assertEquals("baggu", ToStringUtil.getRomanization("バッグ"));
  }
}
Command
# in a copy of this finding's repro folder (repro/LFV-605/, which holds src/)
# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:$LUCENE/lucene/analysis/kuromoji/build/libs/lucene-analysis-kuromoji-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.analysis.ja.dict.TestToStringUtilSokuon
# the module(s) analysis-kuromoji also need their third-party runtime jars
Expected vs actual
Expected getRomanization("バッグ") to be "baggu"; actual "bagu" (the romaji of バグ), with -ea and -da.

All files of this repro: repro/LFV-605/

More files, environment, full output32 words

Environment

luceneLucene main e1faeba08bde (2026-11-02)
jdk25.0.3+9-LTS

Observed output

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<bag[g]u> but was:<bag[]u>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.ja.dict.TestToStringUtilSokuon.testSokuonBeforeVoicedConsonant(TestToStringUtilSokuon.java:25)

Why it looks unintended

  • Both getRomanization overloads say "Romanize katakana with modified hepburn"; Hepburn doubles the consonant after a small tsu, and the method itself does so for k, s, t and p.
  • The other rows are simply missing: no case, test, comment or CHANGES entry covers them.
  • Lucene's own completion table maps ッグ, ッド, ッズ and ッハ to ggu, ddo, zzu and hha.
The full argument287 words
  • The contract. Both overloads say "Romanize katakana with modified hepburn" (ToStringUtil.java:240, :251). Hepburn writes the sokuon by doubling the consonant that follows (tch for ch), with no exception for voiced consonants (the Hepburn page that TestToStringUtil.java:36 points to).
  • The method applies the rule itself for four rows (rokku, zasshi, kitto and matchi, kippu); the other rows get no handling at all, the inner switch just falls through.
  • Lucene's own later romaji table doubles them. The completion filter's romaji_map.txt ("covers romanization systems: modified Hepburn-shiki, Kunrei-shiki (Nihon-shiki), and Wāpuro shiki", line 3) has ッグ,ggu (:178), ッジ,zzi,jji (:193), ッズ,zzu (:194), ッド,ddo (:212), ッハ,hha (:216), ッフ,hhu,ffu (:218), ッバ,bba (:224), ッブ,bbu (:226). It is supporting precedent, not an oracle for this method: lines 4-5 say the table "does not strictly comply with the romanization systems listed above" and covers input-method keystrokes, and some rows differ from what fix B writes (ッヂ,ddi at :209, where fix B gives jji because this method romanizes ヂ as ji).
  • "Used only for debugging" is stale. The class javadoc says so (ToStringUtil.java:24), but the TODO in the same method says "now that this is used by readingsfilter and not just for debugging" (:252-253), and in #5499 (LUCENE-4433, メ romanized as mi) a committer wrote "Now its used by at least a tokenfilter"; that table error was fixed as a bug in 2012.
  • Nothing pins the current output. No upstream test has a ッ input (TestToStringUtil, TestJapaneseReadingFormFilter), and no comment, CHANGES or MIGRATE entry mentions the sokuon.
  • Precedent. Romaji output defects of this filter were filed under "Bug Fixes": GITHUB#12885 (OOV hiragana, 9.10.0) and SOLR-3906 (OOV half-width kana).
The devil's-advocate check191 words

The strongest case for "intended" (the independent reproducer, the intended-behaviour checker and the cross-model lenses rate it moderate): the romanizer is approximate on purpose. It drops the long-vowel mark (マージャン → majan, ロバート → robato, both asserted by upstream tests), so ビール and ビル already share a term; TestToStringUtil says its Hepburn table "isnt even thorough or really probably what we want!"; native and Sino-Japanese words double only k, s, t and p, which may be why only those rows exist; and romaji mode is opt-in.

Why it fails: the approximations cited here are explicit code (case 'ー': break;, ン written m before b/p/m) with tests, while this is an unhandled fall-through with no comment or test; the output is inconsistent (kitto but bagu), so it is not a deliberate "ignore doubling" search key; the native-word pattern explains why the cases were missed, not that dropping was chosen, since the dictionary's readings of common loanwords contain exactly these sequences; and Lucene's own completion table doubles these rows. What remains is a labelling question (bug fix or improvement), not intent. Verdict: not intended, a missing-cases defect.

Suggested fix

Recommended Add the missing rows to the inner switch (fix B)

Add the missing rows to the inner ッ switch in ToStringUtil.getRomanization: ガ row → g, ザ/ズ/ゼ/ゾ/ヅ → z, ジ/ヂ → j, ダ/デ/ド → d, バ row → b, ハ/ヒ/ヘ/ホ → h, フ → f, ヴ → v (+42 lines). The repro and the whole kuromoji test task pass with assertions on and off. Romaji fields need reindexing.

All options and trade-offs2 options · 851 words

Both options change only the case 'ッ' block, and neither changes the output of any string without ッ (differential over all 761,943 katakana strings of length 1-3: fix B changes 4,758 outputs, fix A 9,984, every one containing ッ). For a ッ directly before a kana of the g, z/j, d, b or h/f rows or ヴ they agree. They differ in 5,252 strings. In 5,226 of them fix B keeps today's output while fix A also doubles before kana that romanize to n, m, y, r or w, before ヷ-ヺ (ッヷ vva), ン, a second ッ or ー (ッナ nna, ッン nn, ッッカ kkka, ッーカ kka). In the other 26, a double ッ before a row that fix B adds (ッッガ), both change today's output and still differ: stock ga, fix B gga, fix A ggga.

Add the missing rows to the inner switch (fix B) (validated by the checks) (recommended)

In the style of the existing k/s/t/p cases: the ガ row appends g; ザ, ズ, ゼ, ゾ and ヅ (zu) append z; ジ and ヂ (both ji) append j; ダ, デ and ド append d; the バ row appends b; ハ, ヒ, ヘ and ホ append h; フ (fu) appends f; ヴ appends v. Result: バッグ baggu, ベッド beddo, ビッグ biggu, キッズ kizzu, グッズ guzzu, ブリッジ burijji, スタッフ sutaffu, バッハ bahha, ホッブズ hobbuzu; the k/s/t/p results are unchanged.

            case 'ガ':
            case 'ギ':
            case 'グ':
            case 'ゲ':
            case 'ゴ':
              builder.append('g');
              break main;
            case 'ザ':
            case 'ズ':
            case 'ゼ':
            case 'ゾ':
            case 'ヅ':
              builder.append('z');
              break main;
            case 'ジ':
            case 'ヂ':
              builder.append('j');
              break main;
            case 'ダ':
            case 'デ':
            case 'ド':
              builder.append('d');
              break main;
            case 'バ':
            case 'ビ':
            case 'ブ':
            case 'ベ':
            case 'ボ':
              builder.append('b');
              break main;
            case 'ハ':
            case 'ヒ':
            case 'ヘ':
            case 'ホ':
              builder.append('h');
              break main;
            case 'フ':
              builder.append('f');
              break main;
            case 'ヴ':
              builder.append('v');
              break main;

Trade-offs

  • Explicit and allocation-free, and it touches only rows that occur in loanwords.
  • It leaves the rare n, m, y, r, w rows and ヷ-ヺ as today: the ッ still vanishes there (these sequences are phonotactically rare, and the checks advised against claiming them).
  • Changes indexed romaji terms of these words: existing romaji fields need reindexing.

What the checks validated

  • Whole :lucene:analysis:kuromoji:test task plus the paste-ready test, gradle, seed DEADBEEF, scratch worktree at a9779ef3c193: assertions on: 31 classes, 198 tests, 0 failures, 0 errors, 7 skipped; assertions off: 31 classes, 198 tests, 0 failures, 0 errors, 7 skipped. The 7 skips are pre-existing module tests (TestExtendedMode 1, TestFactories 1, TestJapaneseNumberFilter 2, TestJapaneseTokenizer 3); the run includes TestToStringUtil (3), TestJapaneseReadingFormFilter (8, with testRandomData) and TestJapaneseReadingFormFilterFactory (2).
  • ecjLintMain, ecjLintTest and checkGoogleJavaFormat of the module with fix B and the test: pass.
  • Earlier, in the hunt: the finder's JUnit passes with this variant shadowed in.

Double the first consonant of whatever follows (fix A) (validated by the checks)

In case 'ッ', romanize the next up to 3 characters into a scratch StringBuilder and append its first letter if it is a consonant (t before ch). It replaces the four existing rows (+11/-28) and gives the same k/s/t/p results.

          // the sokuon doubles the first consonant of the following syllable (ch becomes tch)
          if (i < len - 1) {
            StringBuilder next = new StringBuilder();
            getRomanization(next, s.subSequence(i + 1, Math.min(len, i + 4)));
            if (next.length() > 1 && next.charAt(0) == 'c' && next.charAt(1) == 'h') {
              builder.append('t');
            } else if (next.length() > 0
                && next.charAt(0) >= 'a'
                && next.charAt(0) <= 'z'
                && "aiueo".indexOf(next.charAt(0)) < 0) {
              builder.append(next.charAt(0));
            }
          }

Trade-offs

  • With less code, covers every following sequence whose romanization starts with an ASCII consonant, including the rare n/m/y/r/w rows; for a trailing ッ, ッー and ッ before the unconverted small ヵ/ヶ it still writes nothing for the ッ.
  • Allocates one StringBuilder and recurses on up to 3 characters per ッ: at 4M characters the security screen measured about 101 ns per character on all-ッ input versus 3.9 stock, and 51 versus 10.7 on alternating ッグ input (assertions on, best of 3 on a shared host; the numbers are noisy, and there was no superlinear growth up to 4M characters).
  • Its extra outputs are debatable: ッッカ becomes kkka, ッーカ becomes kka. It also doubles non-katakana text after a ッ in a direct call (ッb → bb; stock and fix B b), which the katakana-only differential does not cover.
  • Changes indexed romaji terms too.

What the checks validated

  • Whole :lucene:analysis:kuromoji:test task plus the paste-ready test, gradle, assertions on: 31 classes, 198 tests, 0 failures, 0 errors, 7 skipped.
  • ecjLintMain, ecjLintTest and checkGoogleJavaFormat of the module with fix A and the test: pass.
  • Earlier, in the hunt and the registration check: the finder's JUnit, the independent reproduction's JUnits (6/6, assertions off) and the 11 upstream TestToStringUtil + TestJapaneseReadingFormFilter tests pass with it shadowed in; the finder's exhaustive check then reports no sokuon mismatch except ッヵ/ッヶ (the separate small ヵ/ヶ pass-through).

Downstream note

Search servers that offer this filter's useRomaji option through the factory (its javadoc shows the Solr schema form) run the same code, so their romaji fields should drop the same ッ (inferred from the code; not run on those servers).

How it was found

Semi-formal. A bounded-exhaustive property check ran the stock romanizer on all 761,943 katakana strings of length 1 to 3 and checked the Hepburn rule "ッ doubles the next syllable's first consonant" for every syllable; it flagged the rows other than k, s, t and p. A public-API JUnit then reproduced the common loanword cases end to end.

The method in full135 words
  • Unit. A hunt unit that checked kuromoji and nori token filters against their javadoc, with a bounded-exhaustive harness over the stock kuromoji jar.
  • Section C of the harness. All 761,943 strings of length 1-3 over U+30A1..U+30FA plus ー were romanized twice (determinism: 0 mismatches) and checked for romaji-only output; then, for every one- or two-kana syllable, ッ + syllable was checked against "double the syllable's first consonant, ch → tch". 145 of 200 consonant-initial syllables lost the ッ. That count includes rare sequences (ッ before n, m, r, y, w), so it is not the headline.
  • End to end. The realistic subset, loanwords with ッ before g, z/j, d, b and h/f, was then reproduced through JapaneseTokenizer + JapaneseReadingFormFilter(useRomaji=true) with a public-API JUnit, and checked on releases back to 3.6.0 with a version-independent probe.

Prior reports, history and checks

Prior reports

No prior report found (searched 2026-10-26: 115 tool queries, 61 manual GitHub, JIRA, list and web searches, all 407 open PRs listed, the file lists of the 17 analysis or Japanese ones read). Closest: #5499 (メ → mi, fixed 2012), #12885 (OOV hiragana), and a downstream plugin that replaced this romanizer in 2018 without naming the sokuon.

Searches5 searches

No report of this defect was found. The upstream items about this method are different defects, and the closest downstream item criticized the romaji format for other reasons.

What was searched, when, and what turned up
WhereWhenHowFound
GitHub issues, PRs and commits (apache/lucene), downstream repositories, mailing lists (dev@, java-user@, issues@, commits@), JIRA, web2026-10-26the scripted prior-report search, 115 queries in two runs, plus the finder's 45-query run280 distinct hits, 1 high and 16 medium, none about the sokuon
GitHub, JIRA, lists and web, by hand2026-10-2661 searches, e.g. getRomanization, kuromoji sokuon, kuromoji romaji ッ, useRomaji, hepburn; JIRA LUCENE and SOLR romaji / useRomaji / hepburn / sokuon / romanization#5499 (LUCENE-4433, メ → mi, fixed 2012), #12885 (OOV hiragana, merged 2024), SOLR-3906 (OOV half-width kana), #11140 / #297 (the separate completion romanizer), #4988 (the filter's introduction): all different
Open PRs2026-10-26all 407 open apache/lucene PRs listed; the changed files of the 17 labelled analysis or naming kuromoji/Japanese readnone of the 17 changes the ッ handling (#16911 deletes all of branch_10x; not a fix)
Downstream2026-10-26Japanese analysis plugins and other Lucene-based projectselasticsearch-sudachi #29 / PR #32 (2018) called this romaji format odd (macrons, ロ゜ → lo) and replaced it with a romanizer that doubles every consonant row except the ラ row (k, s, t, n, h, m, y, w, g, z, d, b, p); it never names the sokuon and nothing was reported upstream
Git history2026-10-26log --follow, -S "case 'ッ':", --grep romaji / hepburn / sokuonthe ッ block is unchanged since the original contribution

History

In the method since it was written (cd372bdc838, LUCENE-3305, 2012-01-12); the ッ block never changed. First released in 3.6.0; fails on all 12 releases run from 3.6.0 to 10.5.2 and on main.

Versions tested5 rows

Introduced with the method in the original kuromoji contribution: cd372bdc838 "LUCENE-3305: add Kuromoji Japanese morphological analyzer" (2012-01-12; issue now #4378, no PR) and its branch_3x copy 8c9485fb664 (2012-01-14). git log -S on the block's lines, git log -L and git blame -C -C -M name only that commit. The output reached indexed terms with c3ddb9dc672 "Added KuromojiReadingFormFilter (LUCENE-3915)" (#4988, 2012-03-25). The only later fix to the table, fc84112ece8 "add hepburn test and fix some corner cases" (2012-05-17), left ッ alone. 3.5.0 has no kuromoji module. Not fixed on upstream main a95c754ee00; none of the 17 open analysis or Japanese PRs whose file lists were read changes it.

Versions and branches executed, with results
Version or commitVerdictResult
main a9779ef3c193 affectedpaste-ready test fails 2 of 3 (gradle and runner, assertions on and off); the finder's, the independent reproduction's and the intended-behaviour check's tests fail too
10.5.2, 10.0.0 affectedpaste-ready test fails with the same messages (published jars, assertions on)
3.6.0, 3.6.2, 4.0.0, 4.10.4, 5.5.5, 6.6.6, 7.7.3, 8.11.4, 9.0.0, 9.12.3, 10.0.0, 10.5.2 affectedversion-independent probe: 7 of 7 loanwords lose the ッ, the filter gives bagu for バッグ, the probe's controls pass (ッ before the t and p rows: キット, マッチ, キップ; plus バグ)
3.5.0 no APIno kuromoji module (the probe reports the API absent)
main + fix B, main + fix A not affectedpaste-ready test passes; the whole kuromoji test task passes

Independent checks

The finder, an independent reproducer, an intended-behaviour checker, the adjudication and a credibility audit all ran code and said real. Reviews by another model family (OpenAI GPT-5.6) could not run code; on reading they agree with the root cause. This doc's gradle runs confirm the repro and both fixes.

Checks and verdicts287 words

Every verdict except the cross-model reviews' rests on executed runs on main a9779ef3c193. The audit kept the bug and lowered its likelihood to rare (opt-in, already lossy romanizer).

Each independent check and its verdict
CheckVerdictWhat it found
finder candidateBounded-exhaustive harness (section C); JUnit fails 2/3 on main (assertions on and off), 10.5.2 and 10.0.0; the fix passes the repro and 11 upstream tests.
independent reproducer realOwn JUnits from the claim only: 4/6 fail on main (assertions on and off) and 10.5.2, 2/3 on 9.12.3; 9 of 9 loanwords wrong; k/s/t/p controls pass. Rated the intended-behaviour case moderate.
intended-behaviour checker realBuilt the strongest 'approximate by design' case and rejected it; factory-path probe gets 10 of 10 loanwords wrong on main and 10.5.2 while native words come out right.
refute review (OpenAI GPT-5.6) not real, low (procedural)Could not execute; agrees with the root cause and that the inputs are legal; its own static case (ビッグ → bigu vs キック → kikku) supports real. It caught an overbroad first title, since narrowed.
intended review (OpenAI GPT-5.6) not real, low (procedural)Could not execute; on substance leans real (stale 'debugging only' javadoc; the completion table doubles these consonants). A separate run re-ran the finder's JUnit: 2/3 fail.
adjudication confirmed, highRe-ran the reproducer's JUnits with a new seed on main: 4/6 fail, controls pass.
credibility audit kept, likelihood rareRe-ran the finder's JUnit (assertions on and off): 2/3 fail; lowered the likelihood because romaji mode is opt-in and the romanizer is lossy by design.
security screen noneNo superlinear growth in cost up to 4M characters, CheckIndex clean, term queries match the analyzer's output exactly.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,300 words

Finding record

LFV-605, snapshot of 2026-11-06

Registry title
Kuromoji's romaji reading (ToStringUtil.getRomanization, used by JapaneseReadingFormFilter with useRomaji=true) writes the small tsu ッ as a doubled consonant only before the k, s, t and p rows and silently drops it before any other kana, so loanwords with ッ before g, z/j, d, b or h/f lose it: バッグ 'bag' and バグ 'bug' both become 'bagu', ベッド becomes 'bedo'
Area
analysis / kuromoji romanization (ToStringUtil.getRomanization; JapaneseReadingFormFilter romaji mode)
Priority
Low, rank 506: Kuromoji romaji mode silently drops the small tsu before voiced consonants, so loanword pairs such as bag and bug collide, which is deterministic but sits in an opt-in, rarely used and already lossy romanizer, so it heads the CJK rare-input rows after the factory rows and just above LFV-459.
Impact class
silent wrong results
Impact note
Applications that index or query Japanese text through JapaneseReadingFormFilter in romaji mode (useRomaji=true or the factory parameter of that name; the default is katakana) or call ToStringUtil.getRomanization directly, in every release from 3.6.0 to 10.5.2 and main: every occurrence of a common loanword with ッ before g, z/j, d, b or h/f (バッグ, ベッド, ビッグ, キッズ, グッズ, スタッフ, ブリッジ, バッハ) is indexed without the doubled consonant, so it shares a term with a different word (バッグ/バグ -> 'bagu', キッズ/キズ -> 'kizu') and a query typed in Hepburn or keyboard romaji ('baggu', 'beddo') does not match it; no scale is needed, there is no error and no index damage, and a fix changes indexed romaji terms, so such fields would need reindexing.
Summary
You analyze Japanese text with JapaneseTokenizer and JapaneseReadingFormFilter(useRomaji=true), whose romanizer is documented as 'Romanize katakana with modified hepburn', and expect the small tsu ッ to double the next consonant, as the method already does for キット -> 'kitto' and マッチ -> 'matchi'. Instead the 'ッ' case appends a consonant only for the カ, サ, タ and パ rows and appends nothing otherwise, so バッグ 'bag' becomes 'bagu', the same term as バグ 'bug', ベッド becomes 'bedo' and スタッフ becomes 'sutafu'. Distinct words then match each other, and a romaji query 'baggu' finds no document that contains バッグ.
Introduced
cd372bdc83836148a7ae2e2958407ff0a37b7fc2 (trunk, svn r1230748, 2012-01-12, committed by GitHub rmuir: LUCENE-3305 'add Kuromoji Japanese morphological analyzer', now apache/lucene#4378, no PR) and its branch_3x copy 8c9485fb66427e54ff0dac9708fdcbe72ba62e94 (svn r1231536, 2012-01-14): they created ToStringUtil.getRomanization with the 'ッ' switch that appends only k/s/t/p [read: git log -L on the block, git log -S "case 'ッ':" and git blame -C -C -M on main all name only cd372bdc838]. The output reached indexed terms with c3ddb9dc67242b9dacbea753e28a27787d6d3305 (LUCENE-3915 / #4988, 2012-03-25, the reading form filter with useRomaji; branch_3x dd51f252c55) [read]. Not a regression: the method was born with the gap [exec at release granularity: 3.5.0 has no API, 3.6.0 has the bug] (2012-01-12 (romaji filter path added 2012-03-25))
First affected version
3.6.0 [exec], the first release with kuromoji (contrib lucene-kuromoji; there the useRomaji factory parameter is in the Solr factory, Lucene's own factory exists from 4.0.0 [read]); 3.5.0 has no kuromoji module, so no release is unaffected in the strict sense [exec: API absent]; on the trunk line 4.0.0-alpha [read] and 4.0.0 [exec]
Affected versions
every release from 3.6.0 through 10.5.2 (the last release before 10.6.0) and main a9779ef3c193. [exec, the follow-up's version-independent main() probe RomajiSokuonHistoryProbe, JDK 25] 7 of 7 loanwords lose the ッ, the filter turns バッグ into 'bagu' and the k/s/t/p controls pass on 3.6.0, 3.6.2, 4.0.0, 4.10.4, 5.5.5, 6.6.6, 7.7.3, 8.11.4, 9.0.0, 9.12.3, 10.0.0, 10.5.2 and main (-ea and -da); with the candidate fix shadowed in, the probe passes on main. [exec, JUnits] the finder's JUnit fails 2/3 on main (-ea, -da), 10.5.2 and 10.0.0; the independent reproduction's JUnits fail 4/6 on main (-ea, -da) and 10.5.2 and 2/3 (filter test only) on 9.12.3. [read] releases in between were not executed; the 'ッ' block is unchanged since cd372bdc838 (git log -L) and tag containment covers them. Not fixed upstream: ToStringUtil.java is identical on ea57360912ed and a9779ef3c193, no commit after a9779ef touches lucene/analysis/kuromoji (upstream head a95c754ee00, 2026-10-26), and no open PR changes the ッ handling.
Status
documented (brief, checks and a per-finding doc exist)
Verification
Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its low-confidence “not real” is procedural), an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by
semi-formal: exhaustive check
Upstream and prior reports
none found (2026-10-26, read-only; the prior-report record, verdict new plus the finder's run: 115 scripted prior-report search queries over apache/lucene issues, PRs and commits, downstream repositories, the Lucene mailing lists, the JIRA archive and the web, 61 manual GitHub, JIRA, list and web searches, the finder's 45 scripted prior-report search queries, and the changed files of all 407 open apache/lucene PRs): no issue, PR, JIRA entry, list thread or downstream report says the romanizer drops ッ. Related, different defects: #5499 (LUCENE-4433, メ romanized as 'mi', fixed 2012 by fc84112ece8, which left the ッ switch alone), #12885 (out-of-vocabulary hiragana not romanized, merged 2024) and its java-user@ thread, SOLR-3906 (OOV half-width kana, 2012), #11140 / #297 (LUCENE-10102, the separate KatakanaRomanizer; a reviewer asked for 'jji' for ジャッジ there). Closest downstream item: WorksApplications/elasticsearch-sudachi#29 (2018) called ToStringUtil's romaji format odd (macrons, ロ゜ -> lo) and PR #32 replaced it with a romanizer that doubles every row; neither mentions the sokuon (issue #29 re-read via the GitHub API on 2026-10-26) and nothing was reported upstream. Not fixed on upstream main a95c754ee00. Gaps: one Stack Overflow page and one blog could not be read (bot check); GitHub search does not stem.
Lucene commit tested
main a9779ef3c193 (gradle in a scratch worktree with assertions on and off, and the standalone JUnit runner); the release jars 10.5.2 and 10.0.0 (this doc's paste-ready test); 3.6.0, 3.6.2, 4.0.0, 4.10.4, 5.5.5, 6.6.6, 7.7.3, 8.11.4, 9.0.0, 9.12.3, 10.0.0 and 10.5.2 (the history step's version-independent probe); fix B and fix A on main (gradle, the whole kuromoji test task).

Lucene and upstream sources

Finding record
  • lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/ToStringUtil.java:24 (at a9779ef3c193) (class javadoc 'used only for debugging'), :240 and :251 (javadoc 'Romanize katakana with modified hepburn'), :252-253 (TODO 'now that this is used by readingsfilter and not just for debugging'), :264-295 (case 'ッ': inner switch appends 'k', 's', 't', 'p' for the カ/サ/タ/パ rows, then a bare 'break' that appends nothing)
  • lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilter.java:27-28 (at a9779ef3c193) (javadoc 'reading of a token in either katakana or romaji form'), :62-99 (romaji mode calls ToStringUtil.getRomanization on the reading or the OOV term text)
  • lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilterFactory.java:44-50 (at a9779ef3c193) (useRomaji, default false)
The finding in 30 seconds
  • lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/ToStringUtil.java:240, (at a9779ef3c193) :251-295 (javadoc; lookahead; case 'ッ')
  • lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilter.java:26-29, (at a9779ef3c193) :62-99 (javadoc; romaji mode)
Who is affected
  • lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilterFactory.java:26-34, (at a9779ef3c193) :44-50 (Solr schema example; useRomaji default false)
  • lucene/analysis/kuromoji/src/java/module-info.java:25 (at a9779ef3c193) (exports ja.dict)
Why it looks unintended
  • lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/ToStringUtil.java:24, (at a9779ef3c193) :240, :251-253, :264-295, :768-800, :1089-1090
  • lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/completion/romaji_map.txt:3, (at a9779ef3c193) :4-5, :178, :193, :194, :209, :212, :216, :218, :224, :226
  • lucene/analysis/kuromoji/src/test/org/apache/lucene/analysis/ja/dict/TestToStringUtil.java:28-37 (at a9779ef3c193) (majan; Hepburn page; 'isnt even thorough')
  • lucene/analysis/kuromoji/src/test/org/apache/lucene/analysis/ja/TestJapaneseReadingFormFilter.java:96-101 (at a9779ef3c193) (robato)
  • lucene/CHANGES.txt:3042 (at a9779ef3c193) (GITHUB#12885, 9.10.0 Bug Fixes), :14729 (SOLR-3906, Bug Fixes)

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

About the diagrams

Made in parallel with this doc: every code row is the source at a9779ef3c193 and every romaji string, term and hit list is read from an executed probe on the stock main jars and with the hunt's fix shadowed in (assertions on and off). Its fix step is based on the generic variant (fix A); fix B, the recommended option, gives the same romaji for every word drawn.

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.