JapaneseIterationMarkCharFilter turns なゝめ into などめ: the unvoiced iteration mark after な..の, っ or ぱ..ぺ yields a different kana

JapaneseIterationMarkCharFilter (kuromoji, opt-in, SPI name japaneseIterationMark) expands the Japanese iteration marks: ゝ (ヽ in katakana) repeats the kana before it without a voicing mark, ゞ (ヾ) repeats it voiced. For 10 hiragana that have no voiced form, っ, な に ぬ ね の and ぱ ぴ ぷ ぺ, and their 10 katakana forms, the unvoiced mark becomes the code point before the kana instead: なゝめ → などめ instead of ななめ, のゝしる → のねしる, ぱゝ → ぱば, ナヽメ → ナドメ. The voicing table stores a kana without a voiced form as a self-mapping, exactly like a voiced kana, and isHiraganaDakuten treats every self-mapping as voiced. Words in older orthography are indexed misspelled and searches for the modern spelling miss them, with no error, in every release from 4.0.0-BETA (2012) to 10.6.0 (10.6.0 by reading) and on main. A two-line fix checks the code point before the kana instead.

• Iteration marks (odoriji): ゝ (U+309D, hiragana) and ヽ (U+30FD, katakana) repeat the kana before them; older texts write ななめ ("diagonal") as なゝめ and ののしる ("to revile") as のゝしる. The voiced marks ゞ/ヾ repeat the kana with a voicing mark (みすゞ → みすず).
• Voicing mark (dakuten): か → が, し → じ, は → ば. The unvoiced mark repeats a voiced kana without it: じゝ → じし. Kana of the na-row (な に ぬ ね の), the small っ and the p-sounds ぱ ぴ ぷ ぺ (which carry a handakuten) have no voiced form.
• JapaneseIterationMarkCharFilter: opt-in, in kuromoji since 4.0.0; SPI name japaneseIterationMark. Its javadoc: "Normalizes Japanese horizontal iteration marks (odoriji) to their expanded form" (JapaneseIterationMarkCharFilter.java:26). Kana normalization is on by default (:48).

1. The voicing table h2d has 50 entries for か (U+304B) .. ぼ (U+307C) (:82-131; katakana k2d is derived from it, :133-138). A kana maps to its voiced form (か => が), a voiced kana to itself (が => が), and a kana without a voiced form also to itself, at index 24 (っ, :106), 31-35 (な..の, :113-117) and 38, 41, 44, 47 (ぱ ぴ ぷ ぺ, :120, :123, :126, :129). lookupHiraganaDakuten documents a self-mapping as "c itself if no dakuten variant exists" (:385).
2. isHiraganaDakuten(c) is inside(c, h2d, '\u304b') && c == lookupHiraganaDakuten(c) (:408; katakana :418). That is true for が and equally for な.
3. For ゝ, normalizedHiragana returns isHiraganaDakuten(c) ? (char) (c - 1) : c (:303; katakana :321). For が that removes the dakuten (U+304C - 1 = か); for な (U+306A) it gives ど (U+3069), for の ね, for っ ぢ, for ぱ ば.
4. The voiced mark returns lookupHiraganaDakuten(c) directly (:305), so なゞ → なな is right. ぽゝ → ぽぽ is right only because ぽ (U+307D) lies one code point past the table.

20 of 20 pairs wrong on main with assertions on and off. A document indexed as なゝめ (keyword tokenizer): TermQuery ななめ → 0 hits, などめ → 1 hit. With JapaneseTokenizer behind the filter, なゝめに切る becomes [など, め, に, 切る] while ななめに切る gives [ななめ, に, 切る]; のゝしる声 becomes [の, ね, しる, 声]. In the pair and sequence sections of the exhaustive check this is the only discrepancy: over all strings of length up to 5 on a 14-symbol alphabet (of the affected kana it contains only な) under all 4 flag combinations, 0 mismatches in the strings without な (1,608,932 checks), 24,985 in those with な (707,844 checks), 0 with the fix. Two separate gaps remain with or without the fix (see How it was found).

• Who. Applications that add JapaneseIterationMarkCharFilter to an analyzer: the class, the factory (SPI name japaneseIterationMark, for example in CustomAnalyzer), or a downstream search server's iteration-mark char filter, which constructs this class with Lucene's default flags (read from their source; not run there). Opt-in: JapaneseAnalyzer installs only CJKWidthCharFilter. It reproduces with the default flags.
• What triggers it. Text that writes one of the 10 kana (or its katakana form) before the unvoiced mark: older orthography and names such as なゝめ (ななめ), なゝつ (ななつ), のゝしる (ののしる), ナヽ. っゝ is not attested in real text, and handakuten kana before ゝ are uncommon. No frequency was measured. Deterministic: no scale, seed or assertion dependence.
• What breaks. The filter exists to make the marked and the expanded spelling match; for these words they do not. A document written なゝめ is not found by ななめ and is found by などめ. Query text analyzed by the same analyzer goes through the same filter (for multi-term queries the factory's normalize() builds it too), so a query typed with the mark matches the marked documents but not those written ななめ.
• Silent. No exception, no log line. The output keeps the input length, so offsets stay right.
• After a fix. Documents indexed with the filter keep the wrong terms until they are reindexed; query-side analysis is right as soon as the fix is deployed.
• Workaround today. A MappingCharFilter in front of this filter that maps the 20 pairs (なゝ => なな, ナヽ => ナナ, ...) fixes the single mark (なゝめ → ななめ), but not a span of marks: なのゝゝ (expected なのなの) gives なのどね on stock main and なののね with the workaround.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<な[な]め> but was:<な[ど]め>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.ja.TestIterationMarkUnvoicedKana.testUnvoicedMarkRepeatsNa(TestIterationMarkUnvoicedKana.java:32)

• The javadoc contradicts the predicate. lookupHiraganaDakuten returns "c itself if no dakuten variant exists" (JapaneseIterationMarkCharFilter.java:385); isHiraganaDakuten (javadoc "true if c is a hiragana dakuten", :405) reads that same self-mapping as voiced (:408). な, の, っ and ぱ are not voiced kana.
• The contract. "Normalizes Japanese horizontal iteration marks (odoriji) to their expanded form" (:26). The expanded form of なゝ repeats な; など is a different word.
• The documented fallback. For an illegal sequence the filter "emits the illegal source character as-is" (:28-30), which would give なな again. No documented reading produces など, のね or ぱば.
• One branch away it is right. The voiced mark uses the lookup directly: なゞ → なな.
• Internally inconsistent. ぽゝ → ぽぽ is asserted upstream (TestJapaneseIterationMarkCharFilter.java:135), yet ぱゝ → ぱば, only because ぽ lies just outside the table and ぱ inside it.
• Never tested, never discussed. The coverage tests (:189-215) list the か..ぼ kana that have a voiced form, each followed by its voiced form, and skip the 10 affected kana; no Lucene test has an affected kana before ゝ/ヽ. The introducing issue (LUCENE-4201, now #5273) discusses illegal sequences, the full stop, surrogates, offsets and random tests, never these kana. CHANGES has only the "Added" entry.

The strongest case (the intended-behaviour checker and the cross-model review): the author wrote the self-mapping entries on purpose, kana iteration marks are rare in modern text, the class already lists limitations (illegal sequences, the full stop, no vertical marks) and carries @IgnoreRandomChains, so odd inputs could be called unsupported; and handakuten kana with ゝ have no settled rule.

Why it fails: the self-mapping is the class's own documented sentinel for "no variant", and the predicate misreads it; the listed limitations do not cover these kana, and the documented illegal-sequence fallback would give なな anyway; @IgnoreRandomChains and the open #11394 concern offsets and an index-out-of-bounds exception, a different symptom and code path; and an unvoiced mark that outputs a voiced kana (ぱば) or a kana of another row (のね) is wrong under any rule. Rarity lowers the priority; it does not make the output intended. Not claimed: that ぱぱ, rather than ぱは, is the required answer for handakuten kana; the docs only rule out ぱば.

The recommended option was validated in a scratch worktree at a9779ef3c193: gradle, the whole :lucene:analysis:kuromoji:test task plus the paste-ready test, assertions on and off, and the module's lint and format checks; and with the standalone runner (the fixed class shadowed) on the paste-ready test and the probe. The other options were not run.

Two lines per predicate, with a short comment, plus a test method testKanaWithoutVoicedForm in TestJapaneseIterationMarkCharFilter (なゝめ, のゝしる, the 10 hiragana and the 10 katakana pairs). Voiced kana stay voiced (が: か maps to が), unvoiced kana stay unvoiced (か: the code point before it is outside the table), and the 10 kana become unvoiced (な: ど maps to ど, not な), so ゝ repeats them: なな, っっ, ぱぱ.

   private boolean isHiraganaDakuten(char c) {
-    return inside(c, h2d, '\u304b') && c == lookupHiraganaDakuten(c);
+    char base = (char) (c - 1);
+    return inside(base, h2d, '\u304b') && lookupHiraganaDakuten(base) == c;
   }
   private boolean isKatakanaDakuten(char c) {
-    return inside(c, k2d, '\u30ab') && c == lookupKatakanaDakuten(c);
+    char base = (char) (c - 1);
+    return inside(base, k2d, '\u30ab') && lookupKatakanaDakuten(base) == c;
   }

Trade-offs

• Reuses the table; one lookup per unvoiced mark, as today. Only the unvoiced-mark branch changes; the voiced mark and the span logic are untouched.
• For handakuten kana it gives ぱぱ, consistent with the existing ぽゝ → ぽぽ test; a maintainer may prefer ぱは. Either way ぱば is gone.
• Leaves the separate gap outside the table (ゔゝ, ヴヽ, ヷヽ..ヺヽ repeated unchanged; うゞ, ウヾ, ワヾ..ヲヾ not voiced; 12 pairs) as it is.
• Existing indexes keep the wrong terms until reindexed.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the paste-ready test fails 3 of 4 on stock main with assertions on and off; the new upstream test method alone fails on stock main (term 0 expected:<な[な]め> but was:<な[ど]め>; the other 14 pass).
• With the fix (code and test method) plus the paste-ready test, the whole :lucene:analysis:kuromoji:test task passes: 31 classes, 200 tests, 0 failures (7 skipped, all pre-existing nightly or large-data tests), with assertions on and off. That includes TestJapaneseIterationMarkCharFilter (15, with its random-string tests) and TestJapaneseIterationMarkCharFilterFactory (5).
• ecjLintMain, ecjLintTest and checkGoogleJavaFormat of the module pass.
• With the fixed class shadowed, the paste-ready test passes 4 of 4 (assertions on and off) and the probe prints no wrong line. Earlier, the finder's sketch of the same predicate cleared the exhaustive check (20 → 0 pair mismatches, 24,985 → 0 sequence mismatches) and the 14 upstream tests.

Treat c as voiced when its NFD form ends with U+3099, and use the first character as the unvoiced kana (java.text.Normalizer).

Trade-offs

Independent of the table, and it would also cover the voiced kana outside it (ゔ, ヴ, ヷ..ヺ). But it adds a Normalizer call per unvoiced mark and changes output for those out-of-table kana (ゔゝ would become ゔう), a separate gap that needs its own decision. Not run.

For example a separate flag array, or a sentinel value that lookup turns back into c.

Trade-offs

Makes the table self-describing, at the cost of a larger diff that also touches the voiced-mark path. Not run.

Map the 20 pairs (なゝ => なな, ナヽ => ナナ, ...) before this filter.

Trade-offs

Fixes the single mark (なゝめ → ななめ, ぱゝ → ぱぱ) but not a span of marks: なのゝゝ gives なののね instead of なのなの.

What the checks validated

Run in the probe on stock main: right for the 6 single-mark words, wrong for the span.

• Method: bounded-exhaustive translation validation (the hunt's semi-formal tier). A harness ran the stock main jar against a reference expander written from the class javadoc and the upstream tests, with Unicode canonical decomposition (U+3099) as the independent voicing oracle.
• Pair section. Every character from U+3041 to U+30FF except the four kana marks (and a kanji, a letter and a question mark) followed by each of the five iteration marks, except the 60 pairs that need a voiced form outside the table: 890 checks (178 sources x 5 marks), 20 mismatches (the 20 pairs), 0 with the fix. The 60 out-of-table pairs were counted separately (the gap noted below).
• Sequence section. All strings of length up to 5 over a 14-symbol alphabet (か が な ぽ カ ガ 時, the five marks, 。 and a supplementary character) under all 4 flag combinations: 1,608,932 checks without な, 0 mismatches; 707,844 with な, 24,985 mismatches, 0 with the fix. So the per-kana rule is the only discrepancy in that alphabet. Non-vacuity: a reference without the full-stop barrier gives 98,884 mismatches.
• Reduction. A public-API JUnit (keyword analyzer behind the filter) on the stock main jars, 2026-10-26; this doc's paste-ready test adds the search miss.
• Also seen, not part of this bug: the table stops at か..ぼ, so 12 pairs outside it (ゔゝ, ヴヽ, ヷヽ..ヺヽ, うゞ, ウヾ, ワヾ..ヲヾ) are not voiced or unvoiced; read(cbuf, off, 0) returns -1 instead of 0. Both are the same on stock main and with the fix.