ICUTokenizer with cjkAsWords=false drops the Japanese iteration mark 々: 人々 is indexed as 人

ICUTokenizer has a switch, cjkAsWords (default true). With cjkAsWords=false (new DefaultICUTokenizerConfig(false, true), or the icu factory with cjkAsWords="false"), CJK text is "segmented according to UAX#29 defaults" by Lucene's own break rules, Default.brk, instead of ICU's dictionary; the module's own tests use this mode, also in front of CJKBigramFilter. In this mode every U+3005 IDEOGRAPHIC ITERATION MARK vanishes: 人々 ("people") gives [人], 時々 ("sometimes") [時], 様々な人々 [様, な, 人], 佐々木 [佐, 木], where StandardTokenizer and Lucene 4.6.1 give [人, 々] and the default mode [人々]. The cause is Default.rbbi: it removes every Han-script character from the letter class and gives a status only to Ideographic characters; 々 is Han but not Ideographic, so its segment gets rule status 0, and ICUTokenizer skips status-0 segments like punctuation. The other two Han-script letters, U+303B and U+16FE3, are dropped the same way. Nothing throws; 々 is simply not in the index, and with CJKBigramFilter 人々 is indexed as the term 人. It came with LUCENE-4381 (4.7.0) and is in every release up to 10.5.2 and on main; a one-line rule change, with Default.brk regenerated, fixes it.

• cjkAsWords (default true): with true, Han, Hiragana and Katakana text goes to ICU's own word iterator, which uses a dictionary and keeps 人々 as one word. With false, CJK text is "segmented according to UAX#29 defaults" (DefaultICUTokenizerConfig.java:94-96) by Lucene's rule file Default.rbbi, compiled to Default.brk. That is the mode for CJKBigramFilter users: the module's own TestWithCJKBigramFilter uses it.
• Rule status: ICU break rules give each piece of text a number. In Default.rbbi, letters get 200, kana 300 and ideographs 400 (Default.rbbi:125-131); pieces that no status rule matches get 0, ICU's "none" status for spaces and punctuation.
• 々 (U+3005): the Japanese iteration mark; it repeats the kanji before it (人々 = 人 + 人, "people"). It is Script=Han, Ideographic=No and Word_Break=ALetter, a letter under UAX #29.

1. Default.rbbi defines $Han = [:Han:] (:54), puts it into $dictionaryCJK (:88-89) and removes $dictionaryCJK from the letter class: $ALetterPlus = [$ALetter-$dictionaryCJK ...] (:95). So no Han-script character gets the letter status 200.
2. The only status rule left for Han is $Ideographic $ExFm* {400}; (:131): Lucene's tailoring removed ICU's rule that chains Han and kana (header :23). 人 is Ideographic and gets 400. 々 is not, so its piece gets status 0. The Han exclusion and this status rule are ICU's own word.txt text; in ICU, the final rule $KanaKanji $KanaKanji {400} and the dictionary keep a 々 after a kanji inside a status-400 word, and Lucene's mode removes both.
3. ICUTokenizer.incrementTokenBuffer skips status-0 pieces (ICUTokenizer.java:209-214), the loop that skips spaces and punctuation. 々 is dropped exactly like a comma: in 486 of 486 tested contexts the tokens equal those of the same text with 、 in its place.
4. The boundaries are right (人 | 々, as UAX #29 says); only the status is wrong.

人々 -> [人]; 時々 -> [時]; 様々な人々 -> [様, な, 人]; 佐々木さん -> [佐, 木, さ, ん]; 人〻 (U+303B) and 人𖿣 (U+16FE3) -> [人]. StandardTokenizer gives [人, 々], the default mode [人々]. With CJKBigramFilter, 人々 gives the unigram [人] instead of the bigram [人々], and 様々な人々 gives [様, な人] instead of [様々, 々な, な人, 人々]. Indexing the documents 人々 and 人 through that chain leaves one term, 人 (df=2), for both documents: TermQuery 人々 finds 0 documents, 人 finds 2 and 々 finds 0. With the default mode or the recommended fix the 人々 document gets the bigram 人々 (1 and 1); with <ALPHANUM> typing, as StandardTokenizer gives, it gets the term 々 (人々 0, 人 2, 々 1).

• Who is exposed. Users of the opt-in analysis-icu module who choose cjkAsWords=false: new ICUTokenizer(new DefaultICUTokenizerConfig(false, ...)), or ICUTokenizerFactory (SPI name icu) with cjkAsWords="false" (the factory default is true, ICUTokenizerFactory.java:96). A maintainer described this as a supported setup on #13239: "the user can configure false here and use CJKBigramFilter if they prefer an n-gram approach". ICUTokenizer and DefaultICUTokenizerConfig are @lucene.experimental (the factory is not). All of this is with the stock rules: a factory rulefiles entry for the Han script replaces Default.brk for Han text (ICUTokenizerFactory.java:54-66, :86-95, :120-131).
• Not exposed. The default ICUTokenizer() and the factory default (cjkAsWords=true). Elasticsearch and OpenSearch hard-code cjkAsWords=true in their icu_tokenizer.
• What triggers it. Ordinary Japanese text: 々 is in common words (人々, 時々, 様々, 色々, 我々, 日々) and names (佐々木). Deterministic: every occurrence, every document, assertions on or off.
• Effect. 々 is never indexed and cannot be searched. 人々 is indexed as 人: with CJKBigramFilter, the 人々 and 人 documents get the same single term 人, so no search on the field can tell them apart; a query for 人々 analysed the same way becomes 人, so it still finds the 人々 document but also every document with an isolated 人 (lower precision). Bigrams across the mark are lost.
• Not affected. Other tokens and their offsets, index integrity (CheckIndex clean over 3,000 documents), cost (linear, the same order of growth as punctuation and no amplification; all-々 input was about 40% slower than all-、 input in the recorded run). Existing indexes keep the missing terms until reindexed.
• Scale. Every release from 4.7.0 (2014) to 10.5.2, and main.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[人, 々]> but was:<[人]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.icu.segmentation.TestICUTokenizerIterationMark.testIterationMarkKeptWithCjkAsWordsFalse(TestICUTokenizerIterationMark.java:40)

• The contract. "@param cjkAsWords true if cjk text should undergo dictionary-based segmentation, otherwise text will be segmented according to UAX#29 defaults. If this is true, all Han+Hiragana+Katakana words will be tagged as IDEOGRAPHIC." (DefaultICUTokenizerConfig.java:94-96). The class javadoc: "Generally tokenizes Unicode text according to UAX#29 (BreakIterator.getWordInstance(ULocale.ROOT))" with a dictionary as the only CJK tailoring (:30-36); next to Default.brk: "the same as ROOT, except no dictionary segmentation for cjk" (:80). ICUTokenizer: "Breaks text into words according to UAX #29" (ICUTokenizer.java:30-32). Under UAX #29, U+3005 is Word_Break=ALetter, a letter. UAX #29 sets the boundary and the letter class, not ICU's rule statuses or which segments a tokenizer emits; ICUTokenizer drops only status-0 segments, ICU's status for spaces and punctuation.
• The tailorings on record. The Default.rbbi header names two: "Prevent dictionary break: disable final rule that would chain han and kana" and "For kana rules, change from 400 to 300 status" (:23-24). A diff against ICU 77.1's word.txt shows exactly those two changes. Neither says Han-script letters stop being tokens.
• ICU's convention. Status 0 is ICU's "none" status, documented for spaces and most punctuation; 々 is gc=Lm, a letter.
• Every reference keeps it. StandardTokenizer ([人, 々]), ICU's root word iterator (人々 as one status-400 word), the default cjkAsWords=true ([人々]), and Lucene 4.6.1 ([人, 々]).
• History. LUCENE-4381 (4.7.0) added cjkAsWords, changed the letter class to [$ALetter-$dictionaryCJK ...], and moved the module's existing tests to DefaultICUTokenizerConfig(false), which suggests that false was meant to keep the earlier UAX #29 behaviour (the commit message does not say so). Its only CHANGES entry is "Upgrade analysis/icu to 52.1" under Build. With the ICU version held fixed, the 4.6.1 rules give 々 status 200 and the 4.7.0 rules 0.
• Maintainer on record. #13239 (2024): Default.rbbi "is used only when the user overrides defaults to disable ICU's dictionary-based CJK segmentation", quoting the "UAX#29 defaults" javadoc, with cjkAsWords=false plus CJKBigramFilter as the use case.
• Tests. TestICUTokenizer (main analyzer at :78, used by testJapanese at :436) and TestWithCJKBigramFilter (:49) run in this mode. No test in analysis/icu contains 々; the only tests that do are in kuromoji (the iteration-mark char filter tests, TestJapaneseCompletionFilter, TestJapaneseAnalyzer).
• Outside Lucene. Unicode CLDR decided in 2009 that 々 should behave as an ideograph for Japanese word break (CLDR-2009).

The intended-behaviour checker and the cross-model lenses built the strongest case for "works as designed": Lucene only removed ICU's chain rule, and status 0 is what ICU's grammar then gives; UAX #29 defines boundaries, not which pieces are words, and the boundary is right; status-0 pieces are skipped on purpose; the broad Han exclusion survived every later rule change and is ICU's own grammar text; a TODO calls cjkAsWords "super-expert stuff" (DefaultICUTokenizerConfig.java:86); the classes are @lucene.experimental; ICU root itself drops a lone 々; kuromoji's JapaneseIterationMarkCharFilter can expand 々 first.

Why it fails: the javadoc promises UAX #29 defaults and 々 is a letter there; status 0 is ICU's status for spaces and punctuation, and ICUTokenizer turns it into a deletion. Every reference keeps the character, including the 4.6.1 behaviour that the test migration suggests cjkAsWords=false was meant to keep. No javadoc, CHANGES entry, test, commit message or discussion found accepts the loss. The lone-々 point holds only for the default mode (ICU root's own behaviour), which is not claimed here; with cjkAsWords=false even a lone 々 is in scope. The char filter is a partial workaround, not a contract. Not settled by this: the recommended fix is itself a tailoring (CLDR-2009's "treat 々 as ideograph"), and a maintainer may call this a long-standing limitation of an experimental mode. Verdict: not intended (the cross-model reviews rated the works-as-designed case weak and the independent reproduction, the intended-behaviour check and the adjudication found the bug real; the later cross-model fact check of this doc rated that case moderate).

One fix was validated with gradle in a scratch worktree at a9779ef3c193, with Default.brk regenerated by the build's own task. A broader variant was validated earlier by compiling the rules at class init; a third variant was only probed. Default.brk is generated: change Default.rbbi, then run ./gradlew :lucene:analysis:icu:regenerateRbbi, which also updates the checksum file that check verifies and runs applyGoogleJavaFormat on the module's sources.

One line in Default.rbbi (:131), then regenerateRbbi. 々, 〻 and 𖿣 become <IDEOGRAPHIC> tokens, as the default mode types them ("all Han+Hiragana+Katakana words will be tagged as IDEOGRAPHIC", DefaultICUTokenizerConfig.java:95-96) and in line with CLDR-2009, so CJKBigramFilter forms 人々, 様々 and 佐々. Of the 335 Han-script code points that are not Ideographic, it changes exactly these 3 letters; the 330 radicals and 2 marks stay as on main. Boundaries do not change.

--- a/lucene/analysis/icu/src/data/uax29/Default.rbbi
+++ b/lucene/analysis/icu/src/data/uax29/Default.rbbi
@@ -128,7 +128,7 @@ $HangulSyllable {200};
 $Hebrew_Letter $ExFm* {200};
 $Katakana $ExFm* {300};             # note:  these status values override those from rule 5
 $Hiragana $ExFm* {300};             #        by virtue of being numerically larger.
-$Ideographic $ExFm* {400};          #
+[$Ideographic [$Han & $ALetter]] $ExFm* {400};  # Han letters such as U+3005 are not Ideographic
 
 #
 # rule 5

Trade-offs

• Changes analysis output for cjkAsWords=false users (new tokens 々, new bigrams): needs a CHANGES line and a reindex note.
• With cjkAsWords=true, Han-script text goes to ICU's root iterator, not Default.brk, so the default mode should not change (from the code; the cjkAsWords=true suite is @AwaitsFix and was not run).
• Not plain UAX #29: each Han letter gets its own single-character rule, so 々々 (as in 部分々々) gives [々, 々], where StandardTokenizer gives one <ALPHANUM> token [々々] (fix B splits it too). The two Han Word_Break=Extend marks (U+16FF0, U+16FF1) stay outside every token, as on main and as with StandardTokenizer. It is a tailoring in the direction of CLDR-2009.

What the checks validated

• Gradle, JDK 25, scratch worktree at a9779ef3c193: on stock the new test fails 4 of 5; regenerateRbbi rewrote Default.brk (22,792 to 22,808 bytes) and the checksum file, and left MyanmarSyllable.brk unchanged.
• :lucene:analysis:icu:check passes: the checksum check, checkGoogleJavaFormat, ecjLint, forbiddenApis, checkLicenses and all module tests: 17 suites, 163 tests, 0 failed (the 5 new tests and 148 upstream tests; the 10 tests of TestICUTokenizerCJK are @AwaitsFix).
• The module tests pass with assertions off and in nightly mode (163 tests, 0 failed each).
• The finder's and the independent reproducer's tests pass with the fix (16 of 16 with the new test).
• The regenerated Default.brk gives the same tokens as the rules compiled at runtime.

[$Ideographic $Han] $ExFm* {400}; at Default.rbbi:131. Same result for the three letters; the finder's run passed its repro (4 of 4), TestICUTokenizer (45) and TestWithCJKBigramFilter (12) with the rules compiled at class init. The diagram shows this variant; N gives the same tokens and counts for every input in it.

Trade-offs

Also turns the 330 Word_Break=Other radicals and the 2 Han marks into <IDEOGRAPHIC> tokens (332 of 332 kept after 人 in the probe), which goes beyond UAX #29 and beyond what 4.6.1 did.

Add [$Han & $ALetter] $ExFm* {200}; after Default.rbbi:131. 々 becomes <ALPHANUM>, as in StandardTokenizer and 4.6.1; the radicals stay as on main. Probed with rules compiled at runtime; not run in gradle.

Trade-offs

CJKBigramFilter passes <ALPHANUM> tokens through, so 人々 gives [人, 々] rather than the bigram 人々; the new test's testBigramFilter fails with B by design.

Use the default cjkAsWords=true, which keeps 人々 as one word; or put kuromoji's JapaneseIterationMarkCharFilter in front of the tokenizer, in both the index and the query analyzer: it expands a 々 that follows a kanji (人々 -> 人人, tokens [人, 人]).

Trade-offs

The first changes CJK segmentation as a whole. The second needs the kuromoji module and is partial: it passes a lone or leading 々 through unchanged (JapaneseIterationMarkCharFilter.java:28-34), which the tokenizer still drops (々 -> [], 々人 -> [人]), and it does not handle U+303B or U+16FE3 (executed).

Method: bounded-exhaustive translation validation (the hunt's "semi-formal" tier), hunt unit "ICU script iterator and ICUTokenizer against reference semantics", 2026-10-27, on main a9779ef3c193. No model checker, SMT solver or proof produced the signal.

• A harness ran the stock ICUTokenizer on every string over a 35-character alphabet up to length 4 and a 16-character alphabet at length 5: 2,593,337 strings x 2 configurations (cjkAsWords true and false), 25.9 million checks, against reference semantics for the script iterator and the token loop (0 mismatches there).
• A "content dropped" oracle compared it with StandardTokenizer, the core UAX #29 tokenizer: every letter or digit StandardTokenizer puts in a token must lie inside some ICUTokenizer token. It flagged three dropped-content classes; U+3005 was one, the other two became separate rows (a word joined to an emoji by ZWJ, and Hangul followed by a combining mark).
• An AI tool listed the 335 Han-script code points that are not Ideographic with ICU's UnicodeSet, separated the cjkAsWords=false defect from ICU root's own handling of a lone 々 in the default mode, found the status rule in Default.rbbi, and wrote the public-API JUnit. Independent AI tools then reproduced and judged it (see the checks).