JapaneseTokenizer and KoreanTokenizer keep emoji and other punctuation outside the BMP despite discardPunctuation

Kuromoji's JapaneseTokenizer and nori's KoreanTokenizer drop punctuation and symbol tokens when discardPunctuation=true, the default of the two KoreanTokenizer constructors without the flag (KoreanTokenizer.java:92-94, :104-110), both tokenizer factories, KoreanAnalyzer and JapaneseAnalyzer (every JapaneseTokenizer constructor takes the flag explicitly). They decide with a private isPunctuation(char) that switches on Character.getType(char) over Unicode categories such as Po, So and Sm, and they pass it the token's first UTF-16 char. For a character above U+FFFF that char is a high surrogate, whose type is SURROGATE, so emoji such as U+1F600 and punctuation such as U+10100 (Po) stay in the output as tokens, while BMP characters of the same category (U+2605 So, U+3001 Po) are dropped. Every kuromoji mode, the n-best output and nori's default path (outputUnknownUnigrams=false) are affected; the default KoreanAnalyzer indexes the character as a term, the default JapaneseAnalyzer removes it later with its part-of-speech stop filter and leaves a position increment of 2. Present since kuromoji's first release (3.6.0, 2012) and nori's (7.4.0, 2018); testing the code point in the discard tests and stepping the unknown-word grouping by code point fixes it.

• Supplementary character, surrogate pair: Java strings are UTF-16. Characters above U+FFFF, the BMP's end, take two chars: a high surrogate (U+D800-U+DBFF) and a low surrogate (U+DC00-U+DFFF). Most emoji (😀 is U+1F600), many historic scripts' punctuation (𐄀 U+10100) and math alphanumerics (𝛁 U+1D6C1) are supplementary. Character.getType(char) of either half is SURROGATE; Character.getType(int) of the code point gives its real category.
• discardPunctuation: "true if punctuation tokens should be dropped from the output" (the javadoc of both tokenizers). Default true.
• Unknown word: when the dictionary has no entry, both tokenizers group the following chars of the same character class (kuromoji) or script (nori) into one word, split where "is punctuation" changes.

1. The predicate isPunctuation(char ch) (ja/ViterbiNBest.java:705-727; nori's copy ko/Viterbi.java:416-446) returns true for Zs, Zl, Zp, Cc, Cf, Pd, Ps, Pe, Pc, Po, Sm, Sc, Sk, So, Pi and Pf. SURROGATE is not in the list.
2. The discard sites pass the token's first UTF-16 char: kuromoji's backtrace !isPunctuation(fragment[offset]) (:465), n-best registerNode !isPunctuation(fragment[left]) (:639) and EXTENDED-mode unigrams !isPunctuation(fragment[offset + i]) (:448); nori's shouldFilterToken, isPunctuation(token.getSurfaceForm()[token.getOffset()]) (:412-414).
3. The EXTENDED loop just above (:440-445) deliberately steps back over a surrogate pair so the unigram is the whole emoji, then classifies only its high half.
4. Grouping has the same blind spot (ja/ViterbiNBest.java:122-150, ko/Viterbi.java:79-127): every surrogate half is "not punctuation", so 😀𠮷 (emoji + Extension B letter) becomes one word, where ★吉 splits and drops ★.
5. Scope: nori's default path, outputUnknownUnigrams=false, where the symbol's token goes through shouldFilterToken (ko/Viterbi.java:248-250). With outputUnknownUnigrams=true the unigram branch (:214-236) adds unknown-word unigrams without calling shouldFilterToken at all, so it keeps unknown-word punctuation from either plane; that is a separate bug with its own cause, not this one.

JapaneseTokenizer SEARCH shown; NORMAL, EXTENDED and setNBestCost(2000) give the same (24 of 24 supplementary cases kept). KoreanTokenizer() and KoreanAnalyzer() (default outputUnknownUnigrams=false): [한국, 𐄀, 한국], [한국, 😀, 한국], ... (8 of 8 kept). JapaneseAnalyzer on 日本X日本 gives the terms [日本, 日本], with position increment +2 for the four supplementary characters and +1 for the BMP ones.

• Who is exposed. JapaneseTokenizer in NORMAL, SEARCH and EXTENDED mode and with n-best output, and japanese factory chains without the part-of-speech stop filter: the character becomes a token. KoreanTokenizer(), the korean factory with defaults and the default KoreanAnalyzer: the character becomes a token tagged SY, which is not a default stop tag, and is indexed: in an in-memory index, a TermQuery for 😀 finds the 한국 😀 한국 document (executed). (The factories by code: they build the same tokenizers with the same default.) KoreanTokenizer and KoreanAnalyzer are @lucene.experimental.
• The default JapaneseAnalyzer hides it. The kept token is tagged 記号-一般 (symbol, general), a default stop tag, so the stop filter removes it; it keeps the position, so 日本😀日本 gives increments +1, +2 where 日本★日本 gives +1, +1.
• What changes for the user. Emoji and other supplementary symbols become searchable terms while ★, ♥ or ∇ in the same text do not; positions after such a character shift, so an exact phrase across it stops matching: the phrase "日本 日本" with JapaneseAnalyzer and "한국 한국" with KoreanAnalyzer match the ★ and 、 texts but not the 😀 or 𐄀 texts, and all of them match with the fix (executed; proximity queries were not run). Supplementary punctuation proper (Po, used in historic and minority scripts) is kept, which the javadoc rules out word for word.
• How likely. Emoji are common in user-written Japanese and Korean text, and each one is kept every time; supplementary Po characters are rare in CJK text. The visible harm per character is small, and nobody reported it since 2012.
• Scale (the repeated-input probe). 1,200,000 chars of 😀 give 400,000 tokens through either tokenizer, one per emoji, where ★ gives 0. No hang, no cost amplification beyond ordinary text, every term well-formed UTF-16. This covers the tested default paths, not user dictionaries, non-default nori modes or mixed supplementary text: next to a supplementary letter the emoji stays inside that word (😀𠮷).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 2 test(s) run, 2 failed
    => org.junit.ComparisonFailure: term 1 expected:<[한국]> but was:<[𐄀]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.TestDiscardSupplementaryPunctuation.testKoreanTokenizer(TestDiscardSupplementaryPunctuation.java:38)
  JUnit Vintage:TestDiscardSupplementaryPunctuation:testJapaneseTokenizer
    MethodSource [className = 'org.apache.lucene.analysis.TestDiscardSupplementaryPunctuation', methodName = 'testJapaneseTokenizer', methodParameterTypes = '']
    => org.junit.ComparisonFailure: term 1 expected:<[日本]> but was:<[𐄀]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)

• The contract. JapaneseTokenizer.java:109, KoreanTokenizer.java:119 and KoreanTokenizerFactory.java:61: "true if punctuation tokens should be dropped from the output". A grep of both modules for supplementary, surrogate and BMP finds no restriction. For U+10100 and U+1E95E (Po) the sentence applies word for word; for emoji (So) and math symbols (Sm) it rests on the implementation's own category list, under which ★ and ∇ are dropped.
• Maintainers state the behaviour upstream: "Punctuations are removed by default" (#10020) and "The tokenizer removes punctuations so it's expected that some characters are filtered ..." (#9739).
• The same code handles surrogate pairs on purpose. The EXTENDED unigram loop steps over the pair (it descends from 685035575b3, 2012, "LUCENE-3305: don't split surrogates in extended mode (unless the input is already screwed, GIGO)"), and kuromoji's testSurrogates2 is a "random test ensuring we don't ever split supplementaries". Classifying only the high half right after stepping over the pair reads as an oversight.
• No exemption anywhere. No javadoc, comment, test, CHANGES or MIGRATE entry or upstream discussion exempts supplementary characters; the modules' only supplementary test characters are Extension B letters (Lo), which the predicate does not touch.
• Precedents. Lucene fixed char-wise handling of supplementary characters as bugs elsewhere: LUCENE-8325 (smartcn) and #15899 / #15900 (TruncateTokenFilter), both under Bug Fixes in CHANGES. Earlier, CharTokenizer moved to code points as a 3.1 new feature (LUCENE-2183).

The strongest case (the intended-behaviour checker and the cross-model reviewer): both tokenizers inherit MeCab-style character tables (a 65,536-entry CharacterDefinition; MeCab's char.def has no supplementary ranges), so supplementary characters are simply not modelled; emoji may be worth keeping (StandardTokenizer and ICUTokenizer emit <EMOJI> tokens: LUCENE-8527 and LUCENE-8125); KoreanTokenizer is @lucene.experimental; JapaneseTokenizerFactory calls the flag an "Expert option" in a code comment; in #10430 maintainers leaned towards a token filter instead of this flag.

Why it fails: the discard test is Lucene's own Unicode-category predicate, not a char.def feature, and Java gives the category per code point; the module handles surrogate pairs deliberately elsewhere; neither "experimental" nor "Expert option" documents an input restriction, and the flag defaults to true wherever it has a default (both analyzers, both factories, the two KoreanTokenizer constructors without the flag). Whether emoji should be kept decides the fix direction, not whether the same category treated differently by plane is a defect. Verdict: not intended, a defect.

The recommended option is the hunt's full code-point fix plus a 4-line length guard in each unknown-word loop, as one git diff at a9779ef3c193 (2 files, +75/-25), formatted with gradlew tidy and validated in a scratch worktree: the whole kuromoji and nori test tasks with the paste-ready tests, assertions on and off, plus the modules' lint, forbidden-API and format checks. Option B was run and regresses; C and D were not run.

日本 𐄀 日本 -> [日本, 日本], 한국 😀 한국 -> [한국, 한국], JapaneseAnalyzer position increments +1, and 日本 😀𠮷 日本 -> [日本, 𠮷, 日本], like 日本 ★吉 日本 -> [日本, 吉, 日本].

// discard tests (kuromoji :448, :465, :639; nori shouldFilterToken): the code point, not the first char
!isPunctuation(Character.codePointAt(fragment, offset, offset + length))
private static boolean isPunctuation(int ch) { switch (Character.getType(ch)) { ... } }

// unknown-word loops (kuromoji processUnknownWord, nori processUnknownWord): step by code point
final int len = codePointLenAt(posAhead);
if (unknownWordLength + len > MAX_UNKNOWN_WORD_LENGTH) {
  break; // the cap would split a surrogate pair: end the word before it
}
if (... && isPunctuation(codePointAt(posAhead)) == isPunct) {
  unknownWordLength += len;
  posAhead += len;
}

Trade-offs

• Behaviour change by design. Supplementary punctuation and symbols are now dropped, as their BMP analogues are, so emoji stop being searchable through these tokenizers with the default flag; existing indexes need a reindex to be consistent. Keeping emoji would then need discardPunctuation=false, which keeps all punctuation. Worth a CHANGES or MIGRATE note.
• Unknown-word scan by code point. The loop steps by code point and tests punctuation (nori: also script and digit) on the code point, so it splits an unknown word where a supplementary symbol meets a supplementary letter (😀𠮷 -> [𠮷]), and nori now uses the real script and digit property of a supplementary character instead of UNKNOWN / not-a-digit for its high half. That matches the BMP. It does not make all of the grouping code-point based: the character-class and group lookups in the 65,536-entry CharacterDefinition still read one UTF-16 char, as on stock code. Beyond the module tests and the probes, mixed supplementary text was not surveyed.
• The guard keeps the 1,024-char cap strict when the loop steps by two chars, so a pair is never split at the cap (without it the word could reach 1,025 chars). It covers the coupled cap-split bug in these loops too; one guard is enough if both are fixed. That bug's EXTENDED-mode test expects the emoji kept as a token with discardPunctuation=true and fails with this fix; it would need discardPunctuation=false or a supplementary letter.
• The patch keeps the hunt's helper style (codePointLenAt, codePointAt); a PR could fold them into one codePointAt plus Character.charCount. The extra buffer reads per char were not measured.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the two paste-ready test classes fail on stock main, 5 of 5, assertions on and off; with the fix the whole :lucene:analysis:kuromoji:test task passes, 31 classes, 198 tests, 0 failures (7 skipped), and the whole :lucene:analysis:nori:test task, 14 classes, 89 tests, 0 failures (4 skipped), with assertions on and off. The skips are pre-existing module tests. The closest suites TestJapaneseTokenizer (45, incl. testSurrogates2), TestExtendedMode (5), TestSearchMode (2), TestJapaneseAnalyzer (12), TestKoreanTokenizer (17) and TestKoreanAnalyzer (7) pass.
• ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest and checkGoogleJavaFormat of both modules pass with the fix and the tests.
• With the repro runner and the two fixed classes first on the classpath, the paste-ready tests pass on main with assertions on and off, the finder's 4-test JUnit and the independent reproducer's 8-test JUnit pass.
• Coupled cap-split bug: with this fix its paste-ready tests give 3 of 4 passing; the failing one expects the emoji kept ("emoji at char 1022 is not a token"). Its finder's JUnit: 4 of 6 pass, including an Extension B letter at char 1023; the 2 failures say the emoji is no longer a term, and none is an unpaired surrogate.

Classify Character.codePointAt(...) at the three kuromoji sites and in shouldFilterToken; leave the grouping loops alone. It fixes the single-character cases, but 😀𠮷 is still one unknown word, which it now drops whole, letter included.

Trade-offs

Smallest diff, but loses a letter that stock code keeps; do not propose it alone.

What the checks validated

Run with the paste-ready tests (assertions on): testLetterAfterSupplementarySymbolIsKept fails, NORMAL and SEARCH [日本, 日本], nori [한국, 한국] (EXTENDED passes); the other 3 tests pass. The hunt's cross-model review and intended-behaviour check probes found the same.

#10430 (open) asks for the discard decision on all characters of a token, not the first. A predicate that walks the token by code point would fix both bugs at the shared kuromoji sites.

Trade-offs

Also changes #10430's behaviour (for example （株）), which maintainers discussed there without agreement; keep it a separate decision. The never-merged PR for #10430 tests UTF-16 chars and would not fix this bug.

Document that discardPunctuation only covers the BMP, or stop discarding So and Sm in the BMP too.

Trade-offs

The first contradicts the javadoc for Po characters such as U+10100; the second changes long-standing output for every BMP symbol.

• Scan. The hunt unit on supplementary-plane handling scanned the class files of the main jars for calls of the char overloads of Character (such as Character.getType(C)), which classify one UTF-16 char and so cannot see a supplementary character. It listed ja.ViterbiNBest.isPunctuation and ko.Viterbi.isPunctuation, among others.
• Metamorphic probe. For four pairs of characters with the same Unicode general category, one in the BMP and one outside it (Po U+3001/U+10100, Po U+FF01/U+1E95E, So U+2605/U+1F600, Sm U+2207/U+1D6C1), the token list of <word> X <word> must not depend on the plane of X. On main every supplementary X was kept and every BMP X dropped.
• Reduction. The finder reduced it to a public-API JUnit (4 tests: 3 fail, the BMP control passes). No formal tool (model checker, SMT solver or proof) produced the signal; the first signal was a static bytecode scan.