JapaneseTokenizer and KoreanTokenizer split an emoji in half at the 1024-char unknown-word cap

Kuromoji's JapaneseTokenizer and nori's KoreanTokenizer group a run of characters that their dictionary does not know into one unknown word and stop it at MAX_UNKNOWN_WORD_LENGTH = 1024 UTF-16 chars, a limit commented "For safety". Neither checks whether char 1023 is the first half (high surrogate) of a supplementary character such as an emoji or a CJK Extension B ideograph. When it is, the character is split across two terms: one ends with a lone high surrogate, the next starts with the lone low surrogate, and the index stores both as U+FFFD. In the single-character modes (kuromoji EXTENDED, nori outputUnknownUnigrams=true) a TermQuery for the character then misses the document. It needs a run of more than 1,023 chars that the tokenizer cannot break, such as long space-free Thai through kuromoji or long emoji runs; the default JapaneseAnalyzer drops these words and is not affected, the default KoreanAnalyzer is. Present since kuromoji's first release (3.6.0, 2012) and nori's (7.4.0, 2018); a 5-line guard per tokenizer fixes it.

• Supplementary character, surrogate pair: Java strings are UTF-16. Characters above U+FFFF, such as most emoji (😀 is U+1F600) and CJK Extension B ideographs (U+20000), take two chars: a high surrogate (U+D800-U+DBFF) followed by a low surrogate (U+DC00-U+DFFF). Either half alone is not a character.
• Unknown word: when the dictionary has no entry, kuromoji and nori make a word from the run of following chars that share the first char's character class (kuromoji) or Unicode script group (nori).
• Unigram modes: kuromoji's EXTENDED mode and nori's outputUnknownUnigrams=true emit an unknown word as one token per character; that code keeps a surrogate pair together on purpose.

1. The cap: // For safety: protected static final int MAX_UNKNOWN_WORD_LENGTH = 1024; in the base class both tokenizers share (morph/Viterbi.java:41-42).
2. Kuromoji's loop (ja/ViterbiNBest.java:137-149) and nori's (ko/Viterbi.java:92-126, the same header for (int posAhead = pos + 1; unknownWordLength < MAX_UNKNOWN_WORD_LENGTH; posAhead++)) add one UTF-16 char at a time. When the length reaches 1024 they stop, and nothing looks at the last char.
3. Both surrogate halves are class DEFAULT in both tokenizers, like Thai letters and U+FE0F; nori sees each half as script UNKNOWN and joins it to Common or Inherited characters such as U+FE0F. So a long Thai run (kuromoji) or an emoji run with U+FE0F (nori) continues through the emoji, and only the cap ends it. A pair-aligned run such as 600 × 😀 never splits; it takes an odd number of BMP chars before the cut.
4. When the emoji's high surrogate is char 1023 (or 2047, 3071, ...), one word ends with it and the next starts with the low surrogate. The unigram code works inside one word and cannot rejoin them.
5. At indexing, UnicodeUtil.UTF16toUTF8 writes each lone half as U+FFFD (UnicodeUtil.java:172-176), as IndexWriter documents (IndexWriter.java:1483-1485). The index stays well-formed, so nothing fails.

Indexed: with 😀 at char 1023, no SEARCH-mode term contains it and two terms contain U+FFFD (at char 1023 of the 1,024-char term and char 0 of the 176-char term); EXTENDED-mode TermQuery(😀) gets 1, 0 and 1 hits with 😀 at chars 1022, 1023 and 1024, and 1 at 1023 with the fix.

• Who is exposed. JapaneseTokenizer in SEARCH (the default), NORMAL and EXTENDED mode, with discardPunctuation true or false, and the japanese tokenizer factory with default parameters, in any chain that keeps these words: the hunt's sweep flagged 15 of the 16 kuromoji-based factory chains it built (with reading, base-form, katakana-stem, number and other filters); the 16th, with the part-of-speech stop filter, emitted no broken term, which fits the default stop tags dropping these words. KoreanTokenizer, new KoreanAnalyzer() and the korean factory with defaults on emoji runs joined by U+FE0F, and nori's unigram mode. Both nori classes are @lucene.experimental.
• Not exposed: the default JapaneseAnalyzer. Kuromoji tags these words 記号-一般 (symbol, general), and the default stop tags drop that tag; it emits 0 tokens for the probe inputs.
• What it takes. More than 1,023 UTF-16 chars of one unknown-word class with no break, and a supplementary character whose first half lands exactly on a cut (char 1023, 2047, ...). Ordinary Japanese and Korean prose rarely does this: kanji and kana are other classes, and Korean text has spaces. Realistic cases are long space-free Thai (or other text kuromoji has no class for) in a Japanese chain, and long emoji runs for nori. How often real text hits a cut was not measured.
• What the user sees. The index stores U+FFFD in place of the character, in two terms, and token offsets point inside the pair. In the unigram modes a search, or a term-based exclusion filter, for that character misses the document. In SEARCH, NORMAL and the default KoreanAnalyzer a TermQuery for the character never matched the 1,024-char run token anyway, so its hit count does not change (prefix, wildcard and query-parser queries were not tested); what remains there is the U+FFFD terms, exact-phrase matches between two long runs that differ only in the character at the cut, and lone surrogates in highlight snippets for a U+FFFD query.
• Scale. One character per 1,024-char cut in that field value. The executed probes found no exception and no measured extra cost (timings 0.88-1.00× a shifted control on about 262,144 chars), CheckIndex clean, stored value intact.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 2 test(s) run, 2 failed
    => java.lang.AssertionError: term of length 1024 is not valid UTF-16
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.TestUnknownWordCapSurrogatePair.assertTermsValidUTF16(TestUnknownWordCapSurrogatePair.java:34)
       org.apache.lucene.analysis.TestUnknownWordCapSurrogatePair.testKoreanTokenizer(TestUnknownWordCapSurrogatePair.java:51)
  JUnit Vintage:TestUnknownWordCapSurrogatePair:testJapaneseTokenizer
    MethodSource [className = 'org.apache.lucene.analysis.TestUnknownWordCapSurrogatePair', methodName = 'testJapaneseTokenizer', methodParameterTypes = '']
    => java.lang.AssertionError: term of length 1024 is not valid UTF-16
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.TestUnknownWordCapSurrogatePair.assertTermsValidUTF16(TestUnknownWordCapSurrogatePair.java:34)
       org.apache.lucene.analysis.TestUnknownWordCapSurrogatePair.testJapaneseTokenizer(TestUnknownWordCapSurrogatePair.java:44)

• Kuromoji's tests state the contract. TestJapaneseTokenizer.testSurrogates2 (:414-431) and TestExtendedMode.testSurrogates2 (:58-72), "random test ensuring we don't ever split supplementaries", assert UnicodeUtil.validUTF16String(termAtt) for every term. They came with the original contribution (LUCENE-3305), before the cap existed, and their 100-char random inputs never reach 1,024.
• The same code keeps pairs together. The unigram paths pair a low surrogate with its high surrogate (ja/ViterbiNBest.java:442, ko/Viterbi.java:218); in kuromoji this predates the cap: it came with the original contribution (LUCENE-3305, 2012) and was kept when the cap was added.
• Precedents. lucene/CHANGES.txt:7479 (7.4.0): "LUCENE-8325: Fixed the smartcn tokenizer to not split UTF-16 surrogate pairs", where the nori author wrote "the tokenizer should not split surrogate pairs into multiple tokens"; :1198-1200 (10.5.0): TruncateTokenFilter "no longer produce half surrogates" (#15899, #15900). In the LUCENE-3767 discussion that added the cap, one comment names "don't separate a lead surrogate from a trail surrogate" as a general rule.
• No javadoc we examined allows it. The javadocs of both tokenizers and their factories say nothing about the cap or supplementary characters (grep for 1024, surrogate, supplementary). IndexWriter's U+FFFD note covers malformed input; here the input is well-formed and the tokenizer makes the halves.
• For nori the contract is weaker: no nori test asserts it; it rests on nori's pair-preserving unigram code, the loop it copied from kuromoji and the statement above.

The strongest case (the intended-behaviour checker and the cross-model reviewer): the cap is a deliberate safety bound counted in UTF-16 chars ("For safety" above both 1,024 constants; on #4970 the sibling forced-backtrace limit is described as "for safety / prevent adversary"); both tokenizers classify each char separately; nori is @lucene.experimental; the input is unusual; and no javadoc promises well-formed terms.

Why it fails: the cap explains why a word is cut, not why a character is, and backing off by one char keeps the bound (the fix). The per-char design is exactly what the unigram code compensates for, and kuromoji's tests state the property. Experimental status and rare input are about impact, not intent. Verdict: not intended, a defect.

The recommended option is the finder's 5-line guard per tokenizer, re-made as a git diff at a9779ef3c193 (the guard lines are identical; only the code comment is shortened from "when the unknown word reaches" to "when the word reaches") and validated in a scratch worktree with the whole kuromoji and nori test tasks plus the paste-ready tests, assertions on and off, and the modules' lint and format checks. The other options were not run.

The word ends at 1,023 chars and the next unknown word starts with the whole emoji: [0,1023) + [1023,1200); EXTENDED emits the emoji as [1023,1025); the indexed term contains the emoji and the EXTENDED-mode TermQuery finds the document.

// after the unknown-word loop, in ja/ViterbiNBest.java and in ko/Viterbi.java:
// do not cut a surrogate pair in half when the word reaches its maximum length
if (unknownWordLength == MAX_UNKNOWN_WORD_LENGTH
    && Character.isHighSurrogate((char) buffer.get(pos + unknownWordLength - 1))) {
  unknownWordLength--;
}

Trade-offs

• The bound still holds (at most 1,024 chars) and the length becomes 1,023, never 0, so the lattice always moves forward. It fires only when the cap is reached and the last char is a high surrogate; all other text tokenizes as before. No cost change in the security probe (0.98-1.00× with the fix).
• Documents indexed before the fix keep their U+FFFD terms until they are reindexed.
• The guard is duplicated in two classes; a PR could put a small protected helper next to the constant in the shared morph/Viterbi.java (not run).
• In nori the loop may already have updated characterId from the dropped char when the run so far was all Common or Inherited (ko/Viterbi.java:120-125); the halves and U+FE0F are all class DEFAULT, so nothing changes in the probed cases. Reviewed but not run: a run of 1,023 Common or Inherited chars whose next char is the high surrogate, where the shortened word keeps the dropped char's class.
• A PR could add the tests as methods of TestJapaneseTokenizer / TestExtendedMode and TestKoreanTokenizer / TestKoreanAnalyzer; the default KoreanAnalyzer on an emoji run is the default-configuration path worth covering.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the two paste-ready tests fail on stock main, 4 of 4, assertions on and off; with the fix the whole :lucene:analysis:kuromoji:test task passes, 31 classes, 197 tests, 0 failures (7 skipped), and the whole :lucene:analysis:nori:test task, 14 classes, 89 tests, 0 failures (4 skipped), with assertions on and off. The skips are pre-existing module tests. The closest suites TestJapaneseTokenizer (45), TestExtendedMode (5), TestKoreanTokenizer (17) and TestKoreanAnalyzer (7) pass.
• ecjLintMain, ecjLintTest and checkGoogleJavaFormat of both modules pass with the fix and the tests.
• With the repro runner and the two fixed classes first on the classpath, the paste-ready tests pass on main with assertions on and off, and the finder's 6-test JUnit passes; earlier, the independent reproducer's 12 tests and a 300-text random corpus (0 lone-surrogate terms) passed with the same change.
• The gradle runner ran in two rounds: round 1 ran the two stock stages, then stopped on a shell error in the script (an empty bash array) before the fix stages; round 2 ran both fix stages and the lint stage.

Extend by one char instead of backing off.

Trade-offs

The cap is then no longer a strict 1,024 bound, and nothing checked whether other code relies on it. More risk than the back-off for no gain. Not run.

Count a supplementary character as one in the loop.

Trade-offs

Moves the cut in every long run with supplementary characters (up to 2,048 chars), not only in the failing case: a bigger behaviour change than the defect needs. Not run.

• Sweep. The hunt unit on analysis chains built through the SPI factories ran a bounded-exhaustive buffer-edge sweep: 13 filler texts × 13 probes × buffer boundaries × positions, with the oracle "no term with an unpaired surrogate from well-formed input". It reported 900 violations: 60 in each of 15 kuromoji-based chain configurations, at the cuts 1023, 2047, 4095, 8191 and 16383 (a few chars earlier for multi-code-point probes); the 16th kuromoji chain, with the part-of-speech stop filter, reported none. The hits came from the Thai, Hangul, emoji and Extension B fillers with the emoji, Extension B, flag and ZWJ-sequence probes, and with a BMP Hangul jamo probe that shifts an emoji or Extension B filler run by an odd number of chars. No nori configuration reported this violation: its Thai and Hangul fillers break at the emoji (script change), and its emoji and Extension B filler runs stay pair-aligned.
• Follow-up. A probe of the same cap in nori found the emoji + U+FE0F run, which keeps one script group up to the cap.
• Reduction. The finder reduced both to a public-API JUnit (6 tests). No formal tool (model checker, SMT solver or proof) produced a signal.