ICUTokenizer drops a word, and the emoji after it, when a ZERO WIDTH JOINER sits between them

ICUTokenizer with its default configuration emits no token for abc + U+200D + 👍, and hello world + U+200D + 😀 + again gives only hello and again: the word and the emoji are both gone. With a space instead of the ZERO WIDTH JOINER, or without it, both are tokens, and StandardTokenizer keeps both (world+ZWJ, 😀). Unicode word-break rule WB3c (ZWJ × Extended_Pictographic) keeps world ZWJ 😀 in one segment; in Lucene's break rules (and in ICU's own root rules) that segment ends in a rule without a status, so its rule status is 0, which ICUTokenizer treats as a non-token; BreakIteratorWrapper's emoji check reads only the segment's first code point, a letter; and ICUTokenizer skips every status-0 segment. Letters, digits, ideographs and kana are hit alike, with cjkAsWords true or false; a TermQuery for the word misses the document. Nothing throws, CheckIndex is clean and the stored value keeps the text. It started in 7.4.0 with the 2018 ICU rule upgrade: a public-API probe loses the word on all 67 releases from 7.4.0 to 10.5.2 and on main, and keeps it on all 7 releases 7.0.0-7.3.1. In unspaced text that a dictionary segments (Japanese or Chinese with the default cjkAsWords=true, Thai, Myanmar) one trigger loses the whole run before the joiner, such as a whole Japanese sentence. Fix A (+76/-4 in BreakIteratorWrapper) ends the break iterator's text right after such a joiner, but only where the segment holding it would be skipped: the word and the emoji become separate tokens (the same tokens as StandardTokenizer for the tested Latin, digit, single-ideograph and Malayalam inputs), the repro and an exact-token test pass, and the ICU module's tests run with 0 failures.

• Rule status: ICU's word breaker cuts text into segments and labels each with the status of the rule that ended it: 200 for a word with letters, 100 for a number, 300 for kana, 400 for ideographs, and 0 when that rule has no status tag (spaces, punctuation). ICUTokenizer treats status 0 as a non-token and keeps only segments with a nonzero status ("skipping over non-tokens (rule status 0)", ICUTokenizer.java:209-214).
• ZERO WIDTH JOINER (ZWJ, U+200D): an invisible character that glues emoji into one picture, as in 👩‍❤️‍👩. Unicode rule WB4 attaches a ZWJ to the word before it, and rule WB3c ("ZWJ × Extended_Pictographic") forbids a break between a ZWJ and a following emoji.
• Emoji status: ICU gives a lone emoji status 0 as well, so Lucene's BreakIteratorWrapper re-labels a segment as <EMOJI> when its first code point is an emoji.

1. Default.rbbi runs under !!chain (:46). The word rules end in $ExFm*, which includes the ZWJ (:116, :122-131; the Hangul rule at :127 does not, the separate Hangul bug), and WB3c $ZWJ $Extended_Pict; (:104-107) has no status tag. The match of world+ZWJ continues into WB3c, so world+ZWJ+😀 is one segment, and its status is 0. world+ZWJ alone gets 200.
2. ICU4J 78.3's own ROOT word iterator gives the same 0. With cjkAsWords=true, Han and kana runs use exactly that iterator (DefaultICUTokenizerConfig.java:73, :113-114); its dictionary splits ひらがな+ZWJ+👍 into four pieces, all with status 0.
3. BreakIteratorWrapper.calcStatus (:57-66) changes the status only when isEmoji (:75-91) holds, and isEmoji reads only the first code point (:78), here w.
4. ICUTokenizer.incrementTokenBuffer skips the segment (ICUTokenizer.java:211): the word and the emoji are lost together.

Search over three documents, hello world+ZWJ+😀 again, the same with a space, and without the joiner: TermQuery(body:world) finds 2 (never the first), TermQuery(body:😀) 2, TermQuery(body:hello) 3. In a query, the text analyzes to nothing and the classic QueryParser drops the clause (QueryParserBase.java:367-369): +report -secret+ZWJ+🔒 parses to +body:report.

How much one trigger loses: the run of letters, digits or ideographs before the joiner, up to about 4,000 chars (the tokenizer reads 4096-char chunks). In space-separated text that is one word. In unspaced text that a dictionary segments it is the whole run: 私は日本語が好きです+ZWJ+👍 gives no token with the default config (6 tokens lost; with cjkAsWords=false only the last kana), 我爱北京天安门+ZWJ+👍 none (4 tokens), a Thai run (สวัสดีครับ) or a Myanmar run before ZWJ+👍 none with either config, and a 1003-char Japanese run none. Fix A restores all of them.

Related loss, same mechanism: a stray ZWJ before an emoji at the start of the text or after a space loses the emoji: ZWJ+😀 cd gives cd only. Fix A restores it too.

• Who is hit. Applications that analyze text with ICUTokenizer: new ICUTokenizer(), new ICUTokenizer(new DefaultICUTokenizerConfig(false, true)) (both executed), or the icu factory with default parameters, which builds the same config (ICUTokenizerFactory.java:96-97, 112, 156; read). Lucene's default analysis (StandardAnalyzer) is not affected. The class is @lucene.experimental.
• What triggers it. A run of letters, digits, ideographs or kana immediately followed by U+200D and an emoji: an Extended_Pictographic character that is not also a letter, such as 👍 (U+2139 ℹ is both, so abc+ZWJ+ℹ stays one word). The run may itself end in an emoji-like letter or a digit: abcℹ+ZWJ+👍 and abc1️⃣+ZWJ+👍 also give no token. Ordinary keyboards do not type a joiner between a word and an emoji; realistic sources are stray joiners in pasted or generated social-media text and Malayalam in the older <consonant, virama, ZWJ> chillu spelling (the newer one-character chillu letters are not affected). How often real text has it was not measured; the registry rates it rare.
• What goes wrong. That word and that emoji are not in the field's terms: a TermQuery, a phrase or a MUST_NOT exclusion on the word misses the document, and in a query the clause disappears.
• Silent and deterministic. No exception, CheckIndex clean, stored value intact, the rest of the text tokenized normally, the same with -ea and -da.
• Not more than that. No hang: the probe's 1M- and 4M-char inputs finished, the bug pattern faster than a control (it emits fewer tokens), with time roughly proportional to size. Memory was not measured.
• Long-lived. Every release since 7.4.0 (2018). A fix changes tokens only of texts that hit the pattern; existing indexes need a reindex to gain the lost words.
• Workaround until a fix ships: an ICUNormalizer2CharFilter (NFKC_CF) in front of the tokenizer removes the joiner, and the word and emoji become tokens; it also splits real emoji ZWJ sequences (👩‍❤️‍👩 becomes three emoji). A token filter after the tokenizer (ICUFoldingFilter) does not help.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected a token containing "abc", got []
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.icu.segmentation.TestICUTokenizerWordZwjEmoji.testWordFollowedByZwjAndEmojiIsKept(TestICUTokenizerWordZwjEmoji.java:40)

• The contract is word segmentation. "Breaks text into words according to UAX #29: Unicode Text Segmentation" (ICUTokenizer.java:31). DefaultICUTokenizerConfig "Generally tokenizes Unicode text according to UAX#29" with one listed tailoring, dictionary breaking for Thai, Lao, Myanmar, Khmer and CJK (DefaultICUTokenizerConfig.java:28-38). The module overview presents it "As a more thorough replacement for StandardTokenizer" (overview.html:67).
• Lucene overrides ICU's status for emoji on purpose. BreakIteratorWrapper exists for "emitting a rule status for emoji sequences" (BreakIteratorWrapper.java:25-26): ICU gives a lone emoji status 0, and Lucene keeps it as <EMOJI>. Upstream testEmojiTokenization expects poo💩poo to give poo, 💩, poo (TestICUTokenizer.java:491-497); every upstream emoji test with a ZWJ starts with an emoji (:456-465, :506-519).
• Lucene's UAX #29 reference keeps this text. StandardTokenizer gives a+ZWJ, 🛑 for the WordBreakTest line 0061 × 200D × 1F6D1; Lucene's test generator skips that line on purpose, "because it conflicts with TR#51 v11.0 rules" (generateJavaUnicodeWordBreakTest.pl:130-141), and its oracle counts every segment that holds a letter, digit, ideograph or Extended_Pictographic character as a token (WordBreakTestUnicode_12_1_0.java:28-39).
• A regression. 7.3.1 gives abc for abc+ZWJ+👍; 7.4.0 gives nothing. With ICU4J held at 61.1, the 7.3.1 rule file gives abc status 200 and the 7.4.0 rule file gives the joined segment status 0.
• No sign it was decided. The status-0 skip dates from the class's first commit (LUCENE-2414, 2010); that commit's wrapper for dictionary iterators gave a status-0 segment a word or number status when it held a letter or digit. The 2018 upgrade that brought WB3c (LUCENE-8122 / #9170, LUCENE-8175 / #9223) and the emoji status (LUCENE-8125 / #9173) never discusses a word before ZWJ and an emoji; no javadoc, test, CHANGES or MIGRATE entry accepts dropping such text. No evidence was found that the 2018 change meant this text to disappear.

Strongest case for "intended" (rated moderate by the cross-model review lenses and the cross-model fact check): UAX #29 WB3c forbids a break between a ZWJ and an emoji, so one segment is correct; ICU's own root iterator also returns status 0, Default.brk is meant to be "the same as ROOT, except no dictionary segmentation for cjk" (DefaultICUTokenizerConfig.java:80), and ICU calls the status "secondary information" (ICU-20966, closed as a caller-side choice); a test rule file documents that status 0 is "not tokenized by ICUTokenizer" (Latin-break-only-on-whitespace.rbbi:27); valid emoji ZWJ sequences start with an emoji, LUCENE-8125 calls its emoji check "very conservative", StandardTokenizer departs from WB3c on purpose with an explicit TR #51 emoji grammar (StandardTokenizerImpl.jflex:166-208), and the class is @lucene.experimental.

Why it fails: the segmentation is not the defect, dropping the segment is. Skipping status 0 is Lucene's own choice (ICUTokenizer.java:211), and ICU-20966 puts exactly that choice on the caller. Lucene already refuses to drop ICU's status-0 emoji; a status-0 segment that holds a word is the same situation. ROOT parity is about boundaries: Lucene already sets its own status for emoji, and the test rule file uses status 0 for whitespace, a segment with no content. StandardTokenizer, also documented as UAX #29, keeps both, and 7.3.1 kept the word. @lucene.experimental marks API that may change, not license to lose valid text. The case is stronger for the lost word than for the lost emoji: before 7.4.0 there were no emoji tokens, so only the word is a regression. Verdict: not intended, a defect. Whether the right output is one token (world+ZWJ+😀) or the word plus the emoji is a maintainer choice.

Fix A and fix B were each validated with the repro through the repro runner and with gradle in a scratch worktree at a9779ef3c193 (OpenJDK 25.0.3); fix A also with an exact-token test and the module's static checks. Option C was not run. Neither fix was benchmarked. The related ICU status-0 bugs (Han letters such as 々 with cjkAsWords=false; a Hangul syllable before Extend, Format or ZWJ) are separate: their repros still fail with fix A and pass with fix B.

BreakIteratorWrapper looks in each script run for a U+200D directly followed by an Extended_Pictographic character. If the code point before that ZWJ (Extend, Format and ZWJ characters skipped) is an emoji that cannot end a word, the ZWJ is inside an emoji sequence and nothing changes. Otherwise the wrapper asks the break iterator for the segment that holds the ZWJ (following(), getRuleStatus(), previous()). Only if that segment has status 0 and does not start with an emoji, exactly when the tokenizer would skip it, the break iterator gets the text up to and including that ZWJ, and the rest as the next piece; current() and next() add the piece offset. One file, +76/-4.

The result: the word plus its ZWJ, then the emoji, as two tokens, with both configs. For Latin, digits, single ideographs and Malayalam these are StandardTokenizer's tokens (world+ZWJ <ALPHANUM>, 😀 <EMOJI>); the word token is what ICUTokenizer already gives for the word plus ZWJ alone. Words that end in an emoji-like letter or a keycap digit (abcℹ+ZWJ+👍, abc1️⃣+ZWJ+👍) are covered too. On a dictionary path it gives what the tokenizer gives for the run plus ZWJ alone, then the emoji (日本語+ZWJ, 👍 with cjkAsWords=true, where StandardTokenizer splits the ideographs); whole Japanese and Chinese sentences and Thai and Myanmar runs come back token for token. It also restores the emoji after a stray ZWJ (ZWJ+😀 cd gives 😀, cd). Emoji-led sequences stay one <EMOJI> token: 😂+ZWJ+👍, 👩‍❤️‍👩, 👨🏼‍⚕️, a keycap or a flag before ZWJ + emoji, tag sequences, ℹ+ZWJ+👍.

An earlier version decided from the character before the ZWJ alone and missed words ending in an emoji-like letter or keycap digit (aℹ+ZWJ+👍 still gave no token); the cross-model fact check found it, and the final fix asks the break iterator instead.

    for (int i = pieceStart; i < length - 1; i++) {
      if (text[start + i] == 0x200D
          && EXTENDED_PICTOGRAPHIC.contains(UTF16.charAt(text, 0, start + length, start + i + 1))
          && emojiBefore(i) == false
          && isSkipped(i)) {
        textIterator.setText(text, start + pieceStart, i + 1 - pieceStart);
        rbbi.setText(textIterator);
        return;
      }
    }
    rbbi.first();

Trade-offs

• A deliberate deviation from UAX #29 WB3c. StandardTokenizer also breaks there (its grammar replaces WB3c with TR #51's emoji sequences); the class javadoc promises UAX #29, so CHANGES should say so.
• The word token keeps its trailing ZWJ (world+ZWJ), as ICUTokenizer and StandardTokenizer already do for world+ZWJ alone. The bare word matches after a filter that removes U+200D: with ICUFoldingFilter, TermQuery(body:world) finds 2 of 2 documents (1 of 2 on stock); with the tokenizer alone 1 of 2, as with StandardTokenizer.
• Scope: tokens change only where a segment that holds a ZWJ before an Extended_Pictographic character would be skipped today: the word cases here and the stray-ZWJ cases. This holds for every break iterator, including custom rule files given to the icu factory: their text is split only where their own segment would be skipped (reasoned from the code, not run).
• Not covered: a Hangul word before ZWJ + emoji keeps only the emoji, because Hangul + ZWJ has status 0 even without an emoji, and the Han status-0 letters such as 々: their repros still fail with fix A.
• Cost: one pass over each script run looking for U+200D; at each ZWJ before an emoji a short walk back, and, unless an emoji precedes the ZWJ, one lookup on the break iterator. A rough timing (4M chars per text, median of 21 runs, stock and fix interleaved, one shared host with up to about 20% run-to-run spread): no clear difference on plain Latin, CJK or emoji text without a ZWJ; Malayalam with ZWJ chillus about 10-16% slower; text made only of emoji ZWJ sequences about 50-85% slower; text where every word has the pattern about 1.7 times the same text without the ZWJ (about 2.2 times stock, which skips those segments); ℹ+ZWJ+👍 repeated about 2.1-2.4 times; a 4M-char chain without spaces about 3.6 times. The cost grows with the number of ZWJs before an emoji; the fix was not optimized further. Not benchmarked.
• Tokens change only for affected texts; existing indexes need a reindex to gain the lost words.

What the checks validated

• The repro: 5 of 6 fail on stock, 6 of 6 pass with the fix (the repro runner -ea and -da; gradle module runs).
• An exact-token test, TestICUTokenizerZwjEmojiSplit (10 tests: tokens and types with both configs, offsets for one input, words ending in an emoji-like letter or keycap, the dictionary path, stray ZWJs, emoji sequences unchanged, random strings): 10 of 10 pass with the fix (-ea and -da); on stock 7 of 10 fail; with the earlier version of fix A 2 of 10 fail.
• All of :lucene:analysis:icu:test with both new tests: 18 suites, 174 tests, 0 failed, 10 skipped, with assertions on (seed 5EED0659) and off (seed 0DA50659); TestICUTokenizer 45 of 45. Stock with the repro: 17 suites, 164 tests, 5 failed (all in the repro), the same 10 skipped (TestICUTokenizerCJK, @AwaitsFix LUCENE-8222).
• :lucene:analysis:icu:check -x test passes (checkGoogleJavaFormat, ecjLint, forbiddenApis, the rbbi checksum check).
• Token probes on the inputs above, the kana, ideograph and dictionary-script probes, the edge cases and the search check: identical to the earlier version on every earlier input.

In calcStatus, a status-0 segment that holds a letter becomes WORD_LETTER, one with a digit WORD_NUMBER (+19/-2; the shape of verifier's root-cause probe). It keeps UAX #29 boundaries exactly, and it also keeps the Han and Hangul status-0 segments of the related bugs (one fix site for three bugs): their repros pass with fix B (4 of 4 and 6 of 6).

Trade-offs

• The output is one glued token world+ZWJ+😀 <ALPHANUM>: TermQuery(body:world) still misses the document, even with ICUFoldingFilter (the folded term is world😀), and the emoji is no <EMOJI> token.
• Types change on the dictionary path: ひらがな+ZWJ+👍 gives four <ALPHANUM> tokens; stock types ひらがな as <IDEOGRAPHIC>.
• The emoji after a stray ZWJ stays lost (ZWJ+😀 cd gives cd).
• Wider behaviour change: every status-0 segment that holds a letter or digit becomes a token, not only this pattern (including the segments of the related Han and Hangul bugs).

What the checks validated

• The repro: 6 of 6 pass (the repro runner -ea).
• :lucene:analysis:icu:test with the new test: 17 suites, 164 tests, 0 failed, 10 skipped (assertions on, seed 5EED0659); TestICUTokenizer 45 of 45. Verifier earlier ran its probe of this shape against upstream TestICUTokenizer: 45 of 45.

Give WB3c chains that start with a word the word's status in Default.rbbi (keeps one glued token), or apply WB3c only to a ZWJ after an emoji, then regenerate Default.brk. Not run.

Trade-offs

Neither covers Japanese-script runs with cjkAsWords=true, which use ICU's ROOT iterator (DefaultICUTokenizerConfig.java:73, :113-114), and a tailored WB3c has to be carried across ICU upgrades. ICU may want to know too, since its root rules give the same status 0; both validated fixes are Lucene-side.

Method: bounded-exhaustive translation validation of the stock jar against a reference (the hunt's semi-formal tier for the hunt; the first signal was non-formal).

• First signal (2026-10-26). While assessing the security of the ICUTokenizer surrogate-split bug (LFV-445), an AI tool ran a public-API probe on main and saw ab+ZWJ+😀 cd give only cd, with ICU's root word iterator giving that span status 0. It noted that this "may deserve its own candidate" and did not investigate further.
• The hunt (2026-10-27, main a9779ef3c193). A hunt unit on ICUTokenizer's script iterator ran a harness over 2,593,337 generated strings in two configurations, with a "content dropped" oracle against StandardTokenizer. Every string with X + ZWJ + 👍 (U+1F44D) dropped X, for every letter, digit and ideograph class in its alphabet; a buffer-boundary harness also flagged the WB3c rule. The finder reduced it to a public-API JUnit.
• Checks. Claude verifiers wrote their own tests (the independent reproduction: Latin, mid-sentence, Malayalam with both configs, digits, the emoji, 4 controls, plus a root-cause probe; the intended-behaviour check: a devil's advocate who found Lucene's WordBreakTest stance and the NFKC_CF workaround). Review records by a model from another family (OpenAI GPT-5.6, executed mode) wrote and ran their own tests: both "real, high". An adjudication re-ran the independent reproduction's test with a fresh seed, and an independent audit added new inputs (see you tomorrow, Paris+ZWJ+❤️, x9).