ICUTokenizer loses an emoji or rare CJK character whose surrogate pair straddles its 4096-char read chunk

ICUTokenizer reads its input in 4096-char chunks; while more input follows, it ends each chunk after the chunk's last whitespace character. When a full chunk has no whitespace at all, it uses the whole chunk without checking whether the last char is the first half (high surrogate) of a supplementary character. The ICU word breaker then sees a lone high surrogate at the end of one chunk and a lone low surrogate at the start of the next; neither is a word, both are skipped, and the character is in no token. In about 4,200 chars of Chinese without spaces, U+20000 (an Extension B ideograph) or U+1F600 (an emoji) at char 4095 is lost, while at char 4094 or 4096 it is a token, and a TermQuery for U+20000 over three such documents finds 2 instead of 3. It happens with new ICUTokenizer() and the icu factory with default parameters, on all 16 releases executed from 3.1.0 to 10.5.2 and on main (with and without assertions on main), and every release tag that contains the class has the same fallback. Nothing throws, no malformed term is indexed, CheckIndex is clean and the stored text keeps the character. A 4-line hold-back of the trailing high surrogate fixes it, and the ICU module's 163 tests run with 0 failures with it.

• Supplementary character, surrogate pair: a code point above U+FFFF, such as an emoji (U+1F600) or a CJK Extension B ideograph (U+20000). Java stores it as two chars: a high surrogate (U+D800-U+DBFF) followed by a low surrogate (U+DC00-U+DFFF). Neither half is a character on its own.
• Chunk: ICUTokenizer keeps a fixed 4096-char buffer (ICUTokenizer.java:41-42). It hands the ICU word breaker the part of the buffer up to the last whitespace and carries the rest over to the front of the next chunk.
• Rule status 0: the ICU word breaker marks spans that are not words (spaces, punctuation, unmatched chars) with rule status 0; the tokenizer skips them.

1. findSafeEnd() (:143-150) returns the position after the last UCharacter.isWhitespace char of the buffer, or -1.
2. refill() (:158-181) carries the unused tail over, reads up to the buffer size and, when the input is not exhausted, uses findSafeEnd() (:170). If that is -1, the fallback is usableLength = length; (:176), under the comment "more than IOBUFFER of text without space, gonna possibly truncate tokens" (:172-175). It never checks whether buffer[length - 1] is a high surrogate.
3. breaker.setText(buffer, 0, usableLength) (:180) gives the breaker a chunk that ends in a lone high surrogate; the next refill() starts with the lone low surrogate.
4. incrementTokenBuffer() skips every span with rule status 0 (:211). Both lone halves get status 0 (printed for U+1F600 by the diagram probe), so the character is in no token, and the offsets jump from the high surrogate's position to after the low surrogate.

The cut falls at char 4096k-1 in text with no whitespace at all (4095, 8191, 16383, ...), otherwise 4096 chars after the last whitespace, because each chunk starts with the previous chunk's leftover. Lucene's own helper CharacterUtils.fill holds back a trailing high surrogate for exactly this reason (lucene/core/.../CharacterUtils.java:158-161, 203-204).

A bounded-exhaustive sweep (13 fillers x 13 probes x 7 buffer boundaries x 9 positions) flagged the icu tokenizer with default, cjkAsWords=false and myanmarAsWords=false settings at exactly chars 4095, 8191 and 16383, for every supplementary probe (emoji, Extension B ideograph, flag, ZWJ sequence; the oracle counts each probe's first code point) and every whitespace-free BMP filler (9 of the 13 fillers): 324 of 324 such cases, and these are all the violations the sweep recorded for the three icu configurations. Of the other fillers, 2 contain spaces, and the 2 made of surrogate pairs were never probed at the cut, because the sweep skips insertion points on a low surrogate. BMP base characters at 4095 (U+4E00, U+2764) are kept. An audit with two short lines and then one long Japanese line misses only the document with U+2000B exactly 4095 chars after the last line break.

Scope. The same forced cut also orphans a BMP combining mark that becomes the first char of the next chunk (e + U+0301, or Thai U+0E01 + U+0E34, at 4095-4096: the mark is in no token). That is a separate, grapheme-level effect; this bug is the cut inside a code point, which loses a whole character. The recommended fix does not change the combining-mark case.

• Who is hit. Applications that analyze text with ICUTokenizer: new ICUTokenizer(), new ICUTokenizer(config), or the icu factory (CustomAnalyzer, factory-based configs) with default, cjkAsWords=false or myanmarAsWords=false settings (all executed). The icu_tokenizer factories of Elasticsearch and OpenSearch call new ICUTokenizer() / new ICUTokenizer(config) directly, and Elasticsearch's icu_analyzer chains it (read, not executed downstream). Lucene's default analysis (StandardAnalyzer) is not affected, and the class is @lucene.experimental.
• What triggers it. A run of more than 4,096 chars without any whitespace, not even a line break, and a supplementary character whose high surrogate is exactly the last char of the chunk. Realistic mainly for long unbroken CJK or Thai text (for example extracted text whose newlines were stripped) that contains emoji, Extension B+ ideographs or supplementary-plane letters. The chance per cut is roughly the share of supplementary chars in that text (reasoning, not measured).
• What goes wrong. One character per affected 4096-char window is missing from that field's terms; if it occurs nowhere else in that field, a TermQuery or a MUST_NOT exclusion on it misses the document (executed), and a phrase query containing it would miss too (reasoning, not run). Before 7.4.0 emoji were not tokens at all, so there the loss shows on ideographs such as U+20000; from 7.4.0 (LUCENE-8125) emoji are lost too.
• Silent and deterministic. No exception or warning, no lone-surrogate or U+FFFD term, CheckIndex clean, stored text intact, other documents and fields unaffected, the same with -ea and -da. The cut position depends only on the text.
• Not more than that. No hang and no clear extra cost: 1,048,576-char inputs run at 0.97-1.20x a shifted control (single host, best of 3, noisy).
• Long-lived. Present since the class was added in 2010 (executed on 16 releases; the same fallback in every release tag). Fixing it changes the tokens of exactly the affected documents; existing indexes need a reindex to gain the lost characters.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: tokens equal to U+20000 expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.analysis.icu.segmentation.TestICUTokenizerRefillSurrogatePair.testSupplementaryCharAtChunkEnd(TestICUTokenizerRefillSurrogatePair.java:43)

• The public description is UAX #29 (tailored per script). Class javadoc: "Breaks text into words according to UAX #29: Unicode Text Segmentation" (ICUTokenizer.java:31-32), then "Words are broken across script boundaries, then segmented according to the BreakIterator and typing provided by the ICUTokenizerConfig" (:34-35); package javadoc: "Tokenizer that breaks text into words with the Unicode Text Segmentation algorithm" (package-info.java:18). UAX #29 boundaries, and script-boundary breaks, fall between code points. No public javadoc of the class, the factory or the package mentions the chunking.
• The only stated limit is a maximum word length. Internal comment: "So there is a maximum word limit of 4kB since it will not try to grow the buffer in this case" (:133-135; "4kB" is the 4,096-char buffer, not bytes) and "gonna possibly truncate tokens" (:173-174). TestICUTokenizer.testHugeTerm2 (:47) pins that as a lossless split: 40,960 a become exactly ten 4096-char tokens. Here no long word is split; a 2-char character disappears. The fix keeps testHugeTerm2 passing.
• Upstream tests require supplementary characters to be tokens. testEmoji (:447) and testEmojiTokenization (:491) expect <EMOJI> tokens, including next to CJK text (💩中國💩). In LUCENE-1488 (#2562), the issue that led to the ICU module, rmuir described the planned components in 2009: "All of these components should also work correctly with surrogate-pair data".
• Lucene treats split surrogate pairs as bugs. CharacterUtils.fill guarantees that "high and low surrogate pairs will always be preserved across buffer boarders" (CharacterUtils.java:158-161), and ICUNormalizer2CharFilter in the same module reads through it (:96); TestStandardAnalyzer.testSplitSurrogatePairWithSpoonFeedReader (:696); CHANGES: "LUCENE-8325: Fixed the smartcn tokenizer to not split UTF-16 surrogate pairs" (lucene/CHANGES.txt:7463) and GITHUB#15899/#15900, TruncateTokenFilter "no longer produce half surrogates" (:1182-1183), even for a filter whose job is truncation.
• No intent in the history to split a pair. The bounded 4,096-char truncation is deliberate (the comment above), but nothing in the history mentions cutting a character. The fallback came with the class (LUCENE-2414, 2010); its discussion (#3489: 12 comments before a 2011 bulk-close note) is about tailoring, the build and the API, nothing about chunk ends or surrogates, and no commit on the file ever mentions surrogates.

Strongest case for "intended" (all cross-model review records rated it moderate, the independent reproduction weak, the intended-behaviour check weak to moderate): the class is @lucene.experimental; its comment accepts degraded output when 4kB have no whitespace ("binary data", "possibly truncate tokens"), and testHugeTerm2 locks in arbitrary 4096-char cuts. Tokenizers drop rule-status-0 text by design, and nothing promises that every character is indexed. The forced cut already orphans combining marks. In open #10793 a maintainer called the chunking "a failsafe" that "can cause other issues that look nothing like this, too".

Why it fails: truncating a token gives shorter words that are still indexed; it does not license cutting one valid code point into two non-token halves that the input never contained, and the same character is a token one char earlier or later. Status 0 marks spans the word breaker does not treat as words (spaces, punctuation and other non-word text); the two halves of a valid emoji or ideograph get it only because the cut separates them. @lucene.experimental marks API that may change, not undefined behaviour on valid text. The combining-mark loss is real and is kept out of this claim. #10793 admits the chunking is weak and wants it fixed; it never describes this failure, and the fix keeps the bounded 4096-char buffer. Verdict: not intended, a defect. The minimal fix is local, and #10793's prototype (ICUTokenizer on SegmentingTokenizerBase) would carry the same cut into that class, whose fallback has the same gap.

Fix A and fix B were each validated with gradle in a scratch worktree at ea57360912ed (OpenJDK 25.0.3) and through the repro runner; fix B fails one pinned upstream test. Option C was not run. The similar SegmentingTokenizerBase bug (smartcn and Thai, a 1024-char buffer, the same fallback) is separate.

In the no-whitespace branch of refill() (ICUTokenizer.java:176), if the last char of the full chunk is a high surrogate, end the chunk one char earlier. That char becomes the first char of the next chunk, so the pair is tokenized whole with correct offsets (𠀀[4095,4097)); documents where the hold-back never fires are unchanged. It is the pattern of CharacterUtils.fill. The finder wrote this patch, and an independent verifier wrote the same hold-back. The other two branches cannot end inside a pair: the whitespace branch ends right after a BMP whitespace char, and the end-of-input branch ends where the input ends.

        usableLength = length;
        // do not cut a surrogate pair in half: keep a trailing high surrogate for the next chunk
        if (usableLength > 1 && Character.isHighSurrogate(buffer[usableLength - 1])) {
          usableLength--;
        }

Trade-offs

• Progress: in this branch the buffer is always full (4,096 chars), so holding back one char still advances each refill by at least 4,095 chars; the usableLength > 1 guard is defensive and never fires there. With the fix, 1,048,576 lone high surrogates finish in 56 ms.
• Output change only for documents where the hold-back fires: they gain the lost character (existing indexes need a reindex to gain it). In those documents every later forced cut in the same whitespace-free run moves one char earlier (after char 8190 instead of 8191, and so on) until a whitespace char resets it, so tokens near those later cuts can differ too, for example where a forced cut splits a long word (reasoning from refill(), not run).
• Ill-formed text: it also holds back a lone high surrogate, as CharacterUtils.fill does.
• Does not change the separate orphaning of a combining mark at the forced cut (executed).
• Not benchmarked: one array read per no-whitespace refill.

What the checks validated

• The repro: 4 of 5 fail on stock, 5 of 5 pass with the fix (gradle with assertions on and off; the repro runner with -ea and -da).
• All of :lucene:analysis:icu:test: 17 suites, 163 tests (including the 5 new), 0 failed, 10 skipped, with assertions on (seed 1C0DE445) and off (seed 77AA0445); TestICUTokenizer 45 of 45, including testHugeTerm2. Stock: the same 163 tests, 4 failed (all new), the same 10 skipped (@AwaitsFix in TestICUTokenizerCJK).
• :lucene:analysis:icu:check -x test passes (checkGoogleJavaFormat, ecjLint, forbiddenApis).
• The finder's JUnit (4 of 4), an independent verifier's (8 of 8), the audit probe and the security probe pass with the same hold-back.

In the same branch, ask an ICU character BreakIterator (getCharacterInstance(ULocale.ROOT) over the module's CharArrayIterator) for the start of the last grapheme cluster and end the chunk there (+11 lines).

Trade-offs

• Changes pinned behaviour: TestICUTokenizer.testHugeTerm2 fails, because a run of a now splits into 4,095-char tokens instead of 4,096.
• Wider: it also keeps the combining marks that the forced cut orphans (U+0301, Thai U+0E34; executed).
• Not complete for one huge cluster: if the whole buffer is one grapheme cluster, preceding(length) returns 0 and the lastCluster > 0 guard keeps the full buffer, so that cluster (and a pair at its end) can still be cut (read from the code, not run).
• A second break iterator per tokenizer and a setText per no-whitespace refill; not benchmarked.
• It is the kind of chunking change open #10793 discusses; propose it there, not as the bug fix.

What the checks validated

• The repro: 5 of 5 pass (the repro runner -ea; gradle module run).
• :lucene:analysis:icu:test: 163 tests, 1 failed (testHugeTerm2); checkGoogleJavaFormat passes.

The prototype attached to #10793 (LUCENE-9754, never merged) replaces the ICU chunking with SegmentingTokenizerBase.

Trade-offs

Not a fix on its own: SegmentingTokenizerBase.refill has the same no-safe-end fallback, so the cut would remain unless that class gets the same hold-back. Not run.

Method: a bounded-exhaustive buffer-edge sweep with a metamorphic oracle (the hunt's semi-formal tier for the hunt; the first signal was non-formal).

• First signal. While verifying the similar SegmentingTokenizerBase bug (smartcn and Thai), a verifier dumped ICUTokenizer's tokens on main and saw U+20000 at chars 4095-4096 in no token; that bug's adjudication noted it as a separate fix site that needs its own candidate.
• The sweep. A hunt over analysis chains built through the factory names ran 13 fillers x 13 probes x 7 buffer boundaries x 9 positions with the oracle "the probe appears as often as at a control position". It flagged the icu tokenizer (default, cjkAsWords=false, myanmarAsWords=false) at exactly chars 4095, 8191 and 16383, for every supplementary probe (emoji, Extension B ideograph, flag, ZWJ sequence) and every whitespace-free BMP filler (9 of the 13 fillers; the two fillers made of surrogate pairs were never probed at the cut): 324 of 324 such cases, all the violations the sweep recorded for these three configurations. The finder reduced it to a public-API JUnit.
• Checks. Claude verifiers wrote their own tests (the independent reproduction: Latin and Han fillers, a position sweep and an index; the intended-behaviour check: a devil's advocate who found the combining-mark caveat), an adjudication re-ran the independent reproduction's test with a fresh seed, and an audit wrote an end-to-end probe with a line-broken Japanese document. Review records by a model from another family (OpenAI GPT-5.6) could only read the code; all agreed with the root cause (one real, medium confidence; two procedural "not real", low confidence, because they could not run anything).