A text value that ends in half an emoji loses its tokens (all of them in a short value) in StandardAnalyzer and the other JFlex scanners

When a text value ends in an unpaired high surrogate (U+D800..U+DBFF), which is what you get when a Java String is cut by length in the middle of an emoji, or when a JSON string ends in the escape \ud83d, StandardAnalyzer and StandardTokenizer at default settings emit no tokens for a value shorter than their 255-char buffer and report final offset 0: "hello world" + U+D83D (12 chars) gives [] instead of [hello, world], and the document is not found by title:hello. No error is raised and the stored value keeps its words (IndexWriter replaces the lone surrogate with U+FFFD), so the loss is silent. Longer values lose or split only their last buffer window; the other scanners have larger buffers (4,096 chars for Classic and Wikipedia, 16,384 for HTMLStrip). The cause is one line in the refill method of both Lucene JFlex skeletons: after reading new chars that end in a high surrogate, it reads one more char for the low half, hits end of input and returns true, which its own javadoc defines as "no new input"; the scanner then ends the stream without scanning the chars it just read. The same line is generated into StandardTokenizerImpl and UAX29URLEmailTokenizerImpl (since 7.7.0, LUCENE-8527) and ClassicTokenizerImpl, WikipediaTokenizerImpl and HTMLStripCharFilter (since 9.1.0, LUCENE-10239); 7.6.0 handles the same input correctly. Returning false there (one line in each skeleton, then regenerate) fixes this branch in all five; the regenerated fix passes the core analysis tests and the whole analysis-common test suite. No earlier report was found.

• Surrogate pair: Java strings are UTF-16. A character above U+FFFF, such as most emoji, takes two chars: a high surrogate (U+D800..U+DBFF) then a low surrogate (U+DC00..U+DFFF). 😀 (U+1F600) is \uD83D\uDE00. Cutting a string between the two leaves an unpaired (lone) high surrogate at the end.
• JFlex scanner: StandardTokenizer's engine, StandardTokenizerImpl, is generated by JFlex from a grammar (StandardTokenizerImpl.jflex) and a skeleton, Lucene's template for the buffer code (gradle/regenerate/jflex/). UAX29URLEmailTokenizer, ClassicTokenizer, WikipediaTokenizer and HTMLStripCharFilter are generated the same way.
• Buffer and refill: the scanner reads the text in windows of its buffer size (255 chars for StandardTokenizer, ZZ_BUFFERSIZE); zzRefill() fetches the next window from the Reader.
• Final offset: after the last token, end() must report the total number of chars processed.

1. The contract, :750: "@return false iff there was new input."
2. :769-773: the refill asks the Reader for the free part of the 255-char buffer. For "hello world" + U+D83D the Reader returns all 12 chars, fewer than requested.
3. :780-781: if (numRead > 0) { zzEndRead += numRead; The 12 chars are now in the buffer.
4. :782, :789-790: the last char is a high surrogate and there is room, so it calls zzReader.read() "Expecting to read a paired low surrogate char".
5. :791-792: the text has ended, read() returns -1, and the code does if (c == -1) { return true; }: "no new input", although 12 chars were just added.
6. :1004, :1010-1012: getNextToken() takes true as end of input without decoding the buffered chars. At a token start (:1034-1037) it sets zzAtEOF and returns YYEOF, so StandardTokenizer returns no token (StandardTokenizer.java:159) and end() reports yychar() + yylength() = 0 (:181). Inside a token, the token is closed at the old buffer end, so a word that crosses the refill boundary is split in two.
7. Without the bug, the lone U+D83D would be decoded and matched by the grammar's catch-all rule [^], whose action is "ignore it" (StandardTokenizerImpl.jflex:286), as happens when the same char is in mid-text.

The branch comes from the skeletons (skeleton.disable.buffer.expansion.txt:120-121, skeleton.default.txt:124-125) and is in all five generated scanners: UAX29URLEmailTokenizerImpl.java:33667, ClassicTokenizerImpl.java:505, WikipediaTokenizerImpl.java:658, HTMLStripCharFilter.java:5953.

Two documents titled "hello world" and "hello world" + U+D83D: title:hello matches 1. A whole value is lost when it is one char shorter than the scanner's buffer, counting the surrogate:

A value of exactly the buffer size takes a different branch (numRead == requested, which holds the high surrogate back for the next read): the four tokenizers keep every token and report the full final offset (255 / 4,096). HTMLStripCharFilter keeps every token too but drops the trailing lone surrogate itself: a 16,384-char value comes out 16,383 chars long, final offset 16,383. That one-char loss is a different branch, which the fix below does not change (a possible sibling, not yet triaged).

In a seeded sweep over random word texts (2,000 per length), every text of 10..200 chars before the surrogate lost all tokens, 254 and 255 were unaffected, and at 256..5,000 about 1/6 lost the tail, 2/3 had one word split and 1/6 were unaffected; the fractions depend on the text shape.

Two conditions shape these numbers. The Reader: they are for a Reader that returns as many chars as asked, which is what a String field value gets (Field.java:498 calls Analyzer.tokenStream(String, String), which uses a ReusableStringReader, Analyzer.java:183-190). A Reader that returns fewer chars per call lets earlier tokens through; the branch still fires at the end (in the bounded check, a 1-char Reader gave 262 violations where a whole-string Reader gave 820). Buffer growth: Classic, Wikipedia and HTMLStrip come from the default skeleton, which doubles the buffer when a token outgrows it (skeleton.default.txt:97-104), so their "last window" is the current buffer, which starts at 4,096 or 16,384 chars.

• Default configuration. StandardAnalyzer with nothing changed, which is also what new IndexWriterConfig() uses when no analyzer is given (IndexWriterConfig.java:136-142). By code reading (not executed), it is also what Elasticsearch and OpenSearch build for an index's default analyzer; Elasticsearch's html_strip, classic and uax_url_email wrap the other scanners; Solr's _default configset uses the standard tokenizer for text_general.
• Silent. No exception with -ea or -da. IndexWriter stores the value (the lone surrogate becomes U+FFFD, as its javadoc documents; the words are kept), so the document displays normally but its words cannot be searched. CheckIndex is clean.
• Contained. Only the value that ends in the surrogate loses text. Other values of the same field, other fields and other documents are indexed normally. Inside the same document, a later value of the same multi-valued field gets a shifted term-vector position and offsets (position 0 and [1,6) where 2 and [12,17) are right), so offset-based highlighting can mark the wrong place there.
• How much. A whole value of up to 254 chars (StandardTokenizer, UAX29URLEmailTokenizer), 4,095 (Classic, Wikipedia) or 16,383 (behind HTMLStripCharFilter, whatever tokenizer follows); longer values lose or split their last window (for Classic, Wikipedia and HTMLStrip the current buffer, which can grow). With a larger maxTokenLength, the StandardTokenizer window grows to about that length (read, not run).
• No resource cost. On 1,000,000-char values each scanner reads the same chars with the same number of Reader calls as the control.
• How likely. The value's very last char must be the first half of a supplementary character, with nothing appended after the cut: a length-limited title, snippet or column cut by substring, or a client that sends such a string as a JSON escape. A committer wrote in #15899 / #15900 that half surrogates from UTF-16 truncation are to be expected and code must be prepared for them. No report since 2019, but the loss is silent, and the random text behind the analysis tests' checkRandomData (TestUtil.randomAnalysisString) keeps surrogate pairs whole and trims a high surrogate left by a cut, so those tests do not produce this input. It may be under-reported.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[[hello, world] finalOffset=12]> but was:<[[] finalOffset=0]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.standard.TestStandardAnalyzerTrailingHighSurrogate.testTrailingUnpairedHighSurrogate(TestStandardAnalyzerTrailingHighSurrogate.java:44)

• The method's contract. "@return false iff there was new input" (StandardTokenizerImpl.java:750, both skeletons :77). The branch returns true right after zzEndRead += numRead with numRead > 0.
• The analysis contract. At the end of each field "The Offset attribute must contain the final offset (the total number of characters processed)" (analysis/package-info.java:276-281); BaseTokenStreamTestCase checks text.length() (:1423, and assertAnalyzesTo). Here it is 0 for a 12-char value.
• The scanner accepts lone surrogates by design. [^] { ... ignore it. } (StandardTokenizerImpl.jflex:286); the JFlex manual says [^] matches "any character of the input character set, including unpaired Unicode surrogate characters". Executed controls: a lone high surrogate in mid-text, a trailing lone low surrogate, "high surrogate + space" and a trailing valid pair all keep [hello, world].
• Lucene accepts this input elsewhere. IndexWriter documents the U+FFFD replacement (IndexWriter.java:1483-1485); that shows the input is accepted, not how an analyzer must tokenize it. TestAnalyzers.testLowerCaseFilterLowSurrogateLeftover feeds "BogustermBoguster\ud801" to WhitespaceTokenizer and expects it as a token; TestCharacterUtils.testFill and TestCharArraySet.testSingleHighSurrogate use raw lone high surrogates, including at the end of input. (Not TestHTMLStripCharFilter.testUTF16Surrogates: it uses numeric character references.)
• History. LUCENE-8527 (2019) added the look-ahead for a different problem, well-formed pairs split by Readers that return short reads (jflex-de/jflex#538 / #543); an unpaired surrogate at the true end of input is never discussed, and its CHANGES entries mention only the JFlex 1.7.0 upgrade, Unicode 9.0 and emoji support. LUCENE-10239 (#452, 2021) copied the branch from JFlex's skeleton with the message "Change doesn't alter the behavior of any of the analyzers". 7.6.0 and, for Classic, Wikipedia and HTMLStrip, 9.0.0 keep both words with final offset 12 ([hello, world]; HTMLStripCharFilter + WhitespaceTokenizer gives [hello, world\uD83D], keeping the surrogate in its last token).

The strongest case for "works as designed": an unpaired surrogate is ill-formed UTF-16 (Unicode conformance C10), so garbage in, garbage out. The test framework seems to agree: MockTokenizer fails with "stream ends with unpaired high surrogate" (MockTokenizer.java:271), TestUtil.randomSubString, which randomAnalysisString calls for its text (TestUtil.java:1686, :1703), trims a high surrogate left at the end of a cut (:1740-1744, LUCENE-3911), and in jflex#538 the author of the branch wrote that surrogates "should not be present unpaired in a well-formed UTF-16 sequence". A maintainer could reasonably say that ill-formed UTF-16 has no defined tokenization.

Why it is plausible but outweighed for this result: no javadoc of Analyzer, Tokenizer, StandardTokenizer, StandardAnalyzer or the analysis package sets that precondition, and IndexWriter documents that it accepts such strings (acceptance, not tokenization semantics); the grammar and JFlex's [^] accept them everywhere except at the very end; the output is not "garbage" for the bad char but the loss of every good word before it (one char erases up to 253 good chars, 16,382 behind HTMLStripCharFilter, while the same char one position earlier erases nothing); Lucene's own tests feed raw lone surrogates to other analysis components; and the behaviour arrived in toolchain upgrades meant to change nothing else. MockTokenizer and TestUtil show that Lucene's own tests keep random input well-formed (the code does not say why, and neither is a documented precondition for users), and they are why randomized tests never reached this branch. The cross-model reviews and the intended-behaviour verifier rated the case weak; this doc's OpenAI GPT-5.6 fact check rated it moderate for "ill-formed input is unsupported" in general and weak for this silent loss of the words in front. Verdict: not intended, a defect.

One fix was regenerated into all five scanners and tested with gradle. The others are options for JFlex itself, a more invasive variant, and a workaround for applications.

numRead > 0 chars were just added, so by the javadoc the answer is false. The scanner then scans the buffered chars; the lone surrogate at the end is decoded and matched by [^] (ignored by the tokenizers, passed through by HTMLStripCharFilter), exactly as in mid-text. The next refill calls read() again, gets -1 and returns true from the existing numRead < 0 path (StandardTokenizerImpl.java:802-803). Scope: every value that takes this branch (the read left room in the buffer). It does not touch the numRead == requested branch, so a value of exactly 16,384 chars still comes out of HTMLStripCharFilter one char short (executed: with the fix, 12, 16,383 and 16,385 chars pass through unchanged, 16,384 does not).

--- a/gradle/regenerate/jflex/skeleton.default.txt   (same change in skeleton.disable.buffer.expansion.txt)
+++ b/gradle/regenerate/jflex/skeleton.default.txt
@@ -122,7 +122,7 @@
         } else {                    // There is room in the buffer for at least one more char
           int c = zzReader.read();  // Expecting to read a paired low surrogate char
           if (c == -1) {
-            return true;
+            return false;           // new chars were read; the next call reports EOF
           } else {
             zzBuffer[zzEndRead++] = (char)c;
           }

Trade-offs

• Negligible cost: the branch runs at most once per value. With the fix a trigger value makes one more read() call, which returns -1: 3 Reader calls instead of 2 for "hello world" + U+D83D, the same 3 as for "hello world" without it (executed).
• Calling read() again after -1 is not new: the stock scanner already does it for a value that ends inside a word, such as "hello world" (2 of its 3 calls return -1, executed). Only StringReader and ReusableStringReader were tried; a custom Reader that misbehaves after -1 was not tested.
• JFlex's own skeleton keeps the branch, so a Lucene-only fix would come back if open issue #14645 replaces Lucene's skeletons with JFlex's built-in buffer features (option B).
• It does not fix the sibling LFV-132 (other branches of the same method, maxTokenLength = 1), nor the unverified "H H L" short-read observation in the same block, nor the HTMLStripCharFilter exact-buffer-size case above.

What the checks validated

• Scratch worktree of ea57360912ed, JFlex 1.9.1 from the build: regenerateStandardTokenizer, regenerateClassicTokenizer, regenerateWikipediaTokenizer, regenerateHTMLStripCharFilter and regenerateUAX29URLEmailTokenizer (with -x regenerateTlds, which would download a new TLD list) change exactly that line in each generated file plus five checksum files (12 files, +17/-17).
• With the fix, seed B21EF158: :lucene:core:test --tests 'org.apache.lucene.analysis.*' 24 classes, 186 tests, 0 failures (TestStandardAnalyzer 45/45); the whole :lucene:analysis:common:test 343 classes, 1,929 tests, 0 failures, 18 skipped (TestUAX29URLEmailTokenizer 46, TestClassicAnalyzer 29, TestWikipediaTokenizer 8, TestHTMLStripCharFilter 36).
• The two new tests (9) and the finder's, the first verifier's and the devil's-advocate verifier's tests pass; all seven trailing-surrogate test classes pass 20 iterations with seed 5EED158B (760 runs, 0 failures). checkGoogleJavaFormat and ecjLintTest pass for both modules.
• Without the fix the same gradle run of the two new tests fails 5/5 and 4/4. Earlier, the hunt's shadow of StandardTokenizerImpl passed every reviewer's test. With such a shadow, the bounded-exhaustive checker's final-offset violations for StandardTokenizer at maxTokenLength 255 go from 820 to 0 with a whole-string Reader and from 262 to 0 with a 1-char Reader (re-run by this doc's fact check); the separate "H H L" count with a 1-char Reader stays (465 before, 502 after).

JFlex's skeleton.default (master) has the same branch, so other scanners generated from it may have the same return path and could be checked (for example a copy in a TOML parser of a JSON library's text-format module; whether users see lost text depends on the grammar and the Reader; not tested). A maintainer may prefer this route: fix it in JFlex and carry the Lucene skeleton patch until a JFlex release is picked up.

Trade-offs

Needed for a durable fix if Lucene moves to JFlex's built-in buffer features (#14645); Lucene's own no-buffer-expansion skeleton needs the change either way; on its own it does not help Lucene until a JFlex release is picked up. Not tried.

Keep the high surrogate out of the buffer and decide on the next refill, as the numRead == requested branch does.

Trade-offs

More state and code than the one-line fix, in the branch where the sibling LFV-132 and jflex-de/jflex#1090 live. Not tried.

Replace unpaired surrogates with U+FFFD, or cut strings by code point, or append a separator, before analysis.

Trade-offs

Not a Lucene fix. Reasoned from the verifiers' controls (a lone high surrogate followed by a space keeps the words), not run as a mitigation test.

Method: semi-formal, bounded-exhaustive checking of the stock main jar (a hunt unit, analysis core and case folding, 2026-10-26).

• The harness (StdTokTV) ran all 7,380 sequences of 1..4 symbols over a 9-symbol alphabet (StdTokTV.java:41-44: two supplementary characters, two lone surrogates, five BMP chars; so 1..8 UTF-16 units), with maxTokenLength 1..5 and 255, whole-string and 1-char Readers, through StandardTokenizer and UAX29URLEmailTokenizer: 177,120 runs.
• Its reference tokenizer (an unlimited-buffer run of the same scanner) shares the bug, so a differential check alone missed it; the separate oracle "end() final offset equals the input length" flagged 820 inputs for each tokenizer at the default maxTokenLength 255 with a whole-string Reader (also the number of generated sequences that end in the lone high surrogate: 1 + 9 + 81 + 729). The shortest: the single unit U+D801 gives no tokens and final offset 0.
• A written case analysis of zzRefill() located the c == -1 branch; a seeded length sweep measured how much text is lost, and a deterministic JUnit reproduced it on the stock jar with -ea and -da.
• An independent reviewer from another model family (OpenAI GPT-5.6) then tried to refute it before it became a candidate, and wrote its own failing test.