StandardTokenizer and UAX29URLEmailTokenizer with maxTokenLength 1 stop at the first emoji or other supplementary character, so the rest of the text is never indexed

StandardTokenizer.setMaxTokenLength(n) sizes the JFlex scan buffer to exactly n chars, and Lucene's no-expansion skeleton never grows it. With n = 1, the first code point above U+FFFF cannot fit: zzRefill() holds the high surrogate back and returns true ("no new input") with nothing in the buffer, and at a token start the scanner reads that as the real end of input. So StandardTokenizer, StandardAnalyzer, UAX29URLEmailTokenizer, UAX29URLEmailAnalyzer and their factories with maxTokenLength = 1 emit nothing after the first emoji, CJK Extension B ideograph or mathematical letter in a value, and end() reports a short final offset ("ab 𐐀 cd": [a, b], 3 of 8). No error, with assertions on and off. Only maxTokenLength = 1 is affected on well-formed text; the setting is legal but unusual. It ships in every release from 4.10.0 (LUCENE-5897, 2014) to 10.6.0. A 6-line change to the shared skeleton, regenerated for both scanners, fixes it.

• Supplementary character: a code point above U+FFFF. In Java it is two chars, a high surrogate followed by a low surrogate. The repro uses U+10400 (𐐀, DESERET CAPITAL LETTER LONG I) = \uD801\uDC00; emoji such as U+1F600 behave the same.
• Scan buffer: the char[] the JFlex-generated scanner (StandardTokenizerImpl, UAX29URLEmailTokenizerImpl) reads into. Both are generated from Lucene's skeleton gradle/regenerate/jflex/skeleton.disable.buffer.expansion.txt, whose buffer never grows.
• Final offset: the offset end() reports after the last token; Lucene's consistency checker expects it to equal the input length.

1. setMaxTokenLength(length) calls scanner.setBufferSize(length) (StandardTokenizer.java:95-105, :103), which allocates new char[length] (StandardTokenizerImpl.java:713-718). UAX29URLEmailTokenizer.java:100-108 does the same.
2. zzRefill() requests zzBuffer.length - zzEndRead - zzFinalHighSurrogate chars (skeleton :98). If the last char read is a high surrogate and the read filled the request, it holds that char back (--zzEndRead; zzFinalHighSurrogate = 1;) and, if only 1 char was read, returns true, meaning "no new input" (skeleton :112-117, StandardTokenizerImpl.java:783-788). This exists so that a full buffer ends the current partial token; the next refill puts the surrogate back.
3. With a 1-char buffer, every refill reads 1 char. After a, b and the space, the char read is \uD801: it is held back, the buffer is left empty, and zzRefill returns true before any char of the new token was scanned.
4. The scanner turns that into YYEOF (StandardTokenizerImpl.java:1004-1015) and, because nothing was consumed (zzStartRead == zzCurrentPos), sets zzAtEOF = true and returns YYEOF (:1034-1037); later calls return YYEOF without reading (:997-999). incrementToken() returns false (StandardTokenizer.java:159-161) and end() reports yychar() + yylength() = 3 (:178-185). UAX29URLEmailTokenizerImpl.java:33659-33664 and :33932-33935 are the same code.

An instrumented trace on 645d4fe14a20 confirms the order: two normal chops after a and b, then the numRead == 1 return at a token start, then zzAtEOF. (Earlier records also named a second refill with requested == 0; the trace shows it never runs.)

On an index, with StandardAnalyzer(maxTokenLength 1) on go U+1F600 now, a TermQuery for n finds 0 documents; without the emoji it finds 1. A bounded-exhaustive check of the stock jar (3,294,144 runs: all 137,256 well-formed inputs of 1 to 6 symbols over a 7-symbol alphabet: a, space, 1, ., U+4E2D, and the supplementary U+10400 and U+1F600; maxTokenLength 1-5 and 255, two Reader shapes, both tokenizers) found violations only at maxTokenLength 1, on exactly the inputs with a supplementary code point.

• Trigger. An opt-in, unusual setting: setMaxTokenLength(1), or maxTokenLength="1" on StandardTokenizerFactory / UAX29URLEmailTokenizerFactory (StandardAnalyzer.java:87, :92; UAX29URLEmailAnalyzer.java:92, :97; StandardTokenizerFactory.java:46, :60; UAX29URLEmailTokenizerFactory.java:61). No in-tree code sets 1; a partial public code search found only tests.
• Certain once configured. Any code point above U+FFFF in a field value triggers it: emoji, CJK Extension B ideographs, mathematical letters.
• Effect. Silent wrong results: searches for the later words miss the document, while the stored text still shows them. No exception, CheckIndex clean.
• Contained to one value. Later documents, other fields and later values of the same field are tokenized normally, because reset() clears the scanner state. Inside that document, the lost tokens shift the positions, and the short final offset shifts the offsets, of later values of the same multi-valued field (for ["ab 𐐀 cd", "ef"], e is at position 2 and offset [4,5) instead of 4 and [9,10)), which can misplace offset-based highlighting there.
• Lucene's own checker fails. checkAnalysisConsistency and checkRandomData on StandardAnalyzer with maxTokenLength 1 fail (finalOffset expected:<8> but was:<3>; random data expected:<12> but was:<10>) and pass at 2.

LuceneTestCase is only the base class; no randomness is used. The source is ASCII-only, with a \uD801\uDC00 escape. The lost tokens are asserted first, the final offset second.

Regression test: this same class (or its one method added to TestStandardAnalyzer); it passes with the recommended fix. That fix changes the shared skeleton, so the same method also applies to UAX29URLEmailTokenizer (TestUAX29URLEmailTokenizer). The repro checks the later terms and the final offset only, because the offsets and position increments depend on the behaviour chosen for the code point; the validation-only test also asserts them for the recommended fix.

cp $REPRO/TestStandardTokenizerMaxTokenLengthOne.java $LUCENE/lucene/core/src/test/org/apache/lucene/analysis/standard/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestStandardTokenizerMaxTokenLengthOne -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
   >     java.lang.AssertionError: expected c and d after U+10400, got [a, b]
   >         at __randomizedtesting.SeedInfo.seed([DEADBEEF:F3E43334B8C5D9F6]:0)
   >         at org.junit.Assert.fail(Assert.java:89)
   >         at org.junit.Assert.assertTrue(Assert.java:42)
   >         at org.apache.lucene.analysis.standard.TestStandardTokenizerMaxTokenLengthOne.testTextAfterSupplementaryCharIsKept(TestStandardTokenizerMaxTokenLengthOne.java:44)

:lucene:core:test (FAILURE): 1 test, 1  failure

# the same test without the token assertion: final offset expected:<8> but was:<3>
# plain main(), no test framework, -ea and -da:
maxTokenLength=1 tokens=[a, b] finalOffset=3 inputLength=8
maxTokenLength=2 tokens=[ab, U+10400, cd] finalOffset=8 inputLength=8

• The contract is "chop", not "stop". StandardTokenizer.java:86-94, UAX29URLEmailTokenizer.java:91-99 and StandardAnalyzer.java:65-70: "Tokens larger than this will be chopped up at this token length and emitted as multiple tokens." No javadoc, CHANGES or MIGRATE entry says values below 2 are unsupported or that the stream may end early.
• 1 is explicitly accepted and documented. The setter's range is [1, 1048576]; the lower bound "maxTokenLength must be greater than zero" came with LUCENE-5559 (225a9966b64, 2014-04-01); TestUAX29URLEmailTokenizerFactory asserts that message for -1.
• Where the tokenizer drops text on purpose, it continues. StandardTokenizer.java:163-173 skips a too-long token that the scanner produced, with a position gap, and goes on; nothing in the tokenizer ends the stream early on purpose. (This bug never reaches that path: the scanner itself reports the end of the input.)
• Lucene's checker. checkAnalysisConsistency passes text.length() as the expected final offset (BaseTokenStreamTestCase.java:1423-1471); it fails at maxTokenLength 1 and passes at 2.
• Intent of the change. LUCENE-5897 (#6959) disabled buffer growth so that partial matches are limited to the buffer size, and added the early return after the hold-back so that a full buffer ends the current token (the same branch was posted on jflex-de/jflex#538). No discussion of it mentions a 1-char buffer or supplementary characters, and its tests use maxTokenLength 200 to 8192.
• Siblings read to the end of the input. ClassicTokenizer and WhitespaceTokenizer with maxTokenLength 1 report the full final offset. WhitespaceTokenizer(1) gives [a, b, 𐐀, c, d] on "ab 𐐀 cd". ClassicTokenizer skips, rather than chops, words longer than maxTokenLength, so on "ab 𐐀 cd" it emits no tokens (final offset 8 of 8), and on the single-letter "p q 😀 r s" it gives [p, q, r, s].

The strongest case for "intended" (rated moderate by the works-as-designed reviewer, OpenAI GPT-5.6): the buffer is fixed on purpose for performance (LUCENE-5897), and a 2-char code point cannot be chopped to length 1 without splitting a surrogate pair, so something has to give; and TokenStream.end() (TokenStream.java:142-160) only says it sets "the final offset", not its value.

Why it fails: that argument leaves open only what happens to the one code point that cannot fit (drop it, skip it with a gap, emit it whole). It does not justify losing every later, unrelated word of the value, which no document or test accepts. The repro asserts nothing about the code point itself, and it leads with the lost tokens; the final offset rests on Lucene's checker convention. Verdict: not intended, a defect.

One fix was written and validated in a scratch worktree of 645d4fe14a20; the study's grow-the-buffer sketch was run earlier as a shadow of one scanner only, and rejecting the value was not run. Which behaviour Lucene wants for the one code point that cannot fit is the maintainers' choice.

In zzRefill() of skeleton.disable.buffer.expansion.txt, when the high surrogate just read is the only char in the buffer (zzEndRead == 1, which on well-formed input happens only with a 1-char buffer), return false instead of holding it back: the scanner then reads the two halves one at a time, neither forms a token, and scanning continues. StandardTokenizerImpl.java and UAX29URLEmailTokenizerImpl.java were regenerated with JFlex 1.9.1 from the changed skeleton; each gains exactly the same 6 lines, and the two checksum files change.

        if (numRead == requested) { // We requested too few chars to encode a full Unicode character
          if (zzEndRead == 1) {
            // A one-char buffer (maxTokenLength 1) can never hold the pair. Holding the high
            // surrogate back would leave nothing to scan, which reads as the end of input,
            // so scan it on its own; the low surrogate follows with the next refill.
            return false;
          }
          --zzEndRead;
          zzFinalHighSurrogate = 1;
          if (numRead == 1) {
            return true;
          }

Trade-offs

• Smallest change: one place, both tokenizers, and the buffer still never grows.
• The supplementary code point itself produces no token and no position gap, like punctuation. If maintainers want a gap (as the skip path in StandardTokenizer.java:163-173 gives), the grow-to-2 option does that.
• With a buffer of 2 or more the new branch is unreachable on well-formed input, so the default and every other value behave as before.
• It also changes malformed input: a lone high surrogate alone in a 1-char buffer is now scanned instead of ending the stream. The fact check ran 20,000 random strings with lone surrogates per tokenizer at maxTokenLength 1: on stock about 15,200 end with a short final offset, with the fix none, and no exceptions either way. No test covers that case.

What the checks validated

Gradle --no-daemon in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3 (2026-10-31). Stock: the repro fails with assertions on and off. With the fix:

• regenerated with JFlex 1.9.1 (regenerateStandardTokenizer, regenerateUAX29URLEmailTokenizer with the TLD download skipped): each scanner gains exactly the 6 lines and nothing else; the two checksum files change (+22/-4 in 5 files);
• the repro passes with assertions on and off;
• org.apache.lucene.analysis.standard.* (core, 52 tests: TestStandardAnalyzer 45, TestStandardFactories 3, the repro, 3 validation-only methods) and org.apache.lucene.analysis.email.* (analysis-common, 92 tests: TestUAX29URLEmailTokenizer 46, TestUAX29URLEmailAnalyzer 34, TestUAX29URLEmailTokenizerFactory 10, 2 validation-only methods; 1 skipped): 0 failures with seeds DEADBEEF and CAFEBABE;
• the validation-only methods, which all fail on stock: StandardAnalyzer(1) on "ab 𐐀 cd" gives exactly [a, b, c, d], offsets [0,1) [1,2) [6,7) [7,8), final offset 8; checkAnalysisConsistency and checkRandomData (200 iterations) pass for StandardAnalyzer(1) and UAX29URLEmailAnalyzer(1); UAX29URLEmailTokenizer(1) keeps c and d;
• regenerateStandardTokenizerChecksumCheck, regenerateUAX29URLEmailTokenizerChecksumCheck, checkGoogleJavaFormat, forbiddenApisMain and ecjLintMain pass.

The whole core and analysis-common test suites were not run.

When a high surrogate is pending at a token start in a 1-char buffer, grow the buffer to 2, read the low surrogate, and shrink back once the buffer is empty. By the sketch's design the code point is then scanned as a 2-char token, and StandardTokenizer skips it as too long, with a position gap.

Trade-offs

Breaks the "buffer never grows" design by one char, and needs its shrink step so that later tokens are not scanned 2 chars long (and then skipped). The sketch patches only the generated StandardTokenizerImpl, so UAX29URLEmailTokenizer still fails with it; a real patch has to move it into the skeleton.

What the checks validated

Run earlier as a shadow of the generated StandardTokenizerImpl only: the repro passes on 645d4fe14a20, and the study's run passed the 45 TestStandardAnalyzer tests; the UAX29URLEmailTokenizer case still fails with it.

Change the checks in StandardTokenizer.java:96-97 and UAX29URLEmailTokenizer.java:101-102 to length < 2, and the javadoc range to [2, 1048576].

Trade-offs

Simplest and obviously safe, but a breaking change: applications that set 1 get an IllegalArgumentException, so it fits a major release, and the repro would have to assert the exception instead. Not run.

Method: non-formal (a manual probe first; the exhaustive check came after, to characterise it).

• Finder. A hunting unit on analysis code (StandardTokenizer setMaxTokenLength boundary tests as its secondary method) read the JFlex refill code (zzRefill), probed maxTokenLength 1 with a supplementary code point, and wrote a case analysis of the two return true branches.
• Characterised by bounded-exhaustive translation validation of the stock jar: all 137,256 well-formed inputs of 1 to 6 symbols over a 7-symbol alphabet (a, space, 1, ., U+4E2D, U+10400, U+1F600), maxTokenLength 1-5 and 255, whole-string and 1-char-per-read Readers, both tokenizers, 3,294,144 runs; violations only at maxTokenLength 1.
• Checked by independent reproducers, two executed cross-model reviews, an adjudication, the version-history runs, a current-main re-check and a tie-break (see below).