AutomatonToTokenStream cuts transition labels above U+FFFF to 16 bits: U+10400 comes out as U+0400

AutomatonToTokenStream.toTokenStream(Automaton) turns an acyclic automaton into a token graph; its javadoc says the stream "releases edges from the automaton as tokens". Lucene's text automata label their edges with Unicode code points (Automata.makeString, makeStringUnion, RegExp), but the token text is written with charAttr.append((char) currentEdge.value), which keeps only the low 16 bits of the int label. A label above U+FFFF therefore becomes a different, usually valid character: U+10400 comes out as U+0400, U+1F600 as U+F600, U+10000 as a NUL char, and U+0041 and U+10041 give the same token. Nothing fails.

The cast has been there since the class was added for FlattenGraphFilter tests (LUCENE-9963, 8.10.0), and all 41 releases that have the class fail the one-line test. The reach is small: in Lucene only TestAutomatonToTokenStream and TestFlattenGraphFilter use the class. A one-line fix, charAttr.append(Character.toString(currentEdge.value)), passes the repro, the core analysis tests, TestFlattenGraphFilter and :lucene:core:check -x test.

1. Labels are code points. Automata.makeString stores s.codePointAt(i) as each edge's label (Automata.java:555); makeStringUnion is "codepoint based (full unicode codepoints on transitions)" (:638-639); makeAnyChar, makeChar and makeCharRange speak in code points (:104, :116, :137).
2. One token per label value. toTokenStream walks the states in topological order and adds one EdgeToken per value of each edge's range: for (int val = t.min; val <= t.max; val++) ... edges.add(new EdgeToken(destLayer, val)) (AutomatonToTokenStream.java:97-101). The value stays an int (:174).
3. The cast. TopoTokenStream.incrementToken writes the text with charAttr.append((char) currentEdge.value) (:143); the shipped bytecode is i2c then CharTermAttribute.append(char). (char) 0x10400 is 0x0400.
4. Only the text is wrong. Positions, lengths and offsets come from the layers (:145-149), so the token graph has the right shape. Labels 0 to 0xFFFF (BMP code points or bytes) are exact.

Assertions on and off give the same result; with the fix every row is as expected.

• Who is hit. Any caller of AutomatonToTokenStream.toTokenStream whose automaton has labels above U+FFFF: emoji, historic scripts, CJK Extension B and later. It is a public class in an exported package of lucene-core, with no setting.
• What goes wrong. The token text is a different character, usually a valid one, so nothing looks broken. Distinct labels can give the same token (U+0041 and U+10041 both give A).
• Reach. In Lucene only TestAutomatonToTokenStream (core) and TestFlattenGraphFilter (analysis-common) use the class; no query parser, analyzer, filter, query or codec calls it. A GitHub code search (2026-10-28) found no use in Elasticsearch, OpenSearch or Solr; the other hits were copies of Lucene's source and ports.
• Limits. Deterministic; no exception, hang or resource growth. No in-tree index, query, analyzer or codec path is affected; an outside caller could still index or compare the wrong tokens. Labels up to U+FFFF are exact.
• Priority. Low: a silent wrong result, but in a utility nothing in production calls.

The test sits in org.apache.lucene.analysis, the package of the upstream TestAutomatonToTokenStream, but uses only public API. Nothing in it is randomized: seeds DEADBEEF, C0DE0869, 5EED0869, 7B0E0869 and 7B0D0869 give the same failure. On 8.10.0 to 9.0.0 the same test needs only the older import org.apache.lucene.util.LuceneTestCase.

As a regression test it is the same test: it fails on stock, passes with either fix, and passes :lucene:core:check -x test in the tree.

TestAutomatonToTokenStreamSupplementaryLabel > testSupplementaryLabel FAILED
    org.junit.ComparisonFailure: expected:<[𐐀]> but was:<[Ѐ]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:B0CDCC61D28C5A40]:0)
        at app//org.junit.Assert.assertEquals(Assert.java:117)
        at app//org.junit.Assert.assertEquals(Assert.java:146)
        at app//org.apache.lucene.analysis.TestAutomatonToTokenStreamSupplementaryLabel.testSupplementaryLabel(TestAutomatonToTokenStreamSupplementaryLabel.java:32)
:lucene:core:test (FAILURE): 1 test, 1  failure

• The contract is code points. The javadoc promises the edges as tokens and states one precondition, a finite DAG (AutomatonToTokenStream.java:40-47). The automaton package has a "Unicode alphabet" (util/automaton/package-info.java:33), Automaton.step documents "@param label codepoint to look up" (Automaton.java:654), and every text constructor labels with code points. No public automaton constructor (Automata, RegExp, ...) labels edges with UTF-16 code units (the package-private hunspell TrigramAutomaton labels with a char minus an offset, so at most 0xFFFF, TrigramAutomaton.java:55), and a byte label is at most 255. The raw-int entry points (Automata.makeString(int[], int, int), makeChar, makeCharRange, Automaton.addTransition, Automaton.Builder.addTransition) take any int with no range check, but the Automata ones are documented in code points ("from the specified unicode code points", Automata.java:614-617). So a label above U+FFFF that comes from text is a supplementary code point.
• Public API. public class AutomatonToTokenStream (:36) has no @lucene.internal or @lucene.experimental tag and sits in a package lucene-core exports (module-info.java:26). Its input type is experimental: Automaton, Automata and the automaton package carry @lucene.experimental (Automaton.java:43, Automata.java:44, package-info.java:44). That allows the API to change between releases; it does not make silently wrong text intended.
• Its own tests build code-point automata. Three of the six tests in TestAutomatonToTokenStream use Automata.makeStringUnion (:34, :51, :68), with ASCII strings only; the other three use Automaton.Builder with char-literal labels (:80-130).
• Elsewhere Lucene gets it right. Operations.getCommonPrefix appends each label with appendCodePoint (Operations.java:1176), and TestAutomaton.testCommonPrefixUnicode checks that emoji survive (:188-192). The sibling TokenStreamToAutomaton reads code points with codePointAt when unicodeArcs is set (:184-202).
• An oversight, not a decision. The cast came with the class in #157 (LUCENE-9963); its review never discusses char, code points, surrogates or label limits, and one reviewer wanted to run a Levenshtein automaton, which is code-point labelled, through it. The six later commits on the file leave these lines alone.
• Precedent. #15900 (merged 2026-04-06) fixed the same kind of UTF-16 mistake in TruncateTokenFilter, so it no longer produces half surrogates.

The strongest case for "intended": the class was written as test infrastructure for FlattenGraphFilter and has no production caller; Automaton, Automata and their package are @lucene.experimental; the javadoc never says how an int label maps to token text, and the one-token-per-range-value loop shows it was not designed for wide Unicode ranges, so the cast may reflect a "one edge, one UTF-16 char" assumption.

Why it fails: an experimental tag allows API changes, not silent corruption; the class is public and ships in lucene-core; the standard constructors, including the one three of its own tests use, make full code points from ordinary strings; and the cast turns U+10400 into an unrelated letter. It is about impact, which the low priority reflects, not about validity. The reviews that rate the argument explicitly rate it weak. The read-only fact check of this doc by another model family rates it moderate: every in-tree use is ASCII or byte input and labels are plain ints, so a maintainer could reasonably prefer a documented BMP-only contract, label validation or moving the class; it still judges the repro real and minor / rare / low right.

Two fixes were run through gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF); a third variant was checked earlier by the study.

Replace the cast in TopoTokenStream.incrementToken (AutomatonToTokenStream.java:143).

-      charAttr.append((char) currentEdge.value);
+      // labels are code points: append one char, or a surrogate pair above U+FFFF
+      charAttr.append(Character.toString(currentEdge.value));

Trade-offs

• One small String per token, negligible for this utility.
• A label that is not a code point (negative or above U+10FFFF) now throws IllegalArgumentException instead of becoming a wrong char. String and regex constructors never make one; a caller can still pass such an int through the raw-int entry points (Automata.makeChar, makeCharRange, makeString(int[], int, int), Automaton.addTransition, Automaton.Builder.addTransition; none checks the range). Which behaviour such labels should get is a choice for the maintainers: reject them, as this fix does, or document code points as the contract.
• Character.toString(int) exists since Java 11, so the same line backports to 9.x and 10.x, not to 8.x (Java 8), where the Character.toChars variant does the same.
• Offsets stay graph-layer indices, as for every token today (offAttr.setOffset(currentPos, currentEdge.destination), :149), not offsets into a text: with the surrogate-pair fix shadowed, the diagram probe shows U+10400 as a 2-char term with offsets 0-1; this fix changes only the same line.

What the checks validated

• Stock: the repro fails with assertions on and off.
• Fix: the repro passes with assertions on and off.
• :lucene:core:test --tests 'org.apache.lucene.analysis.*': 22 classes, 169 tests, 0 failures (the upstream TestAutomatonToTokenStream 6/6).
• :lucene:analysis:common:test --tests ...TestFlattenGraphFilter, the only other in-tree caller: 25 tests, 0 failures.
• :lucene:core:check -x test with the repro in the tree passes (google-java-format, ecjLint main and test, forbiddenApis, licenses, checksums).

The finder's fix: keep the cast for a BMP label, otherwise append Character.highSurrogate(value) and Character.lowSurrogate(value).

if (Character.isBmpCodePoint(currentEdge.value)) {
  charAttr.append((char) currentEdge.value);
} else {
  charAttr.append(Character.highSurrogate(currentEdge.value));
  charAttr.append(Character.lowSurrogate(currentEdge.value));
}

Trade-offs

No String per token (append(char) writes into the term buffer, which grows only when it is too small, CharTermAttributeImpl.java:57-66, :158-162), but for a value above U+10FFFF the JDK returns unspecified chars, so an invalid label still turns silently into wrong text. Six lines of code (plus a comment) replace one.

What the checks validated

Gradle: the repro 1/1 and TestAutomatonToTokenStream 6/6 pass. Earlier, shadowed onto the stock jars: the repro with assertions on and off, TestFlattenGraphFilter.testPathsNotLost 20/20 iterations, and the 6-case probe 6/6.

The study's first verifier shadowed the class on main a9779ef3c193 with this line instead of the cast.

charAttr.append(new String(Character.toChars(currentEdge.value)));

Trade-offs

Same behaviour as the recommended option (toChars also throws for a value that is not a code point), and it also compiles on Java 8.

What the checks validated

Shadowed onto the stock jars: the verifier's own 5 tests and the upstream TestAutomatonToTokenStream (6/6) pass. Not run through gradle or with the minimal repro.

Method: bounded-exhaustive translation validation (the study's semi-formal tier), in a study unit on token-graph and automaton conversion.

• An AI tool wrote an independent model of the construction the javadoc describes, with token text = the label's code point, and ran it next to the stock AutomatonToTokenStream on all 250,000 topologically numbered 4-state automata over the alphabet {a, U+4E2D, U+10400, U+10428}.
• The model matched the converter's structure everywhere (0 model mismatches). In the token-by-token comparison, token text differed on 187,744 automata, always at a label above U+FFFF. Against a patched class it differed on none (that patch also changed when the class throws: 0 exceptions instead of 54,720).
• The (char) cast was first spotted, and probed with an emoji ('x' + U+1F600 gives a truncated char), while the hunt planned this unit: the unit's plan already quotes line 143. So the run confirmed and measured a suspected defect; it is not a blind find.
• A public-API JUnit then witnessed it on the stock main jar; independent verifiers wrote their own tests, and the second adjudication re-ran everything on 645d4fe14a20.