ThaiCharFilter leaves a typo in place when a zero-width character or a repeated tone mark sits inside it, so ThaiAnalyzer merges words

ThaiCharFilter, new in 10.6.0 and applied by the default ThaiAnalyzer before tokenization, exists because Thai typos such as a double Sara E or a zero-width character stop the JDK dictionary BreakIterator from finding word boundaries. It fixes each documented typo on its own. But it makes one pass and tests its multi-character rules (double Sara E, decomposed Sara Am, tone mark before an above/below vowel) on the raw next characters, while zero-width removal and deduplication act only on the character at the front of its buffer. When a zero-width character or a second copy of a tone mark sits inside such a pattern, the pattern is never rewritten, and the word splitter merges the word with its neighbour: a correctly typed neighbour such as บ้าน becomes unfindable. Silent, deterministic, in 10.6.0 (released 2026-10-29).

• Tone marks Mai Ek ่ (U+0E48) and Mai Tho ้ (U+0E49) sit above a consonant, after any above/below vowel such as Sara Ii ี (U+0E35): ที่ is ท + ี + ่. Typing the tone before the vowel (ท่ี) or twice (ที่่) is among the "common keyboard typing errors" the issue that proposed the class lists.
• Sara Ae แ (U+0E41) can be typed as two Sara E เเ (U+0E40 U+0E40), which look the same. Sara Am ำ (U+0E33) is often typed decomposed, as Nikhahit ํ (U+0E4D) + Sara Aa า (U+0E32).
• Zero-width characters (ZWSP U+200B, ZWJ U+200D, ...) are invisible, so a stray one is easy to miss.
• The JDK's Thai dictionary BreakIterator finds word boundaries; any of these slips can make it miss one and glue words together. ThaiCharFilter repairs the text first, character by character, so that it does not.

Through new ThaiAnalyzer(): ฉันทำงานท่่ีบ้าน gives [ฉัน 0-3, ทำ 3-5, งาน 5-8, ที่บ้าน 8-16] instead of [ฉัน, ทำ, งาน, บ้าน]; ฉันรักเ+ZWJ+เมวมาก gives [ฉัน 0-3, รักแมวมาก 3-14] instead of [ฉัน, รัก, แมว]; ฉันทํ+ZWSP+างานที่บ้าน gives [ฉัน, ทำงานที่บ้าน] instead of [ฉัน, ทำ, งาน, บ้าน]. A TermQuery for บ้าน or แมว finds 0 documents instead of 1. A second pass of the char filter cleans every char-filter row.

ThaiCharFilter.read() (ThaiCharFilter.java:71-162) looks at the first character of a buffer of up to 4 characters (the current one and up to 3 ahead) and tries, in order: drop it if it is zero-width (:89-95); drop it if it is a diacritic equal to the last character emitted (:97-103); join a double Sara E (:105-113); recompose Sara Am (:115-137); swap a tone with the above/below vowel after it (:139-147); else emit it. The joining, recomposing and swapping rules read the raw next one or two characters. The cleaning rules only run when a character reaches the front, after the character before it has been decided and emitted, and nothing revisits a character already emitted.

Trace for ท ่ ่ ี: ท is emitted. The first ่ is at the front; the next character is the second ่, not a vowel, so no swap; ่ is emitted. The second ่ reaches the front, equals the last emitted character, and is dropped. ี is emitted after the first ่. Output ท ่ ี. For เ ZWJ เ, the join rule sees ZWJ next and emits เ; ZWJ is dropped; the second เ is emitted. For ํ ZWSP า, the Sara Am rule sees ZWSP and emits ํ.

Inside the merged token the spelling looks clean (ที่บ้าน) because ThaiNormalizationFilter runs after the tokenizer; the word boundary is already lost.

• Who is exposed. Every Thai field that uses new ThaiAnalyzer() (it wraps the reader in ThaiCharFilter for indexing and for Analyzer.normalize()), and custom chains that put one thai char filter (SPI name of ThaiCharFilterFactory) before a dictionary-based word tokenizer such as the thai tokenizer, as the factory's javadoc example does. No option is involved.
• What it takes. Two of the documented typos in one character cluster: a tone mark typed twice and before its above/below vowel, or a zero-width character between two Sara E, between a tone and its vowel, or inside a decomposed Sara Am. Each typo alone is fixed. Likelihood: expected to be rare, because two typos must meet in one cluster (a judgement; no frequency was measured).
• What breaks. The affected word and a neighbour become one term, so queries for either word miss the document. Queries go through the same analyzer, so a query matches the merged term only if its text produces the same merge. No error and valid offsets; IndexWriter accepted all 30,000 random strings of a security probe (no CheckIndex was run); deterministic at any index size.
• How often in a bounded space. Over 824,691 inputs over 21 symbols (every string of up to 5 symbols that starts with ก, น, ฤ or เ, plus every string of up to 3 symbols), 3,633 inputs made of a clean word plus documented typos keep a pattern the filter's own rules rewrite (approximate: the reference uses a simplified Thai cluster grammar). Over all 813,615 strings of length 1-5 over 15 symbols, removing the zero-width characters changes today's output on 13,282, and typing one diacritic twice changes it on 4,813 of 85,236 checks.
• Not a regression. 10.5 had no char filter, and the raw JDK segmentation merges the lead sentence too. It is an incomplete new feature, found before its first release.
• JDK dependence. The word merge depends on the JDK's Thai dictionary BreakIterator (all runs OpenJDK 25.0.3); the char-filter outputs do not.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<ท[ี่]> but was:<ท[่ี]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.th.TestThaiCharFilterRepeatedToneBeforeVowel.testRepeatedToneBeforeVowel(TestThaiCharFilterRepeatedToneBeforeVowel.java:29)

• Purpose (ThaiCharFilter.java:27-30): "Applying normalization before tokenization is critical for Thai, because typographic anomalies (such as two consecutive Sara E characters instead of Sara Ae, or zero-width characters) prevent the dictionary-based BreakIterator from locating word boundaries, causing multiple adjacent words to be mistakenly merged into one token."
• Rules (:32-43): removes zero-width characters, replaces double Sara E with Sara Ae, recomposes decomposed Sara Am, deduplicates consecutive repeated diacritics and tone marks, reorders a misplaced tone mark before an above/below vowel. Each lead output still holds a pattern that one of the multi-character rules (double Sara E, Sara Am, reordering) rewrites.
• No documented order explains it. Zero-width removal is first in the javadoc and the code; deduplication is before the reordering in both. In ท่่ี the literal input already contains "tone + above vowel" (่ ี) and the output still does. What does explain it is the code's per-position priority: at each position the first matching rule wins and a rejected pair is not re-tested. That is how the code works; no javadoc, test or PR comment says a rule may be skipped because a character between its two halves is about to be deleted.
• MIGRATE.md (10.5 to 10.6) lists decomposed Sara Am, doubled Sara E, repeated or misordered tone marks and zero-width characters together as input 10.6 now analyzes differently.
• Tests. The upstream TestThaiCharFilter checks each typo alone, including ฉันรักเเมวมาก -> [ฉัน, รัก, แมว, มาก], the sentence whose ZWJ variant merges; testRandom checks offsets, not normalization. No test asserts today's two-typo output; #16716 / #16717 never discuss rule order.
• Concession. Two other sub-cases depend on order: Lakkhangyao after Nikhahit (ทํๅ) and a doubled tone inside a decomposed Sara Am (นํ้้า). Read as passes in the javadoc's list order they stay decomposed, and ThaiNormalizer agrees. They are not the lead, and they overlap the sibling ThaiNormalizer bug (LFV-873).

The strongest case for "works as designed" (the cross-model reviewer and the intended-behaviour checker): the class is a bounded, single-pass typo corrector; its javadoc lists separate repairs and promises neither idempotence nor repair of two typos in one cluster; the upstream tests check one typo at a time; the code's per-position first-match priority explains every output; and the raw JDK segmentation (the pre-10.6 path, no char filter) merges these inputs too.

Why it fails for the lead cases: no documented order explains the output, and the per-position priority is how the code works, not a documented limit; the filter itself deletes the zero-width character or the duplicate and then leaves a pattern its own rule rewrites; the result is the merge the class says it exists to prevent, on the upstream test sentence plus one invisible character; ThaiNormalizer from the same PR normalizes the same inputs (except the ZWJ one: it documents removing only ZWSP and ZWNJ); and nothing documents a single-typo limit.

One fix (A) was built and validated through gradle and an exhaustive differential; B is untested, C is a workaround, D only documents the limit.

Wrap the input, in the constructor, in a private BaseCharFilter that removes zero-width characters, drops a diacritic that repeats the previous character and replaces Lakkhangyao with Sara Aa (unless after Ru or Lu). read() is unchanged: its join, recompose and reorder rules now see cleaned characters. Its own copies of the cleaning rules stay; deduplication still matters after a tone/vowel swap (ก่ั่ั). Each stage keeps its own offset map and CharFilter.correctOffset chains into the inner one (CharFilter.java:72-77). The idea is the finder's sketch, cleaned up.

public ThaiCharFilter(Reader in) {
  // the one-character rules run first, so read() matches its
  // multi-character patterns against cleaned look-ahead
  super(new SingleCharRules(in));
}

private static final class SingleCharRules extends BaseCharFilter {
  // read(): drop U+200B/U+200C/U+200D/U+FEFF and a diacritic equal to the
  // previous char (addOffCorrectMap), map U+0E45 to U+0E32 unless after
  // U+0E24/U+0E26; read(char[], int, int) loops over read()
}

Trade-offs

• Also recomposes the two order-dependent sub-cases (ทํๅ -> ทำ, นํ้้า -> น้ำ), where ThaiNormalizer keeps the decomposed form today (LFV-873); fixing that bug the same way keeps the two classes in agreement.
• Residual: interleaved repeats (ก่ั่ั) and two different tone marks before one vowel still change on a second pass; no documented rule covers two different tone marks.
• Terms change only for inputs with such a typo pair; 10.6.0 (released 2026-10-29) has the class, so indexes built with it, with snapshots or with the RC hold the merged terms and need reindexing for those words after the fix (a MIGRATE note).
• One more Reader layer with a per-character read(); no benchmark was run.

What the checks validated

• Gradle, scratch worktree of a9779ef3c193, seed 876C0A10: all 10 Thai test classes (63 tests): stock 6 failures (the new test), fix A 0 with assertions on and off; the whole :lucene:analysis:common:test task with fix A: 340 classes, 1,916 tests, 18 skipped, 0 failures; check -x test after tidy: pass (google-java-format, ecjLint, forbiddenApis, license checks).
• Repro harness on main, fix A shadowed: the new test 7/7 (-ea, -da), the finder's 10/10 (-ea), the independent reproducer's 9/9 (-da).
• Exhaustive differential, stock vs fix A, all 813,615 strings of length 1-5 over 15 symbols: outputs differ on 36,718; on 36,654 of them a second stock pass still changes the stock output, and the other 64 (three or more Sara E around a zero-width character) get what stock gives without the zero-width character. Removing zero-width characters changes the output on 13,282 inputs with stock, 0 with fix A; typing one diacritic twice changes it on 4,813 of 85,236 checks with stock, 0 with fix A. In this space fix A is never less stable than stock (a second pass changes 6,888 fix outputs, all also unstable with stock; stock has 42,588). Offsets valid for every input, both versions.
• The hunt's bounded check with the finder's sketch (same rules): typo-input mismatches 3,633 -> 51.

Apply the three one-character rules in fillBuffer(), in the same class. Not run.

Trade-offs

Offset corrections are keyed by output offset, which is not known when a character is buffered (a later contraction can shift it), so the deletions must be recorded later; more intricate than a second stage.

new ThaiCharFilter(new ThaiCharFilter(reader)), or two thai char filters in a custom chain. Two chained filters give the clean output for all five char-filter inputs of the independent reproducer.

Trade-offs

Twice the work; not complete (732 inputs of the bounded check need 3 changing passes); ThaiAnalyzer is final, so users either rebuild its chain or wrap it in an AnalyzerWrapper whose wrapReader and wrapReaderForNormalization add a second ThaiCharFilter (not run).

Say in the javadoc that only one typo per character cluster is repaired.

Trade-offs

Keeps the word merges the class exists to prevent.

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier), a hunt unit on Thai normalizers, planned from two probes (an idempotence check of ThaiNormalizer and a char-filter versus normalizer differential).

• The documented rules were encoded as a string-rewriting system; the reference computes every normal form of each input by exhaustive rewriting. The stock filter was compared with it on 824,691 inputs over 21 symbols (including a surrogate pair): every string of up to 5 symbols that starts with ก, น, ฤ or เ, plus every string of up to 3 symbols; with identical counts with assertions on and off.
• 3,633 inputs made of a clean word plus documented typos, whose documented result is unique, got an output that one of the filter's own rules still rewrites. Iterating the filter to a fixed point brought these 3,633 to 0, which pointed at the single-pass look-ahead.
• A second leg applied every single and paired typo to 16 real sentences and ran them through the stock ThaiAnalyzer: every leftover of the char filter's own rules (104 variants) came from a pair of typos; no single typo the char filter documents left one. (Single typos of rules that only ThaiNormalizer documents can merge words too; out of scope here.)
• The finder then wrote a public-API JUnit on the stock main jar (10 of 10 fail).