ThaiTokenizer user dictionary: a word that ends inside a character cluster makes the vowel or tone mark after it disappear

The new ThaiTokenizer user dictionary (GITHUB#16722, on main, branch_10x and branch_10_6; first released in 10.6.0) accepts a dictionary word as a match even when it ends inside a Thai character cluster, right before a combining vowel or tone mark that belongs to the previous consonant. The tokenizer then restarts its word BreakIterator at that point, the rest of the text starts with the mark, and the old skip loop for segments that do not start with a letter or digit throws the mark away. With {แม่น} ("accurate"), แม่น้ำ ("river") becomes the single token แม่น[0,4), so "river" and "accurate" become the same term; with {ผล}, ผลิตภัณฑ์ ("product") becomes [ผล, ต, ภัณฑ์]. Queries are analyzed the same way, so a search for แม่น้ำ also finds a document that has แม่น but not แม่น้ำ (ยิงแม่นมาก, "shoots accurately"). Nothing throws, Lucene's analysis consistency checks accept the stream and the index is well formed. A 13-line guard in the dictionary loop fixes every tested case, and a character-BreakIterator variant does the same; with either, the analysis-common suite has 0 failures (18 tests skipped).

• Character cluster: a consonant plus the combining vowels and tone marks written above or below it, such as น้ำ (น + tone mark U+0E49 + SARA AM U+0E33). The marks are Unicode combining marks (category Mn); SARA AM is a letter (Lo) that Unicode's grapheme rules also attach to the cluster before it (Grapheme_Cluster_Break=SpacingMark).
• User dictionary: the CharArraySet passed to ThaiTokenizer(CharArraySet ...) or loaded by the thai tokenizer factory's dictionary parameter (GITHUB#16722). A dictionary word "takes precedence over default segmentation boundaries" (class javadoc).

1. The dictionary loop (ThaiTokenizer.java:186-207) tries prefix lengths from longest to shortest and takes the first userDictionary.contains(...) hit (:191-192). It never looks at the character after the match.
2. If the match does not end where the word BreakIterator would end the word, and that point is not a word boundary, it sets reanchor (:200-201). The token [start, start + l) is emitted (:209-212): แม่น for แม่น้ำ.
3. The re-anchor (:214-218) hands the rest of the sentence to the BreakIterator as new text. For แม่น้ำ the rest is ้ำ, which starts with the tone mark.
4. On the next call the skip loop (:173-180, from 2014) sees a first segment that starts with U+0E49, not a letter or digit, and skips it whole. At the end of the text that segment is ้ำ, so SARA AM is lost too; mid-text, SARA AM can start the next token (ำเจ้าพระยา), and consonants after a mark survive as one-letter tokens (ต, ด, ง).

Once the re-anchor happens, the rest always starts with the mark, whatever the JDK; how the JDK splits that rest decides only what else is lost. In the tested texts the word BreakIterator put no boundary inside a cluster, so without a dictionary they lose nothing. A separate, older effect, not this bug: without a dictionary, text with no space or sentence break for 1,024 characters is cut where the tokenizer's buffer fills, and a mark right at that cut is dropped too (the same on 10.5.2; neither fix changes it).

At each checked cut point (แม่น้ำ, แม่น้ำเจ้าพระยา, แม่น้ำไป at 4; ที่นี่, ผลิต, ผลิตภัณฑ์, คนึงหา, กล้ำกลืน at 2) neither the Thai word BreakIterator nor the character BreakIterator has a boundary; at the cut points of the two between-cluster controls the character BreakIterator does. In a hand-built copy of ThaiAnalyzer's filter chain with the dictionary passed to the tokenizer, the symptoms survive (แม่น้ำ gives [แม่น], แม่น้ำเจ้าพระยา gives [แม่น, ำเจ้าพระยา]), and a clipped stopword escapes the stop filter: ที่ is a default stopword, so ที่นี่ gives [ที, นี่] instead of [นี่]. (เขาเก่งมาก only fragments, [เก, ง] instead of [เก่ง]; เก่ง is not a stopword.)

End to end: index แม่น้ำ (river) and ยิงแม่นมาก (shoots accurately) with {แม่น}; QueryBuilder.createBooleanQuery("f", "แม่น้ำ") is f:แม่น and counts 2 documents instead of 1. The reverse also happens: แม่น้ำไป is indexed as [แม่น, ำไป], so a query for ไป (go) misses it.

• Who is hit. Users of the ThaiTokenizer(CharArraySet ...) constructors or of the thai tokenizer factory with dictionary (CustomAnalyzer, factory configs). The stock ThaiAnalyzer builds new ThaiTokenizer() without a dictionary. The feature is in 10.6.0 (released 2026-10-29) and on main, branch_10x and branch_10_6; none of the downstream engines searched exposes it yet.
• What triggers it. A dictionary entry that is the beginning of a longer word in the text and is directly followed by a combining vowel or tone mark (or SARA AM), with no longer entry covering that word. Thai above and below vowels and all tone marks are combining marks, so short entries hit common words: ที/ที่, ผล/ผลิต, สม/สมุด, แม่น/แม่น้ำ, เก/เก่ง. Lucene's own 10.6 migration notes list ผล among the "common content words" removed from the default stopwords. Long domain terms (the feature's examples พารากอน, คนขับรถ) should trigger it less often, and a dictionary that also lists the longer words avoids the cut for each word it lists ({ผล, ผลิต} keeps ผลิต); both points are reasoning, not measured on a real dictionary.
• What goes wrong. Marks disappear from the index; one-letter tokens (ต, ด, ง) and tokens that start with SARA AM (ำเจ้าพระยา) appear; different words collapse into one term (แม่น้ำ and แม่น). Queries go through the same analyzer: a query can over-match ("river" finds "accurate") or miss a word glued after a trigger word; a clipped stopword escapes the stop filter.
• Silent and deterministic. No exception or warning; Lucene's analysis consistency checks accept the stream, IndexWriter accepts the offsets, CheckIndex with slow checks is clean and stored text is intact (security probe, 13 documents). It happens per token in the analyzer, so it does not depend on the index; seen with -ea and -da.
• Timing. First released in 10.6.0 (2026-10-29); before that only users of snapshots or of the branches could have built an index with it.
• Not more than that. In the security probe: no crash or hang (18 adversarial runs, 6 texts x 3 dictionary settings, one of them no dictionary, each finish in under 0.4 s), nothing from other documents.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<แม่น[้ำ]> but was:<แม่น[]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.th.TestThaiTokenizerUserDictionaryClusterSplit.testDictionaryWordEndingBeforeToneMark(TestThaiTokenizerUserDictionaryClusterSplit.java:42)

• The contract. ThaiTokenizer.java:31-33: "Supports an optional user dictionary ({@link CharArraySet}) for custom or domain-specific words. When a user dictionary word matches, it takes precedence over default segmentation boundaries." It gives the dictionary power over where words split, not over whether characters are indexed. The contract is implicit.
• The goal of the feature. Issue #16721: domain terms "are frequently fragmented into unnatural sub-words or single characters, making exact search matching and phrase queries difficult or imprecise". This bug makes such fragments (ต, ด, ง).
• No boundary there. At each checked cut point the Thai word BreakIterator and the character BreakIterator both report isBoundary = false; without a dictionary none of the tested texts loses a character (the older buffer-cut loss is a separate path).
• Untested branch. All four upstream dictionary cases (TestThaiTokenizer, TestThaiTokenizerFactory) end on a word boundary or at the end of the text, so the re-anchor branch is never run by them. TestRandomChains can pass a random dictionary of a-z strings to the constructor, but it checks stream consistency only, which these streams pass. CHANGES and the 10.6 migration notes describe the dictionary as an additive feature, without caveats.
• The intended part survives a fix. {แม่} on แม่น้ำ gives [แม่, น้ำ] and {ผล} on ผลไม้ gives [ผล, ไม้], with and without either fix.
• Precedent. CHANGES: "LUCENE-3358, LUCENE-3361: StandardTokenizer and UAX29URLEmailTokenizer wrongly discarded combining marks attached to Han or Hiragana characters". On LUCENE-3358 a committer wrote: "we shouldn't be splitting a base character from its combining mark here". The Kuromoji user dictionary rejects illegal segmentation at load time (LUCENE-8933).

The case for "intended" (raised by the cross-model reviews, which rated it moderate, and examined in detail by the devil's-advocate verifier; none accepted it): CharArraySet.contains is an exact code-unit match, so แม่น does "match" at the start of แม่น้ำ, and a match takes precedence. The code re-anchors on purpose when a match ends where the BreakIterator has no boundary. The mark is then dropped by the 2014 loop that skips non-letter segments, and ThaiNormalizer also removes dangling marks at a token start. ThaiTokenizer promises no grapheme safety, and a short common word that prefixes other words misuses a dictionary meant for "custom or domain-specific words".

Why it fails:

• No reading of "precedence" removes text. Even if the split ผล|ิต were allowed, the vowel would have to land in some token; instead it vanishes, and "river" and "accurate" analyze to the same terms.
• Nothing in the code, the PR or the tests says the re-anchor is meant for ends inside a cluster. The ends it is seen to handle correctly are between clusters (the controls), and those keep working with a fix.
• The skip loop was written for punctuation and spaces; in the tested texts it sees a word's own mark only because the re-anchor restarts the iterator inside a cluster. ThaiNormalizer's rule is for marks already detached in the input, and in a copy of the ThaiAnalyzer chain it does not repair these tokens (ำเจ้าพระยา survives).
• Nothing restricts dictionary contents: the factory loads any word list, and Lucene itself lists ผล among common content words. Through the factory, {ผล} breaks the long word ผลิตภัณฑ์ ("product"), which the stock tokenizer keeps whole.

Still open for upstream: whether such a match should be rejected (both fixes) or extended to the end of the cluster is a design choice. Avoid สมิธ and คนึงหา as headline examples: the stock tokenizer already fragments them.

Fix A and fix B were each validated with gradle in a scratch worktree at ea57360912ed (OpenJDK 25.0.3) and through the repro runner; option C was not run. File the related LFV-225 (the skip loop drops a number after a leading ., # or currency sign) separately: its fix alone does not fix this bug, and the two fixes apply cleanly together.

Before testing a candidate length l, look at the character right after it; if it is a combining mark (NON_SPACING_MARK, COMBINING_SPACING_MARK, ENCLOSING_MARK) or SARA AM, try the next shorter length. With no dictionary hit left, the word that the BreakIterator found stands. Every symptom row becomes the no-dictionary or control result, and the intended splits ({แม่} on แม่น้ำ, {ผล} on ผลไม้) are unchanged. This is the finder's patch.

/** true if {@code ch} cannot start a character cluster (combining mark, or Thai SARA AM). */
private static boolean continuesCluster(char ch) {
  int type = Character.getType(ch);
  return type == Character.NON_SPACING_MARK
      || type == Character.COMBINING_SPACING_MARK
      || type == Character.ENCLOSING_MARK
      || ch == '\u0E33';
}

// in incrementWord, at the top of the dictionary loop:
// a dictionary word must not end inside a character cluster (before a combining mark)
if (l < remaining && continuesCluster(buffer[sentenceStart + start + l])) {
  continue;
}

Trade-offs

• A hand-written rule: three mark categories plus SARA AM, tested per UTF-16 char, so a mark outside the BMP would not be seen (all Thai marks are in the BMP).
• Cost: one Character.getType per candidate length; not benchmarked.
• Output change only for dictionary configurations that hit the bug; the feature first shipped in 10.6.0, so only indexes built with 10.6.0 or a snapshot can need a reindex.
• Does not change the older no-dictionary loss at buffer cuts.

What the checks validated

• The repro: 4 of 5 fail on stock, 5 of 5 pass with the fix (gradle; the repro runner with -ea and -da).
• org.apache.lucene.analysis.th.*: 10 suites, 61 tests (56 upstream + 5 new) pass with -ea (seed 1A2B3C4D) and -Ptests.asserts=false (seed 77AA77AA).
• All of :lucene:analysis:common:test (-ea): 1,914 tests including the 5 new ones, 18 skipped, 0 failed; TestRandomChains: 2 tests pass; checkGoogleJavaFormat passes.
• The finder's random dictionary harness (about 156,000 checks): 25 violations on stock, none with the fix; the other verifiers' JUnits pass with it.
• With the LFV-225 fix on top: all 65 Thai tests (56 upstream + both repros) pass.

Keep a BreakIterator.getCharacterInstance(th) clone with its own CharArrayIterator, set to the sentence in setNextSentence and again on re-anchor, only when a dictionary is configured. After a dictionary hit, if (l < remaining && !clusterBreaker.isBoundary(start + l)) continue;. Gives the same tokens as A on all 14 probe cases.

if (userDictionary.contains(buffer, sentenceStart + start, l)) {
  if (l < remaining && !clusterBreaker.isBoundary(start + l)) {
    continue; // the word would end inside a character cluster (before a combining mark)
  }
  ...

Trade-offs

• Follows the Unicode grapheme rules as the JDK implements them instead of a list; the cross-model reviewers called this the more general form. Behaviour beyond Thai text was not tested.
• A second BreakIterator per tokenizer with a dictionary, and a setText per sentence and per re-anchor; not benchmarked.

What the checks validated

• The repro 5 of 5 with -ea and -da; 61 Thai tests with -ea and -da; all 1,914 analysis-common tests with -ea (18 skipped, 0 failed); TestRandomChains; checkGoogleJavaFormat.
• The random dictionary harness: 156,356 checks, no violations.

Keep the dictionary match but move its end past the marks.

Trade-offs

The token is then no longer the dictionary word ({ผล} on ผลิต would give ผลิ, ต), so it keeps the text but can still produce fragments. Not recommended.

{ผล, ผลิต} keeps ผลิต whole, because the longest match wins.

Trade-offs

Covers only the words listed; for anyone using the feature on 10.6.0 or a snapshot.

Method: property-based testing with random user dictionaries (the hunt's non-formal tier; no formal tool produced the first signal).

• The finder's harness built random texts and random dictionaries over a small Thai vocabulary and checked two oracles: every letter or digit of the text lies inside some token, and a token that starts where a dictionary word matches is the longest such match that ends on a character-cluster boundary. On stock main: 25 violations in about 156,000 checks (19 lost characters, 6 precedence; the precedence oracle uses the same cluster rule as the fix, so only the 19 are independent of it). With the fix: none. The harness's dictionaries include truncated non-words; the realistic examples were then chosen by hand.
• An independent reviewer from another model family (OpenAI GPT-5.6, refute lens) ran its own test ({ผล} on ผลิต) on main with -ea and -da over seven seeds, with two controls, and the fix. A second GPT-5.6 review (works-as-designed lens) could only read the code; a separate run re-ran the finder's JUnit for it.
• Claude verifiers wrote their own tests: the independent reproduction (ที่นี่, เก่ง, คนึงหา, กล้ำกลืน, แม่น้ำ; 5 of 7 fail) and the intended-behaviour check, a devil's advocate, whose intent probe recorded 11 facts on main, including the factory case and that no BreakIterator has a boundary at the cut. An adjudication re-ran both with fresh seeds.