ThaiTokenizer drops a number written right after a period, '#' or a currency sign other than ¢

ThaiAnalyzer, ThaiTokenizer and the thai tokenizer SPI silently drop a number when the JDK word BreakIterator returns it in one segment with a leading period, # or currency sign (any currency sign except the cent sign). ราคา ฿1,500 ("price ฿1,500") is analyzed as [ราคา], the Buddhist Era date พ.ศ.2567 as [พ, ศ], and page.42 as [page]; a document containing ประชุมวันที่ 1 ม.ค.2567 gets 0 hits for TermQuery(f:2567) instead of 1. The JDK segments correctly; ThaiTokenizer.incrementWord skips every segment whose first code point is not a letter or digit, so the whole number is thrown away. Nothing throws and the index is well formed. Every release from 4.8.0 (2014, LUCENE-4984) to 10.5.2 and main is affected (8 releases and main executed; all 107 release tags in that range contain the loop); a small change to the skip loop fixes every tested case.

• Segment: a piece of text between two boundaries of the JDK word BreakIterator for Thai. Some segments are words or numbers, others are spaces or punctuation.
• Number prefix: the JDK word rules' <pre-num> class, "the period, the number sign, and all currency symbols except the cents sign". Such a character goes into the same segment as the number after it, by the rule <pre-num>(<number><word>)*{<number>{<post-num>}}.

1. ThaiAnalyzer starts its chain with new ThaiTokenizer() (ThaiAnalyzer.java:101-102); the filters after it only see what the tokenizer emits.
2. The JDK splits พ.ศ.2567 into [พ][.][ศ][.2567], ราคา ฿1,500 into [ราคา][ ][฿1,500] and page.42 into [page][.42]. The cent sign is not a prefix: ราคา ¢50 gives [ราคา][ ][¢][50].
3. ThaiTokenizer.incrementWord skips "non-tokens" with while (end != BreakIterator.DONE && !Character.isLetterOrDigit(Character.codePointAt(buffer, sentenceStart + start, sentenceEnd))) (ThaiTokenizer.java:173-180): it tests the first code point of the segment only.
4. ., #, ฿ and $ are not letters or digits, so the whole segment, digits included, is skipped. The term is never created; the user-dictionary lookup (:186-207) also starts after the skip.

End to end, one document ประชุมวันที่ 1 ม.ค.2567: TermQuery(f:2567) counts 0 (expected 1); written ม.ค. 2567 it counts 1. At query time the same analyzer drops the number: the classic QueryParser turns +ราคา +฿1,500 into +body:ราคา, which matches any price.

Secondary symptoms of the same loop: a Latin word glued to a prefixed number is lost with it (#1word gives no token, because the JDK rule also absorbs (<number><word>)*); a prefix right after a digit starts its own segment, so 1#2, 1$2 and 1฿2 are analyzed as [1]; a segment that starts with a combining mark (่า, a tone mark with no base consonant, malformed Thai) gives no token.

• Who is hit. Users of ThaiAnalyzer, ThaiTokenizer, or the thai tokenizer through the SPI (CustomAnalyzer, factory configurations), with default settings. It is the Thai analyzer of analysis-common; outside its own package (where ThaiAnalyzer and ThaiTokenizerFactory create the tokenizer), no main Lucene code uses these classes. Downstream engines that expose Lucene's Thai analysis were not tested.
• Everyday-looking input. A price with its sign (ราคา ฿1,500, even after a space), an item number (ข้อ #3), a decimal with a leading period (.5), a date abbreviation followed by the year without a space (พ.ศ.2567, ม.ค.67), Latin text such as page.42 or No.5. Numbers that start with a digit (1,500, 1.5) and spaced dates are not affected. How often each form occurs in real Thai text was not measured.
• Silent and deterministic. No exception or warning; CheckIndex is clean and the stored value is intact. The loss happens in the tokenizer, one segment at a time, so it is the same for every occurrence, with -ea and -da, whatever the index size, on JREs that segment like OpenJDK 11, 17 and 25 (all three were run).
• Both sides. Indexed documents lack the number, so term, phrase and boolean searches for it miss them. In a query the number is dropped: +ราคา +฿1,500 matches any price, and a query of only ฿1,500 matches nothing.
• Not more than that. No crash and no corruption; indexing one document does not change another. One warm timing run, not a benchmark, showed no slowdown. At query time the dropped number can add false positives, as above.
• Reindex. Documents indexed before a fix lack the terms; a fix changes analysis output, so it needs a CHANGES/MIGRATE note.
• How common. Not measured on a corpus.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: token 2 does not exist
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:257)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:573)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:637)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:792)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:777)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:862)
       [...]

• Comment and test. // find the next set of boundaries, skipping over non-tokens (ThaiTokenizer.java:173), tested with Character.isLetterOrDigit (:176). .2567 contains four digits.
• Upstream tests. TestThaiAnalyzer.testDigits (TestThaiAnalyzer.java:131-135) checks that ๑๒๓๔ is indexed as 1234. No explicit upstream test puts a prefix character before a number (random-text tests may generate one but do not check which characters become tokens), and no test, javadoc, CHANGES or MIGRATE entry documents the drop. With it removed (fix A), the 56 upstream Thai tests pass, and all of analysis-common, 1,918 tests including the 9 new ones (18 skipped), runs with 0 failures.
• The JDK means these as numbers. BreakIteratorRules.java:100-102 and :136-139 (JDK 25): the prefix class is "punctuation that can occur at the beginning of a number", and the rule keeps a sequence "that starts with a number-prefix character and a number" together.
• Where the loop comes from. It copies ICUTokenizer's // find the next set of boundaries, skipping over non-tokens (rule status 0) (ICUTokenizer.java:209-214), which skips a span with no letter or number. java.text.BreakIterator has no rule status; the first-code-point test stands in for it (our reading; the commit and issue do not explain the choice).
• Before and beside. 4.7.2's ThaiAnalyzer (StandardTokenizer + ThaiWordFilter) gives [พ, ศ, 2567] and [ราคา, 1,500]; so does the 4.8.0 jar with matchVersion LUCENE_47. ICUTokenizer, named in the ThaiTokenizer javadoc as the portable alternative (ThaiTokenizer.java:35-37), gives [ค.ศ, 2024] and [page, 42]; StandardAnalyzer gives [page, 42]. LUCENE-4984 changed the chain to fix offsets with CharFilters; its discussion says nothing about numbers.
• Precedent. LUCENE-2404 (2010): a committer called the old ThaiWordFilter's first-character test "cheating with the UnicodeBlock check on charAt(0)". LUCENE-5076 (open): a first-code-point-only word test in ICU-module code, called a bug by a committer.

The case for "intended" (raised by the cross-model reviews, which rated it moderate, and examined by a third verifier, which found that it does not survive): ThaiTokenizer promises only that it "uses BreakIterator to tokenize Thai text" and warns that it may not be supported by all JREs; the JDK's own BreakIterator javadoc (the nextWordStartAfter example) calls a span a word only "If the characters between this boundary and the next boundary include at least one letter", and .2567 has none; and พ.ศ.2567 without a space is informal.

Why it fails:

• That idiom scans every character of the span; Lucene's loop scans one. And Lucene's test is letter or digit, not letter. Together, .2567 is a token.
• The JRE splits these inputs correctly: OpenJDK 11, 17 and 25 split every input of the table the same way. The loss is Lucene's.
• The currency case, ราคา ฿1,500 after a space, is a plausible everyday way to write a price (not checked against a corpus), so "informal input" does not cover it.
• "Twelve years and nobody complained": nobody discussed it or closed it as won't-fix either, and the loss is silent.

What a maintainer could still raise:

• Prefix-free terms are not a written contract, so which terms a prefixed number gives is a design choice, and changing it changes analysis output (reindex; CHANGES and MIGRATE notes).
• The input kinds differ: the price after a space is the hardest to call intended; a date without a space (พ.ศ.2567) and Latin or decimal forms (page.42, .5) leave more room for a style or token-policy argument.
• Against a shared design: in the same SegmentingTokenizerBase family, HMMChineseTokenizer has no first-character test, and the test tokenizer in TestSegmentingTokenizerBase skips forward to the first letter or digit, as the fix does.

Option A was validated with gradle in a scratch worktree at ea57360912ed (JDK 25.0.3) and through the repro runner; option C on the 10.5.2, 9.12.3 and main jars; option B was not run. With A, file the related LFV-229 (a user-dictionary match that ends inside a character cluster) separately: A changes its symptom but does not fix it.

Replace the first-code-point test with a scan of the segment: if no code point is a letter or digit, skip the segment as before; otherwise move start to the first letter or digit and emit from there. Results: ราคา ฿1,500 -> ราคา, 1,500 [6,11); พ.ศ.2567 -> พ, ศ, 2567 [4,8); ข้อ #3 -> 3 [5,6); page.42 -> 42 [5,7); #1word -> 1word [1,6). Punctuation-only and space segments are still skipped (ราคา - บาท -> [ราคา, บาท]), and every control is unchanged. A prefix after a digit is fixed too (1#2 -> [1, 2]). This is the finder's patch; an independent verifier wrote an equivalent one.

// ThaiTokenizer.incrementWord: replaces the loop at lines 174-180
int end = wordBreaker.next();
while (end != BreakIterator.DONE) {
  int s = start;
  while (s < end) {
    int cp = Character.codePointAt(buffer, sentenceStart + s, sentenceEnd);
    if (Character.isLetterOrDigit(cp)) {
      break;
    }
    s += Character.charCount(cp);
  }
  if (s < end) {
    start = s; // drop a leading '.', '#' or currency sign, keep the number
    break;
  }
  start = end;
  end = wordBreaker.next();
}

Trade-offs

• New terms, so reindex. Old indexes lack the numbers. Mixing analyzers changes results: after the fix, ราคา ฿1,500 in a query analyzes to ราคา and 1,500, which an old index cannot match. Positions after a recovered number shift by one. Needs a CHANGES entry and a MIGRATE note.
• The token excludes the prefix (1,500, not ฿1,500), as ICUTokenizer and StandardTokenizer produce for these inputs.
• User dictionary. The lookup starts at the trimmed start: with {1,500บาท}, ฿1,500บาท gives 1,500บาท [1,9) (stock: [บาท]). An entry that itself starts with a prefix character (฿1,500) still never matches, as on stock.
• Every leading non-letter/digit is trimmed, not only ., # and currency signs. A segment that starts with a combining mark now yields the rest: ้ำ gives ำ and ่า gives า (stock: nothing); ICUTokenizer gives the same ำ and า for these inputs. Whether a stray tone mark should leave its vowel as a token is its own question. With {แม่น}, แม่น้ำ gives [แม่น, ำ] instead of [แม่น]; that is LFV-229, which needs its own fix.
• Cost. A segment that starts with a letter or digit costs one check, as now; a punctuation or space segment is scanned to its end (usually one character). Not benchmarked.

What the checks validated

• The repro: 3 of 4 fail on stock, 4 of 4 pass with the fix (gradle; the repro runner with -ea).
• The fix check: 3 of 5 fail on stock, 5 of 5 pass with the fix, with -ea and -da.
• :lucene:analysis:common:test --tests 'org.apache.lucene.analysis.th.*': 65 tests (56 upstream + 9 new) pass with -ea (seed 1A2B3C4D) and with -Ptests.asserts=false (seed 77AA77AA).
• All of :lucene:analysis:common:test: 1,918 tests including the 9 new ones, 18 skipped, 0 failed. TestRandomChains (:lucene:analysis.tests): 2 tests pass; random chains need not pick ThaiTokenizer.
• checkGoogleJavaFormat passes on the patched ThaiTokenizer.java.
• The finder's bounded-exhaustive check: 13,107 of 111,110 strings of up to 5 symbols lose a letter or digit on main, 0 with the fix. Its 10-symbol alphabet has no #, currency sign or supplementary character, and it checks that letters and digits land in some token, not that the tokens are right; the fix-check oracle adds #, ฿ and $.
• Scope: the suites ran on main only (OpenJDK 25.0.3, Linux aarch64, one seed per assertion mode). On releases the fix was checked only by the history step's 4.8.0 jar with the loop replaced.

Skip only segments without any letter or digit, but keep start, so the token is ฿1,500 or .2567.

Trade-offs

A smaller change in offsets, but TermQuery(f:2567) still misses พ.ศ.2567, and the terms differ from ICUTokenizer and StandardTokenizer. Not recommended.

What the checks validated

Not run.

Without patching Lucene, put a PatternReplaceCharFilter in front of the ThaiTokenizer chain that inserts a space between a prefix character and a digit, unless the prefix follows a digit (so 1.5 is untouched). Switching to ICUTokenizer (analysis-icu) also keeps the numbers, but segments Thai differently, so it is a reindex with different terms.

Pattern p = Pattern.compile("(?<!\\p{N})([.#\\p{Sc}])(?=\\p{N})");
// in Analyzer.initReader(field, reader):
return new PatternReplaceCharFilter(p, "$1 ", reader);

Trade-offs

Restores the number of every prefixed input in the table, with the same offsets as option A, and needs no Lucene change. But it is an application-side chain that must be used at index and at query time, documents already indexed need a reindex, it leaves a prefix after a digit alone (1#2, 1$2, 1฿2 still lose the 2), and it also changes the segmentation of other ./#/currency-digit pairs that follow a non-digit.

What the checks validated

On 10.5.2, 9.12.3 and main, with ThaiTokenizer + LowerCaseFilter + DecimalDigitFilter + StopFilter(ThaiAnalyzer.getDefaultStopSet()): all 11 prefixed inputs of the table get their number back with the same offsets as option A; 1.5, 50%, ราคา ¢50, ราคา 1,500 บาท and พ.ศ. 2567 are unchanged. On main it also works in front of a copy of the full stock ThaiAnalyzer chain, with ThaiCharFilter and the 10.6 ThaiNormalizationFilter and ThaiRepeatFilter; that copy without the extra char filter gives exactly ThaiAnalyzer's tokens and offsets.

Method: targeted exploration, then a bounded-exhaustive check (the hunt's non-formal tier; no formal tool produced the first signal).

• The finder explored ThaiTokenizer with user-dictionary words that contain periods (พ.ศ.), noticed that พ.ศ.2567 lost its year, and wrote a public-API JUnit: 3 of 4 tests fail on main with -ea and -da, and on 10.5.2 and 9.12.3.
• It then enumerated all 111,110 strings of 1 to 5 symbols over {ก, า, 1, x, '.', ',', '-', space, U+0E48, U+0E4C} with the oracle "every letter or digit lies inside some token": 13,107 strings lose one on main (3,927 after a period, the rest after a tone or cancellation mark), 0 with the fix. The alphabet has no # or currency sign, and the oracle checks coverage, not that the tokens are right. That oracle is stronger than anything Lucene documents; the verdict rests on the number cases.
• An independent reviewer from another model family (OpenAI GPT-5.6) ran its own checks (release.2026 gives [release], ปี.2026 gives [ปี]). Claude verifiers wrote their own tests: the independent reproduction (ค.ศ.2024, ม.ค.67, page.42, a search) and the intended-behaviour check (฿1,500, $100, #3), which also found the JDK's number-prefix rule. An adjudication re-ran both with fresh seeds.