ThaiNormalizer leaves a decomposed Sara Am when a second typo sits inside it

ThaiNormalizer (new in 10.6.0 with ThaiNormalizationFilter, which the default ThaiAnalyzer uses from 10.6.0) documents that it recomposes a decomposed Sara Am, Nikhahit + optional tone mark + Sara Aa, into Sara Am. It also converts Lakkhangyao to Sara Aa, removes a repeated tone mark and keeps the last of two different tone marks. It runs these rules as single forward passes: the recomposition test looks at the next characters as typed, a Lakkhangyao is converted only when the loop reaches it, and an extra tone mark is removed only in the last step. Nothing tests for a decomposed Sara Am again, so a second typo inside one leaves the decomposed form in the output. Through the default analyzer, น้ำ ("water") typed as น + Nikhahit + Mai Ek + Mai Tho + Sara Aa is indexed as นํ้า, and a TermQuery for น้ำ finds 1 of 2 documents; the same marks with Mai Ek first give น้ำ. With the filter on its own, ท + Nikhahit + Lakkhangyao and a doubled tone mark inside the decomposed form fail too. It is deterministic, raises no error and is in 10.6.0 (released 2026-10-29 from the RC1 revision). Re-running the rules while a decomposed Sara Am is left fixes it; all Thai and analysis-module tests pass with that change.

• Sara Am ำ (U+0E33) is one vowel character. Typists often enter it as Nikhahit ํ (U+0E4D) followed by Sara Aa า (U+0E32); with a tone mark (Mai Ek ่ U+0E48, Mai Tho ้ U+0E49) the tone is often typed between the two pieces. This is the decomposed Sara Am. Issue #16716, which proposed the class, calls it "often typed".
• Lakkhangyao ๅ (U+0E45) looks like Sara Aa; the class converts it to Sara Aa unless it follows Ru or Lu.
• The example word is น้ำ, "water": No Nu น U+0E19, Mai Tho U+0E49, Sara Am U+0E33.

1. Step 2 of ThaiNormalizer.normalize is one forward pass (ThaiNormalizer.java:68-109). At a Nikhahit it tests whether the next character is Sara Aa, or the next two are a tone mark and Sara Aa (:79-94), on the characters as typed.
2. A Lakkhangyao after the Nikhahit becomes Sara Aa only when the loop reaches it (:103-108), after the test has failed: ท ํ ๅ leaves step 2 as ท ํ า.
3. With น ํ ่ ้ า the test sees Mai Ek, Mai Tho and fails. Step 4 (:135-147) then keeps the last of the two tone marks ("If two different consecutive tone marks appear, keep the last one", :142), or drops a repeated one (:137-141), and leaves น ํ ้ า.
4. normalize returns after step 4 (:149): the decomposed Sara Am that steps 2-4 produced stays. Calling normalize on its own output recomposes it.
5. With Mai Ek typed first (น ่ ํ ้ า) the test at the Nikhahit sees Mai Tho + Sara Aa and recomposes; step 4 then drops Mai Ek, giving น้ำ. So the result depends on the order the marks were stored in.
6. In the default ThaiAnalyzer, ThaiCharFilter runs first (ThaiAnalyzer.java:119-128). It converts the Lakkhangyao (ThaiCharFilter.java:149-154) and drops a repeated identical tone mark (:97-103), so those two shapes reach the token filter as a plain decomposed Sara Am, which it recomposes. The char filter has no keep-last-tone rule and its own recomposition needs exactly one tone mark (:126-136), so Nikhahit + two different tone marks + Sara Aa reaches ThaiNormalizer unchanged. With a Lakkhangyao in place of the Sara Aa, the char filter converts it and leaves both tone marks, so the same shape arrives.

An index-and-query probe with the default analyzer indexed only the two-tone form: a query for the clean word found 0 documents, a query typed with the same typo found 1.

• Who is exposed. Thai-language indexes on 10.6.0 (released 2026-10-29), main or branch_10_6 (no earlier release has the class):
  • the default ThaiAnalyzer: only Nikhahit + two different tone marks + Sara Aa (or a Lakkhangyao in its place, which the char filter turns into Sara Aa), a wrong tone mark typed and then the right one without deleting the first, in a word typed with the decomposed Sara Am;
  • ThaiNormalizationFilter without ThaiCharFilter: the factory's javadoc example (ThaiTokenizerFactory + ThaiNormalizationFilterFactory, no char filter), a custom chain with thaiNormalization after, for example, a whitespace tokenizer, or a search server that exposes the filter on its own. Here Nikhahit + Lakkhangyao and a doubled tone mark inside the decomposed form fail too.
• What happens. The typed word is indexed as a different term from the clean word, so a query for the clean word misses the document (1 of 2 documents found in the repro). Analyzer.normalize, which Lucene's classic query parser (QueryParserBase) uses for prefix, wildcard and fuzzy terms, returns the same unnormalized form.
• Scale. Deterministic, any index size; no error, no exception, the index stays valid.
• How likely. Rare: it needs two typing anomalies inside one character cluster, and each anomaly on its own is normalized correctly. The hunt's bounded check covered 824,691 inputs over a 21-symbol alphabet: every input of 1 to 5 symbols that starts with one of four initial characters (ก, น, ฤ or the leading vowel เ), plus every input of up to 3 symbols (all inputs of up to 5 symbols would be 4,288,305). 1,134 of them have a unique normal form under the model's encoding of the javadoc rules, pass its simplified well-formedness check, and keep a decomposed Sara Am; 12 of them reach ThaiAnalyzer.normalize. Over the same inputs, every ThaiAnalyzer.normalize output that keeps a decomposed Sara Am (16) comes from Nikhahit followed by two different tone marks.
• Timing. Released in 10.6.0 (2026-10-29): indexes built with 10.6.0 keep the wrong terms for the affected words until reindexed, even after a fix.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<น[้ำ]> but was:<น[ํ้า]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.th.TestThaiSaraAmTwoTones.testDecomposedSaraAmWithTwoTones(TestThaiSaraAmTwoTones.java:27)

• The output is the documented input form. "Recomposes decomposed Sara Am (Nikhahit U+0E4D + optional tone mark + Sara Aa U+0E32) into canonical Sara Am (U+0E33)" (ThaiNormalizer.java:29-30). The result นํ้า is Nikhahit + one tone mark + Sara Aa.
• The upstream tests. testSaraAmDecomposition asserts นํ้า -> น้ำ (TestThaiNormalizer.java:40-41); testConsecutiveDifferentTones asserts "When multiple different tone marks are typed, keep the last one" (:66-69). MIGRATE.md's 10.6 Thai section names "decomposed Sara Am (น + ํ + ้ + า)" as text 10.6 analyzes differently (lucene/MIGRATE.md:768).
• Keystroke order decides the result. Mai Ek typed before the Nikhahit gives น้ำ, and so does the precomposed two-tone form; the code even has a dedicated rule for the tone-first order (:96-101).
• The javadoc list is not an execution order. Dangling-mark removal is listed last (:38) but runs before step 2 (:63-66); issue #16716 lists the same rules in another order, deduplication first.
• No record of a limitation. PR #16717 has no reviews and one thank-you comment; #16716 has no comments; no upstream test combines two anomalies in one cluster.

The checkers built the strongest case for "intended": the javadoc is a list of best-effort repairs and promises no idempotence; its pattern allows one optional tone mark, so Nikhahit + two tone marks + Sara Aa is not literally the documented pattern; keep-last-tone is stated only in a code comment (ThaiNormalizer.java:142) and a test; and two typos in one cluster could be out of scope for "common orthographic variations" (:24).

Why it fails: the class itself emits the documented pattern, the one its own test treats as input to repair; the same marks in another stored order give a different term; and nothing in the javadoc, tests or history limits the rules to one anomaly per cluster. That makes a bug reading persuasive, not certain: a maintainer could still call combined anomalies an enhancement. The independent reproducer, the devil's advocate, the works-as-designed review and the adjudication rejected the case; the refute review said "not real (low)" only on procedure (it ran no command; it rated the intended case moderate and asked for a re-run), so it was discounted. The case survives as a priority argument (Low).

Option A was applied in a scratch worktree of main a9779ef3c193 and run through gradle with assertions on and off; option B is the finder's sketch, run through the repro harness and once through gradle; option C was not run.

Steps 2-4 move unchanged into a private normalizeMarks(s, len); normalize repeats it while hasDecomposedSaraAm(s, len) finds Nikhahit + Sara Aa or Nikhahit + tone mark + Sara Aa.

What it changes, by construction. The first pass is today's code, so a token's output differs from today only when today's output for that token still contains a decomposed Sara Am (the loop test also catches tone + Nikhahit + Sara Aa, which contains Nikhahit + Sara Aa). Each extra pass recomposes at least one, which deletes a character, so the loop ends after at most len extra passes. That bounds the passes, not the time: each pass includes step 3, whose scans repeat until nothing moves, so on long tokens its cost can grow with the square of the token length, as it does today. A token without a leftover decomposed Sara Am costs one extra linear scan; nothing is allocated.

Within an affected token, every part is re-run. The extra pass runs all of steps 2-4 on the whole token, so it can also change a part unrelated to the Sara Am: ThaiNormalizer.normalize on ฤๅ่ followed by ทํๅ (0E24 0E45 0E48 0E17 0E4D 0E45) gives 0E24 0E48 0E45 0E17 0E4D 0E32 today and 0E24 0E48 0E32 0E17 0E33 with fix A: the Lakkhangyao of the Ru pair becomes Sara Aa too. The bounded check (5 symbols) has no room for both patterns, so it cannot show this.

New output. Every failing row of the table gives น้ำ or ทำ, through ThaiNormalizer and ThaiAnalyzer.normalize. The malformed ฤๅ่ on its own (Ru + Lakkhangyao + Mai Ek, not claimed) keeps today's ฤ่ๅ.

    // Steps 2-4 can leave a decomposed Sara Am behind: step 2 tests for it on the characters as
    // typed, but converts a following Lakkhangyao only after that test, and step 4 removes a
    // repeated or extra tone mark between Nikhahit and Sara Aa. Run them again while one is left.
    // Each extra pass recomposes at least one, which shortens the buffer, so the loop ends.
    do {
      len = normalizeMarks(s, len);
    } while (hasDecomposedSaraAm(s, len));
    return len;
  }

  /** Steps 2-4: compositions, canonical reordering and deduplication. */
  private static int normalizeMarks(char[] s, int len) {
    // Step 2: ... (unchanged through the final "return len;")

  /** Nikhahit + Sara Aa, or Nikhahit + tone mark + Sara Aa. */
  private static boolean hasDecomposedSaraAm(char[] s, int len) {
    for (int i = 0; i + 1 < len; i++) {
      if (s[i] == '\u0E4D'
          && (s[i + 1] == '\u0E32'
              || (i + 2 < len && isToneMark(s[i + 1]) && s[i + 2] == '\u0E32'))) {
        return true;
      }
    }
    return false;
  }

Trade-offs

• Targets this bug only: other cases where a later step creates an earlier step's pattern keep today's output when they are alone in a token, such as the malformed Ru + Lakkhangyao + tone class, where step 3 moves the tone mark between Ru and Lakkhangyao; in a token that also holds a leftover decomposed Sara Am they change too (example above).
• Does not fix the sibling ThaiCharFilter bug (LFV-876), which has the same design at another fix site.
• Indexed terms change only for tokens whose output today keeps a decomposed Sara Am; with 10.6.0 released (2026-10-29), those would need reindexing (a MIGRATE note). No CHANGES entry is drafted; no benchmark was run.

What the checks validated

• Gradle in a scratch worktree of a9779ef3c193: the Thai tests (56 upstream plus the 4 new ones) pass 60 of 60 with assertions on and off (stock: the 4 new ones fail); all of :lucene:analysis:common:test with assertions on, 1,913 tests, 18 skipped, 0 failures; check -x test passes (google-java-format, ecjLint, forbiddenApis); tidy changes nothing.
• Shadowed into the repro harness: the new test 4 of 4, the finder's JUnit 6 of 6 and the reproducer's 9 of 9 pass.
• The hunt's bounded check (824,691 inputs): ThaiNormalizer typo-input mismatches 1,756 to 622, with the decomposed-Sara-Am classes (864 + 270) at 0 and only the unclaimed Ru + Lakkhangyao + tone class left; ThaiAnalyzer.normalize 734 to 722, the 12 two-tone inputs at 0; no ThaiAnalyzer.normalize output with a decomposed Sara Am is left (stock: 16).
• All of the above ran on main only: branch_10_6 (4ae946e5a9b) has a byte-identical ThaiNormalizer.java, but fix A was not built or tested there.

The same method split, with a do { copy; steps 2-4 } while (changed) loop that compares the buffer with a copy (of the hunt).

Trade-offs

• More general: it removes every typo-input mismatch of the bounded check (1,756 to 0), including the Ru + Lakkhangyao + tone class, where it turns ฤๅ่ into ฤ่า (Ru + tone + Sara Aa), breaking up the Ru + Lakkhangyao pair that the Lakkhangyao rule exists to keep; whether that is wanted is a separate question.
• Allocates a copy of the token on every call. It terminated on every input of the bounded check, but no simple argument bounds the passes, and a fixed two passes would not be enough (in the bounded check, 4 inputs, all Ru + Lakkhangyao + Nikhahit, need 3 changing calls of the stock normalizer).
• Not formatted or run through check (one comment line is 111 characters).

What the checks validated

• Through the repro harness: the finder's JUnit with TestThaiNormalizer, TestThaiAnalyzer and TestThaiNormalizationFilterFactory, 30 tests, 0 failures; the reproducer's JUnit 9 of 9.
• Gradle in the scratch worktree: the Thai tests with assertions on, 60 of 60.
• The bounded check: typo-input mismatches 1,756 to 0.

Convert a Lakkhangyao and collapse repeated or extra tone marks before the Sara Am test in step 2.

Trade-offs

No extra pass, but it changes the rule order for every input, must repeat the Ru/Lu exception, and step 3 can still move marks afterwards. Not executed.

• Method (semi-formal): bounded-exhaustive translation validation of the stock main jar, in the hunt's unit on the 10.6 Thai normalizers. The javadoc's rules were encoded as a string-rewriting system (the model's reading of the javadoc, not a Lucene contract); for 824,691 inputs over a 21-symbol alphabet (every input of 1 to 5 symbols that starts with one of four initial characters, ก, น, ฤ or the leading vowel เ, plus every input of up to 3 symbols; all inputs of up to 5 symbols would be 4,288,305) the reference computed all normal forms by exhaustive rewriting, and the real ThaiNormalizer was compared with them.
• Signal. 1,756 inputs with a unique normal form under the model, passing its simplified well-formedness check, got an output that one of the class's own rules still rewrites: 864 + 270 in this decomposed-Sara-Am class, 622 in a malformed Ru + Lakkhangyao + tone class that is not claimed. Iterating the stock normalizer to a fixed point removed every mismatch. Through ThaiAnalyzer.normalize, 12 inputs of the class remained, all with two different tone marks.
• First lead. A planning probe before the unit, a bounded idempotence check over 333,453 inputs, had found 4,632 inputs whose output changes on a second call, for example ก ํ ๅ -> กํา -> กำ.
• Then reading the code gave the mechanism, and the finder wrote the public-API JUnit and the fixed-point sketch; independent checkers followed (see "Independent checks").