ThaiRepeatFilter gives the repeated word the offsets of a character a CharFilter deleted, not of its ๆ

ThaiRepeatFilter (new in 10.6, #16720) expands the Thai repetition mark Maiyamok ๆ (U+0E46): บ่อยๆ becomes the token บ่อย plus a second บ่อย that stands for the ๆ. It computes their offsets by counting back from the end of the incoming span: the word gets [start, end - 1) and the repeat the last character (ThaiRepeatFilter.java:114-123 on main 645d4fe14a20). That is right only when the span maps 1:1 onto the term. The stock ThaiAnalyzer puts ThaiCharFilter in front (ThaiAnalyzer.java:119-123), which deletes zero-width characters (ThaiCharFilter.java:89-94), and CharFilter offset correction ends a token after the characters deleted right behind it (BaseCharFilter.java:40-53). So for บ่อยๆ + U+200B + ครับ the span is [0,6), the word gets [0,5) (which contains the ๆ) and the repeat [5,6), only the deleted zero-width space. Terms, positions, matching and scoring are correct; offsets in the index and highlights are wrong. Lucene's other offset-splitting filters keep the whole span when its length differs from the term's; this one has no such check.

• Maiyamok ๆ (U+0E46): the Thai repetition mark. บ่อยๆ ("often") is read บ่อย บ่อย. ThaiRepeatFilter (10.6, #16720) emits the word, then one copy per ๆ, each copy at its own position with the offsets of its ๆ.
• Offsets: OffsetAttribute start and end, the range of the original text a token came from. They are stored with postings or term vectors and used by highlighters.
• CharFilter: edits the text before the tokenizer sees it and records where, so the tokenizer can map offsets back (correctOffset). ThaiCharFilter, which the stock ThaiAnalyzer always uses, deletes U+200B, U+200C, U+200D and U+FEFF; HTMLStripCharFilter deletes markup.

Input offsets: บ0 ่1 อ2 ย3 ๆ4 U+200B 5 ค6 ร7 ั8 บ9.

1. ThaiAnalyzer builds ThaiTokenizer -> LowerCaseFilter -> DecimalDigitFilter -> ThaiNormalizationFilter -> ThaiRepeatFilter -> StopFilter and wraps every reader in ThaiCharFilter (ThaiAnalyzer.java:101-109, :119-123).
2. ThaiCharFilter drops U+200B at output offset 5 and records addOffCorrectMap(5, 1) (ThaiCharFilter.java:89-94). The tokenizer sees บ่อยๆครับ.
3. BaseCharFilter.correct(off) adds the diff recorded at the largest offset <= off (BaseCharFilter.java:40-53), so the token บ่อยๆ, which ends at output offset 5, ends at source offset correct(5) = 6: it arrives as [0,6). That widening is the long-known CharFilter end-offset behaviour (#6796, #7653) and harmless by itself: the span still covers the whole word.
4. ThaiRepeatFilter (case 2, ThaiRepeatFilter.java:104-129) counts 1 trailing ๆ, cuts the term to บ่อย, sets baseEnd = Math.max(origStart, origEnd - maiyamokCount) = 5 (:114), gives the word [0,5) (:115) and the repeat [baseEnd, origEnd) = [5,6) (:118-123). Nothing compares origEnd - origStart with the term length (:75-77).
5. For a token of two or more ๆ (case 1, :79-102) the copies get one character each, counted from the start, the last ending at origEnd (:87-96): the same one-character-per-ๆ assumption. A single standalone ๆ is not split: the if (count > 1) block (:87-100) is skipped and the token keeps its incoming span.

With HTMLStripCharFilter + ThaiTokenizer + ThaiRepeatFilter, บ่อยๆ</b>ครับ arrives as [0,9), so the word gets [0,8) = บ่อยๆ</b and the repeat [8,9) = >.

• Default chain, no opt-in. ThaiAnalyzer always uses ThaiCharFilter and ThaiRepeatFilter. Any of U+200B, U+200C, U+200D or U+FEFF right after an attached ๆ triggers it. ThaiCharFilter's javadoc names zero-width characters among the typographic anomalies it exists to remove (ThaiCharFilter.java:27-30).
• JDK-dependent reach. The JDK word BreakIterator must keep the ๆ in the word's token. On JDK 25.0.3 it does for จริงๆ, บ่อยๆ, ดีๆ, อื่นๆ and not for เร็วๆ, มากๆ, whose ๆ becomes its own token.
• Custom chains: chains with ThaiRepeatFilter (SPI name thaiRepeat) where a CharFilter deletes characters right after a word ending in ๆ and the tokenizer keeps that ๆ in the word's token, so the deleted characters land in the tail of the span. Executed with HTMLStripCharFilter (a tag right after the word is enough) and ThaiCharFilter, also with a whitespace tokenizer and with two or more ๆ.
• What breaks: offsets in postings, term-vector offsets and highlights. Terms, positions, matching, scoring and filtering are unchanged; nothing throws; every span stays inside the token's original span.
• How it shows: a phrase or other position-sensitive highlight of the repeated word leaves the ๆ out; a term query highlights both tokens, so it still covers the ๆ. In raw-HTML highlighting the highlighter's tags land inside the following tag (</i + >). HTMLStripCharFilter already mis-nests highlight markup before a closing tag for plain tokens (<i><b>abc</i></b>); ThaiRepeatFilter adds the cut in the tag text. With escape=true only the highlight position is wrong.
• Timing: in 10.6.0, released on 2026-10-29 from the RC1 revision.
• Downstream: Elasticsearch merged a thai_repeat port of the filter into its Lucene-snapshot branch (elastic/elasticsearch#160853); opensearch-project/OpenSearch#23151 (open) plans one.

The input's U+200B is written as the escape \u200B. The assumeTrue mirrors Lucene's Thai tests (skipped on a JDK without the Thai dictionary BreakIterator). The trigger also needs the JDK to keep บ่อยๆ in one token, which JDK 25.0.3 does; on a JDK that splits off the ๆ the filter takes its other branch and the test passes.

On 10.5.2 the test fails earlier, at the terms check ([บ่อยๆ U+200B, ครับ]), because 10.5.2's ThaiAnalyzer has neither ThaiCharFilter nor ThaiRepeatFilter: the bug cannot occur there.

The finder's wider test (5 methods: this case, the HTMLStripCharFilter chain, both highlight symptoms and a control) fails 4 of 5 on 645d4fe14a20; it is kept out of the minimal repro.

cp $REPRO/TestThaiRepeatFilterZeroWidthSpace.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/th/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.th.TestThaiRepeatFilterZeroWidthSpace -Ptests.seed=5EED0231
# also with -Ptests.asserts=false

TestThaiRepeatFilterZeroWidthSpace > testRepeatedTokenCoversMaiyamokAfterZeroWidthSpace FAILED
    java.lang.AssertionError: the repeated token must cover the Maiyamok at offset 4; offsets: [[0,5), [5,6), [6,10)]
        at __randomizedtesting.SeedInfo.seed([5EED0231:856B084623294010]:0)
        ...
        at org.apache.lucene.analysis.th.TestThaiRepeatFilterZeroWidthSpace.testRepeatedTokenCoversMaiyamokAfterZeroWidthSpace(TestThaiRepeatFilterZeroWidthSpace.java:55)
:lucene:analysis:common:test (FAILURE): 1 test, 1  failure

# the same with -Ptests.asserts=false, on the standalone runner (-ea, -da) and on the 10.6.0 RC1 jars (-ea, -da)
# without the zero-width space: บ่อย[0,4) บ่อย[4,5) ครับ[5,9)
# with the recommended fix: :lucene:analysis:common:test (SUCCESS): 1 test

• The record promises exact offsets. lucene/CHANGES.txt:494-495: "Add ThaiRepeatFilter and ThaiRepeatFilterFactory to expand Thai repetition mark (Maiyamok) with exact OffsetAttribute and highlighter compatibility." The squash commit b7c2c202ba69 carries the in-PR message: "Ensure both the primary term and repeated Maiyamok tokens receive precise, non-overlapping start/end offsets. For attached Maiyamok (e.g. 'เร็วๆ'), set primary token endOffset to boundary before 'ๆ' and set repeated token offsets to match the corresponding 'ๆ' character(s)." TestThaiRepeatFilter#testAttachedMaiyamok asserts [0,4) and [4,5) for เร็วๆ (TestThaiRepeatFilter.java:38-44). The class javadoc does not restrict the input (ThaiRepeatFilter.java:27-34).
• The repro asks for less than "exact": only that the repeat's span contains its ๆ. The current output breaks even that.
• The default analyzer composes it. ThaiCharFilter (#16717) and ThaiRepeatFilter (#16720) came from the same author and were merged 45 minutes apart; ThaiAnalyzer uses both (ThaiAnalyzer.java:101-123).
• Lucene guards this case elsewhere. WordDelimiterGraphFilter: "if length by start + end offsets doesn't match the term's text then set offsets for all our word parts/concats to the incoming offsets" (:257-263); WordDelimiterFilter.hasIllegalOffsets (:432-434); CJKBigramFilter: "crazy offsets (modified by synonym or charfilter): just preserve" (:284-285).
• Why the tests missed it. The explicit tests use MockTokenizer or ThaiTokenizer input with no CharFilter (TestThaiRepeatFilter.java:30-91); the randomized test checks consistency and offset bounds, not which characters a token covers. #16720 had one approving comment and no review threads.
• Not to report: the repeats' position increment of 1 was a deliberate choice in #16720.

The strongest case for "works as designed" (the refute review and the cross-model review rated it moderate; the intended-behaviour check built it and rejected it; nobody accepted it): the analysis package says "Token filters should not modify offsets. If you feel that your filter would need to modify offsets, then it should probably be implemented as a tokenizer" (package-info.java:416-417); #9496 (LUCENE-8450, open) says filters "cannot perform simple arithmetic to calculate the original ('correct') offset of a character in the interior of the token" once CharFilters change token lengths, and #8817 (LUCENE-7766, open) reports the same for WordDelimiter(Graph)Filter; OffsetAttribute allows a span that differs from the term length (OffsetAttribute.java:27-29); and CharFilter end-offset attribution is an old open limitation (#6796, #7653).

Why it fails: the filter's record promises exact offsets and highlighter compatibility; a span that is too wide would be the accepted outcome, but here the repeat's span misses its own ๆ and the word's span takes it, and the defensive handling Lucene uses elsewhere (keep the whole span, a convention that #8817 disputes) is missing; the trigger is the shipped default analyzer; and in LUCENE-4984 (#6048) the earlier Thai splitting filter was replaced by ThaiTokenizer because, as a token filter, it created positions and updated offsets.

One option was built and validated (A). B to D are alternatives from the reviews and the brief, not built.

The finder's guard, re-made as a git diff on 645d4fe14a20 (1 file, +10 -6). Compute exactOffsets once per token; when it is false, the word and every repeat keep [origStart, origEnd), in both branches (case 1 gets the same change for firstEnd, s and e). The idiom of WordDelimiterGraphFilter and CJKBigramFilter: a conservative fallback, not an exact mapping. The length test is a heuristic: equal lengths do not prove a 1:1 mapping (a replacement that keeps the character count, or a deletion and an insertion that cancel out, were not tested), and a length difference does not prove that a CharFilter acted.

+      // Offsets can only be split per character if they map 1:1 onto the term (a CharFilter may
+      // have removed or contracted characters inside the token's span, e.g. a zero-width space
+      // or markup after the Maiyamok); otherwise every emitted token keeps the original span.
+      boolean exactOffsets = origEnd - origStart == len;
...
-        int baseEnd = Math.max(origStart, origEnd - maiyamokCount);
+        int baseEnd = exactOffsets ? Math.max(origStart, origEnd - maiyamokCount) : origEnd;
...
-          int s = Math.min(origEnd, baseEnd + i);
-          int e = (i == maiyamokCount - 1) ? origEnd : Math.min(origEnd, s + 1);
+          int s = exactOffsets ? Math.min(origEnd, baseEnd + i) : origStart;
+          int e = (i == maiyamokCount - 1 || !exactOffsets) ? origEnd : Math.min(origEnd, s + 1);

Trade-offs

• Safe, not exact: after a CharFilter change the word's span also covers the ๆ. Tests that require the word not to cover it still fail: the independent reproduction's test fails 5 of 9 with the guard (its 4 ownership tests and one control; its 2 highlight tests pass), the refute reviewer's exact-offset tests 2 of 4.
• It widens a case main gets right by coincidence: a deletion before the ๆ (มาก + U+200B + ๆ).
• The same coarseness in WordDelimiterFilter is the subject of open #8817 (LUCENE-7766).
• Today's output is kept only when the incoming span length equals the term length (the exactOffsets branch is today's code).
• Any other mismatch also gets the whole span, whatever caused it, including a token filter that changed the term's length. ThaiNormalizationFilter, which runs right before ThaiRepeatFilter in ThaiAnalyzer, shortens terms (ThaiNormalizationFilter.java:44): with WhitespaceTokenizer + ThaiNormalizationFilter + ThaiRepeatFilter and no CharFilter, ดีีๆ นะ gives the exact ดี[0,3), ดี[3,4) on main and ดี[0,4) twice with the guard.

What the checks validated

• The minimal test passes with assertions on and off (gradle, and the standalone runner).
• :lucene:analysis:common:test with the fix, assertions on and off: 340 suites, 1910 tests, 18 skipped, 0 failed (all 10 th suites included); :lucene:analysis:common:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• The finder's 5-method test passes 5 of 5 with it, so both highlight symptoms go away.
• Offsets with the fix: บ่อย[0,6) twice for the zero-width input, บ่อย[0,9) twice for the HTML input; without a CharFilter change unchanged ([0,4), [4,5)).
• Every run of the guard was on main; on branch_10_6 and branch_10x the diff applies cleanly (same ThaiRepeatFilter.java blob) but was not run.

A tokenizer can map each character back through the CharFilter chain with correctOffset(int); a token filter cannot. Expanding ๆ in ThaiTokenizer (or handing it the split points) would let the word end at the ๆ's own corrected offset. This follows the analysis package's guidance ("it should probably be implemented as a tokenizer", package-info.java:416-417), rmuir's position on #9496 ("The thing dividing text up into tokens, tokenizer is the lucene class for that.") and LUCENE-4984 (#6048), where the earlier Thai splitting filter was replaced by ThaiTokenizer.

Trade-offs

A design change to a new public filter (thaiRepeat works after any tokenizer today); the split would have to account for ThaiNormalizationFilter, which runs before ThaiRepeatFilter. Not built; the offsets it would give are reasoning from BaseCharFilter.java:40-53, not executed.

Change where ThaiCharFilter records the correction for a removed zero-width character.

Trade-offs

Fixes only the default-analyzer trigger; HTMLStripCharFilter and other CharFilters still produce it. It moves the next token's start back over the removed character, and since correct() serves start and end offsets alike, a deletion at a token boundary must belong to one side. Not built.

Reword the CHANGES entry and the class javadoc so that exact offsets hold only when no CharFilter changed the token's characters. Could accompany A.

Trade-offs

No behaviour change; on its own it leaves the misattribution in place.

Method: bounded-exhaustive translation validation of the ThaiAnalyzer chain on the stock main jar, by a hunt unit (analysis, new filters). The harness runs the real classes, so there is no transcription gap; no model checker, SMT solver or proof produced the signal.

• Scope: every text of up to 3 characters over a 16-character Thai slice (consonants, vowels, tone marks, the Maiyamok, a Thai digit, space, U+200B and U+200D), inside 12 prefix/suffix word contexts: 52,428 texts, 460,810 checks at length 3, in 1.7 s.
• Oracle O4, written from the CHANGES entry: each repeated token covers exactly its own Maiyamok and the base token covers none. 306 texts violate it at length 3; the first violation came at length 2 (ดีๆ + U+200B: expected [0,2), [2,4), got [0,3), [3,4)).
• Vacuity check: the oracle goes quiet with the guard fix.
• The HTMLStripCharFilter and highlighter symptoms were then found by targeted runs, and the finder wrote a 5-method JUnit; this doc's repro is its first method as a self-contained class.