BengaliNormalizer's khio rule tests the wrong character, so the conjunct ক্ষ is never folded

BengaliNormalizer (package-private; used by BengaliNormalizationFilter, by BengaliAnalyzer at index and query time and by the bengaliNormalization factory) says it implements the algorithm of the paper "A Double Metaphone encoding for Bangla and its application in spelling checker" (UzZaman and Khan). Its rule // Khio (Ka + Hoshonto + Murdorno Sh) rewrites the conjunct ক্ষ (KA + HASANTA + SSA) as খ at the start of a word and as কখ elsewhere, mirroring the paper's two khio cases. But the condition tests the third character against '\u09BF' (BENGALI VOWEL SIGN I) instead of '\u09B7' (BENGALI LETTER SSA). So the rule never fires on correctly encoded ক্ষ: ক্ষমা normalizes to ক্সমা (the separate "All sh" rule turns ষ into স), and BengaliAnalyzer indexes it as ক্সম, while খমা is indexed as খম. The rule fires instead on the malformed KA + HASANTA + VOWEL SIGN I (ক্িমা becomes খমা). Every release from 7.1.0 to 10.5.2 and main has it (63 release tags read, 16 releases run). The fix is one character, and it changes the indexed term of every word with ক্ষ, so affected fields need a reindex; deleting the rule is the alternative that keeps today's terms.

• ক্ষ (khio): the conjunct KA U+0995 + HASANTA U+09CD + SSA U+09B7. A hasanta (virama) joins two consonants; a vowel sign such as ি (U+09BF) follows a consonant, never a hasanta, in well-formed text.
• BengaliNormalizer: a left-to-right rewrite of each token, one rule per character (switch (s[i])). Its javadoc says it implements the cited paper. It is package-private and is reached through BengaliNormalizationFilter, which BengaliAnalyzer uses at index time and in normalize(), and through the bengaliNormalization factory.

1. The rule is case '\u0995': under // Khio (Ka + Hoshonto + Murdorno Sh) (BengaliNormalizer.java:56-68). "Murdorno Sh" is ষ, U+09B7.
2. Its condition is s[i + 1] == '\u09CD' && s[i + 2] == '\u09BF' (:58). U+09BF is BENGALI VOWEL SIGN I, the code point that the DirghoI rule writes two cases earlier (:48).
3. Its branches are right: word-initial gives খ, otherwise কখ (:59-66), the paper's two cases.
4. On correctly encoded ক্ষ the branch never matches, so the later "All sh" rule turns ষ into স (:125-129): ক্ষ ends as ক্স. On the malformed ক্ি the branch does fire and folds it to খ (or কখ), dropping the vowel sign.

The hunt's bounded check (all 551,881 strings of length 0 to 4 over 27 rule characters) found the khio rule violated on all 28 inputs that start with ক্ষ.

• Who is exposed. BengaliAnalyzer (the default constructor included), BengaliNormalizationFilter, and the bengaliNormalization factory, for example in a CustomAnalyzer. No setting switches this rule alone; only keyword-marked tokens skip it, together with the whole normalizer (BengaliNormalizationFilter.java:48-49), for example words in BengaliAnalyzer's stem-exclusion set (BengaliAnalyzer.java:123-125) or after a SetKeywordMarkerFilter, as the filter javadoc suggests.
• Ordinary input. ক্ষ occurs in everyday words: ক্ষমা, পরীক্ষা, শিক্ষা, রক্ষা, বৃক্ষ, দক্ষিণ and many more. Every such word gets a term the rule would not produce, at index and query time alike.
• What the damage looks like. Silent: no error, and identical spellings still match (with the same analyzer version at index and query time). What fails is the match the rule exists for. In a six-document index (ক্ষমা, খমা, ক্সমা, ক্িমা, বক্ষ, বক্স) the term query for ক্ষমা matches ক্ষমা and ক্সমা, the one for খমা matches খমা and ক্িমা, and বক্ষ matches বক্ষ and বক্স. With the fix, ক্ষমা and খমা both match ক্ষমা and খমা, and বক্ষ matches only itself.
• Both directions. Over 1,508,597 generated short strings (a synthetic set, not a corpus), the bug loses 36,270 conflation pairs that the rule would create and keeps 40,223 that the fix removes; the fix changes the output of 1,804 strings, each containing ক্ষ (902) or the malformed ক্ি (902).
• Scale. One word reproduces it; assertions on and off behave the same.
• Reindex. A fix that makes the rule fire changes the term of every ক্ষ word, so fields analyzed before the fix keep the old terms until they are reindexed. Until then even identical text misses: the document ক্ষমা indexed by the old code has the term ক্সম, and the query ক্ষমা analyzed with the fix looks for খম.
• What is not shown. The evidence is a mismatch between the code and its own comment and cited paper. No corpus evaluation or native-speaker check was done, so nothing here shows that the fold improves Bengali search, or that খমা is a spelling in actual use for ক্ষমা rather than the one the paper's phonetic code equates with it.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<[খ]মা> but was:<[ক্স]মা>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.bn.TestBengaliKhioRule.testWordInitialKhio(TestBengaliKhioRule.java:25)

• The contract. The comment at BengaliNormalizer.java:56 names the rule, and the class javadoc (:24-26) says the class implements the cited paper. The paper lists the khio conjunct as \u0995 \u09CD \u09B7 with "Case 1: In the initial position of a word ... it is given the same code as" KHA and "Case 2: In the middle or at the end of words, it is similar to" KA KHA. The code's branches mirror these two cases (the paper assigns phonetic codes for a spelling checker; the code applies them as character rewrites); only the third code point differs. The finder, the independent reproducer and the devil's advocate each extracted the paper text.
• No contract names the coded sequence. Neither the comment nor the paper, nor any test, CHANGES entry or discussion, names KA + HASANTA + VOWEL SIGN I; the only mention found is the outside wiki page (see the prior reports), which transcribes the code. The sequence does not occur in well-formed Bengali, and the reviewed history and discussions identify no damage pattern that rewriting it as খ (dropping the vowel sign) would repair.
• Tests. TestBengaliNormalizer tests every other commented rule and has no khio case (nor one for the uncommented ৎ-to-ত rule, BengaliNormalizer.java:142-144), so no test pins the current khio output. The 4 Bengali suites pass with either fix.
• History. The rule came with the contributor's first commit 1bca06b8a938 (2017-08-24, LUCENE-7940, now gh#8989; apache/lucene-solr PR #238); later commits on these lines (8ef6a0da568, f99deb58b74) only reformat. The review never discussed khio.
• Precedent. Lucene changes analysis output when it is wrong and asks users to reindex (MIGRATE.md, "Romanian analysis", 10.0), and fixed missing Telugu normalization as a bug (GITHUB#13059).
• No public javadoc for the mapping. The normalizer is package-private; the public filter says only that it normalizes "the orthography" and that "the normalization may cause unrelated terms to conflate". The contract is the comment plus the cited paper.

The strongest "intended" case (the devil's-advocate verifier and the intended-lens review by another model family, rated moderate): no public javadoc names the mapping; no test covers khio; the output has been stable for 9 years, and the original review measured relevance with this code; a maintainer, rmuir, defends the Indic folds in gh#14659 as intended for damaged text from legacy fonts and PDF extraction ("The filter here is working as documented ... Just don't use the filter if you dont want the transformations that it does."); and another contributor's wiki page reports that a native-speaker reviewer rejected the filter's phonetic rules for search, so a dead khio rule may be "accidentally better".

Why it fails: a comment that names the rule plus a javadoc that cites the paper defining it are a stated contract, and both say KA + HASANTA + SSA. Nothing explains why a branch so labelled matches a vowel sign. Long life is not evidence that anyone chose it: no comment, test, CHANGES entry or discussion reviewed mentions khio. And "working as documented" cuts the other way, because the code does not do what is documented. Verdict: a defect (adjudication: confirmed, high). What the counter-case gets right: the search value of the fold is debatable, the paper describes phonetic codes for a spelling checker rather than search normalization, and a fix changes indexed terms.

Two options were built and checked; maintainers may prefer either. Add a khio case to TestBengaliNormalizer with whichever is chosen.

One character at BengaliNormalizer.java:58. Then ক্ষমা → খমা, পরীক্ষা → পরিকখা, বক্ষ → বকখ (no longer the same as বক্স), and the malformed ক্ি is left alone.

// Khio (Ka + Hoshonto + Murdorno Sh)
case '\u0995':
  if (i + 2 < len && s[i + 1] == '\u09CD' && s[i + 2] == '\u09B7') {

Trade-offs

• Changes the indexed term of every word with ক্ষ: needs a CHANGES entry and a MIGRATE note (reindex fields analyzed with BengaliAnalyzer, BengaliNormalizationFilter or the bengaliNormalization factory), as the Romanian change did in 10.0. Main is the natural target; a 10.x backport is debatable.
• The filter folds more: word-initial ক্ষ words merge with খ words, as the paper intends and as the filter javadoc's conflation warning covers.
• Same complexity and branch body, but the work moves from ক্ি inputs to ক্ষ inputs: on one repeated 96,000-character token (far beyond StandardTokenizer's default 255-character limit) stock takes about 0 ms on ক্ষ and 62 ms on ক্ি, and the fix the reverse; negligible for word-length tokens.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test fails 3 of 3 on stock (assertions on and off) and passes with the fix; org.apache.lucene.analysis.bn.*: 5 suites, 25 tests (22 upstream + 3 new), 0 failures, seed 5EED0667 with assertions on and 5EED0668 with them off; :lucene:analysis:common:checkGoogleJavaFormat passes; the whole :lucene:analysis:common:test: 340 suites, 1,912 tests, 18 skipped, 0 failures.
• Standalone runner with the fix shadowed: the new test 3 of 3 pass with -ea and -da; earlier, the finder's test 3 of 3 and the 4 upstream suites 22 of 22; the release probe passes on 7.1.0 with the fix.

Remove BengaliNormalizer.java:56-69. Well-formed text keeps today's terms (ক্ষমা stays ক্সমা); only the malformed ক্ি stops being folded.

Trade-offs

• No reindex for well-formed text (only an indexed malformed ক্ি gets a new term), and the code no longer claims a rule that it does not implement.
• Gives up the paper's khio fold and keeps the বক্ষ/বক্স collision. Fits the view that the Indic folds are already aggressive (gh#14659).

What the checks validated

The 4 upstream Bengali suites, 22 of 22, pass with it (assertions on and off); the khio test fails under it, as designed.

Describe what the code does. Not run.

Trade-offs

Keeps every indexed term, so it is the most compatible option; but it documents a rule for a sequence that well-formed text does not contain, keeps the vowel-sign deletion and leaves the paper's khio case unimplemented.

Method: bounded-exhaustive translation validation on the stock main jars (the study's "semi-formal" tier). A hunt unit, "script normalizers against their documented rules", 2026-10-27.

• For every string of length 0 to 4 over a 27-character alphabet built from the rule characters (consonants, vowel signs, hasanta and the other signs; 551,881 inputs, the empty one included), the output of BengaliNormalizationFilter was checked against a directed oracle per documented rule. The khio oracle (a word-initial ক্ষ must become খ, from the rule comment) failed on all 28 inputs that start with the conjunct.
• A second relation (ী is documented to map to ি, so swapping one for the other in the input must not change the output) failed on ক্ী (→ ক্ি) versus ক্ি (→ খ), which pointed at U+09BF in the khio condition; reading the code confirmed it.
• Then an AI finder wrote the public-API JUnit, and independent AI tools reproduced, reviewed and judged it (see the checks).