BengaliNormalizer's visarga rule copies the next letter before folding it, so নিঃশব্দ and নিঃসব্দ get different terms

BengaliNormalizer (package-private; used by BengaliNormalizationFilter, by BengaliAnalyzer at index and query time and by the bengaliNormalization factory) is one left-to-right pass over the term. It folds SHA শ and SSA ষ to SA স everywhere (and NNA, RRA, RHA, khanda-ta, II, UU and NGA one to one), and it turns a medial visarga ঃ into a copy of the next letter: s[i] = s[i + 1] at BengaliNormalizer.java:121. That copy is made before the loop has folded the next letter, and the loop never returns to it, so নিঃশব্দ ("silent") becomes নিশসব্দ (SHA SA), নিঃষব্দ becomes নিষসব্দ, and only নিঃসব্দ gives নিসসব্দ. The class's other gemination, Ba-phalaa, copies an already folded letter (বিশ্ব -> বিসস, pinned by an upstream test), and the cited paper gives SHA, SSA and SA one code. Through BengaliAnalyzer the three spellings get three terms and do not match each other. Every release from 7.1.0 to 10.5.2 and main has it. The fix, s[i] = foldSimple(s[i + 1]), changes indexed terms of the affected words, so those fields need a reindex.

• Visarga ঃ (U+0983): inside a word it doubles the sound of the next consonant, so নিঃশব্দ is said like nishshabdo. The normalizer models this as gemination: the visarga becomes a copy of the next letter. A final visarga becomes হ in a short word and is deleted in a longer one.
• The folds: one-to-one rewrites the class applies at every position: SHA শ U+09B6 and SSA ষ U+09B7 to SA স U+09B8 (// All sh), NNA ণ to NA ন, RRA ড় U+09DC and RHA ঢ় U+09DD to RA র U+09B0, khanda-ta ৎ to TA ত, NGA ঙ to anusvara ং, and the vowel signs II ী to I ি and UU ূ to U ু.
• BengaliNormalizer: a left-to-right rewrite of each token, one switch (s[i]) case per character. It is package-private and reached through BengaliNormalizationFilter, which BengaliAnalyzer uses in its main chain (indexing and analyzed queries) and in normalize() (multi-term queries), and through the bengaliNormalization factory. Tokens marked as keywords skip it.

1. The visarga case (BengaliNormalizer.java:112-123) runs s[i] = s[i + 1]; (:121) for a medial visarga.
2. At that moment position i + 1 has not been visited, so the copy is the raw letter. The loop then folds the original at i + 1 (// All sh, :125-129) and never returns to i. For নিঃশব্দ: ন ি ঃ শ ব ্ দ -> ন ি শ শ ব ্ দ -> ন ি শ স ব ্ দ.
3. Ba-phalaa, the class's other gemination, copies backwards, s[i - 1] = s[i - 2] (:106), from a position already folded: বিশ্ব -> বিসস. The visarga version of the same doubling, বিঃশ, gives বিশস.
4. The result is not even a fixed point of the class's own rules: normalizing নিশসব্দ again gives নিসসব্দ.

The hunt's bounded check found 23,947 violations of "no character that an unconditional rule rewrites or deletes may survive" (chandrabindu and the visarga included) over all 551,881 strings of up to 4 characters on a 27-character alphabet; every one contains a visarga. A folded character survives in 20,097 of them, and the recommended fix removes all of those; the other 3,850 keep only a chandrabindu or a visarga, cases that no fix option covers.

• Who is exposed. BengaliAnalyzer (the default constructor included), BengaliNormalizationFilter, and the bengaliNormalization factory, for example in a CustomAnalyzer. No setting turns the rule off. Only tokens marked as keywords skip it, together with the whole normalizer (BengaliNormalizationFilter.java:48); BengaliAnalyzer marks the words of its stem-exclusion set, if one is given (BengaliAnalyzer.java:123-125).
• Ordinary input. Tatsama words such as নিঃশব্দ, নিঃশেষ, নিঃশ্বাস and দুঃশাসন have a medial visarga before শ. Every such word gets the unfolded term, at index and query time alike.
• What the damage looks like. Silent: no error, and identical spellings still match. What fails is the match the folds exist for: a document and a query that spell the letter after the visarga differently (শ, ষ or স; ণ or ন) do not meet, although the analyzer conflates them anywhere else in a word. With one document per spelling of নিঃশব্দ, each spelling's query finds 1 of 3 documents; with the fix, 3 of 3. An independent end-to-end test: the স-spelled query finds 0 of 1 শ-spelled documents, while the no-visarga control finds 1 of 1.
• Only lost matches. Over 2,000,718 generated short strings (a synthetic set, not a corpus), 9,779,308 pairs of strings get the same output with the fix but different outputs on stock, and no pair the other way; the fix changes the output of 167,664 strings, each containing a visarga, and never changes an output's length.
• How likely. Rare: a missed match needs the two sides to spell that one letter differently, and no report of it was found in 9 years.
• Scale. One word reproduces it; assertions on and off behave the same.
• Reindex. A fix changes the term of every affected word, so fields analyzed before the fix keep the old terms until they are reindexed.
• What is not shown. No corpus count or native-speaker check measures how often the variant spellings (নিঃষব্দ, নিঃসব্দ for নিঃশব্দ) occur; the cross-spelling impact is inferred from the class's own folds and the cited paper's codes.
• Downstream remark. An application that filters on analyzed text (a denylist query, a MUST_NOT clause) misses a variant spelling the same way any query does. Authorization is the application's job under Lucene's SECURITY.md, so this is not a Lucene security issue.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<নি[স]সব্দ> but was:<নি[শ]সব্দ>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.bn.TestBengaliVisargaRule.testVisargaCopyIsFolded(TestBengaliVisargaRule.java:27)

• The class's own folds. // All sh, // check na and the others are unconditional. After a visarga the output still holds a letter the class rewrites everywhere else, and a second pass changes it (নিশসব্দ -> নিসসব্দ). That alone does not prove a bug: another rule interaction also changes on a second pass (কঁ্ি -> ক্ি -> খ, where deleting the chandrabindu exposes the khio pattern). It supports the diagnosis together with the next two points.
• The other gemination. Ba-phalaa (BengaliNormalizer.java:106, s[i - 1] = s[i - 2]) copies a letter the loop has already folded, and TestBengaliNormalizer.testBaPhalaa (same commit) asserts check("বিশ্ব", "বিসস"). The visarga copy gives বিঃশ -> বিশস for the same doubling. The difference comes only from loop order: one rule looks back at a processed position, the other ahead at an unprocessed one.
• The cited paper (UzZaman and Khan, named in the class javadoc). Visarga "Case 1: In the medial position ... acts as a diacritic, geminating the sound of the consonant it follows. So, it gets the code of next character"; its example of a medial visarga before স encodes to <dussmy>. SHA, SSA and SA share the code <s>, and the Ba-phalaa example বিশ্ব encodes to <biss>.
• Tests. testVisarga checks দুঃখ -> দুখখ, উঃ -> উহ, পুনঃ -> পুন. খ is not folded, so a raw copy and a folded copy give the same output: no test pins the current behaviour, and the fix breaks none. testRandom can generate a folded letter after a visarga but checks only the length.
• History. The copy and the folds arrived together in the contributor's first commit 1bca06b8a938 (2017-08-24, LUCENE-7940, gh#8989, lucene-solr PR #238). Two later commits of the same PR only added the Ja-phala and Ba-phalaa bounds checks from the review (3197d5af71d, 40dddf9324e); commits after the 2017-09-05 landing only reformat or change visibility. None of them touches the visarga lines, and the review never discussed the visarga.
• No public javadoc for the mapping. The normalizer is package-private; the public filter says only that it normalizes "the orthography" and that "the normalization may cause unrelated terms to conflate". No javadoc promises a folded copy or a stable output, so there is no public contract; the evidence of intent is the class's rules, its tests and the cited paper.

The strongest "intended" case (the devil's-advocate verifier and the intended-lens review by another model family, rated moderate): the class is a one-pass transcription of the paper, and "gets the code of next character" can be read as "copy the next character", which s[i] = s[i + 1] does literally; the paper describes phonetic codes for a spelling checker, not search normalization; nothing promises that the output is closed under the folds or stable under a second pass, and another rule interaction is not stable either; the normalizer is package-private; no expected-output test covers a folded letter after a visarga; the output has been stable for 9 years and the original review measured relevance with this code; a maintainer defends the Indic folds as "working as documented" in gh#14659; another contributor's evaluation found the Bengali folds too aggressive for search; and nothing measures how often the variant spellings occur. So the missing conflation may be called harmless or not worth a reindex.

Why it fails: in the paper "code" is the phonetic code, which SHA, SSA and SA share; the class's own Ba-phalaa gemination copies the folded letter and an upstream test from the same commit pins it; the output is not a fixed point of the class's rules (supporting evidence, not proof on its own); and nothing documents the ordering. Long life is not evidence that anyone chose it. Verdict: a defect (adjudication: confirmed, high). What the counter-case gets right: the impact is small, a fix changes indexed terms, and documenting the raw copy is a legitimate maintainer choice; a maintainer may also ask for a Bengali-language check first.

Options A and B give identical output on every probe word; B avoids a second copy of the fold table. Add the new visarga cases to TestBengaliNormalizer with whichever is chosen. Not covered by any option (not claimed): what the copy should become when the next character is itself deleted (chandrabindu) or part of a context rule (KA + HASANTA, Ja/Ba-phala). Both options fold only the nine one-to-one characters; before anything else (ZWNJ or ZWJ, punctuation, a Latin letter, a supplementary character) the visarga is still a raw copy of one UTF-16 code unit. With a single-token KeywordTokenizer, কঃ😀 gives [0995 D83D D83D DE00], an unpaired high surrogate that IndexWriter stores as U+FFFD, on stock and with both fixes; BengaliAnalyzer splits the emoji off, so it is not affected (executed).

The seven one-to-one cases (II, UU, NGA, SHA/SSA, NNA, RRA/RHA, khanda-ta) move with their comments into private static char foldSimple(char c); the switch's default calls it, so every non-visarga output stays the same, and the medial visarga copies foldSimple(s[i + 1]). +40/-37 in BengaliNormalizer.java.

} else {
  // the copy is normalized like the next character itself
  s[i] = foldSimple(s[i + 1]);
}
break;

default:
  s[i] = foldSimple(s[i]);
  break;

Trade-offs

• Changes the indexed term of every word with a medial visarga before a folded character: this would likely warrant a CHANGES entry and a MIGRATE note (reindex fields analyzed with BengaliAnalyzer, BengaliNormalizationFilter or the bengaliNormalization factory); MIGRATE.md has such a note for the Romanian change in 10.0 (a precedent, not a rule). Main is the natural target; a 10.x backport is debatable.
• The filter conflates more, but only within its own folds.
• A larger diff than the minimal patch, but the table of one-to-one folds exists once, which removes the risk of two copies drifting apart (a special case added to the switch later could still differ from the copy). Cost: one static call for each character that reaches the switch's default and for each medial-visarga copy (chandrabindu, KA, YA, BA and the visarga keep their own cases); no allocation (by reading).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test fails 3 of 3 on stock (assertions on and off) and passes with the fix; org.apache.lucene.analysis.bn.*: 5 suites, 25 tests (22 upstream + 3 new), 0 failures, with assertions on (seed 5EED0676) and off (seed 5EED0677); :lucene:analysis:common:checkGoogleJavaFormat passes; the whole :lucene:analysis:common:test: 340 suites, 1,912 tests, 18 skipped, 0 failures, with assertions on (seed 5EED0678) and with the seed's choice (seed 5EED0676).
• Standalone runner with the fix shadowed: the new test 3 of 3 pass and the 4 upstream suites 22 of 22, each with -ea and -da; the probe gives 3 of 3 hits for every spelling and leaves the upstream examples and the digit case unchanged.

Change line 121 to s[i] = foldSimple(s[i + 1]); and add a helper with the same nine mappings as the switch's one-to-one cases (+25/-1). The finder's original fix.

-            s[i] = s[i + 1];
+            s[i] = foldSimple(s[i + 1]);

Trade-offs

The smallest behavioural diff, but the mapping table then exists twice, so a later edit to one fold must update both. Same reindex note.

What the checks validated

Gradle: the 5 Bengali suites, 25 tests, 0 failures with assertions on and off; checkGoogleJavaFormat passes. Standalone: the new test 3 of 3 pass (-ea, -da); earlier the finder's test 3 of 3, the upstream suites 22 of 22 and the independent reproducer's test 9 of 9.

Pin নিঃশব্দ -> নিশসব্দ in a test and comment that the copy is raw. Not run.

Trade-offs

It keeps every indexed term (no reindex, exact output stability), no public javadoc promises a folded copy, and nothing measures how often the variant spellings occur, so a maintainer may reasonably prefer it or ask for a Bengali-language check first. Its cost: the two geminations stay inconsistent, the output stays not a fixed point of the class's own rules, and the copy keeps differing from the cited paper's coding.

A one-line idea that fixes the sibilant cases.

Trade-offs

It loops forever on a doubled visarga: কঃঃক gives no result within 2 s through the filter and BengaliAnalyzer, while the stock code and both fixes return কঃকক (executed).

Method: bounded-exhaustive translation validation on the stock main jars (the study's "semi-formal" tier). A hunt unit, "script normalizers against their documented rules", 2026-10-27.

• A closure oracle (no character that an unconditional rule rewrites or deletes may survive in the output: the folded characters, chandrabindu and the visarga) and a mapping-invariance oracle (SHA/SSA/SA and the other folded pairs must give the same output) ran over all 551,881 strings of length 0 to 4 on a 27-character alphabet (consonants, vowel signs, hasanta, chandrabindu, anusvara, visarga, khanda-ta; 23,947 closure violations) and all 5,380,840 strings of length 0 to 7 on a 9-character alphabet (1,010,898).
• A fact check later split the 23,947: a folded character survives in 20,097, and only a chandrabindu or a visarga in the other 3,850. With the recommended fix (option B), 4,372 remain, all chandrabindu/visarga-only (cases no fix option covers), and no folded character survives.
• Every closure violation contains a visarga, and a residual run that excludes visarga and KA + HASANTA inputs has 0 closure and 0 mapping-invariance violations, which isolates the cause to the visarga copy (KA + HASANTA is the separate khio bug).
• Then an AI finder wrote the public-API JUnit, and independent AI tools reproduced, reviewed and judged it (see the checks).