TeluguNormalizer joins ఒ + ౌ into AU but never folds that AU to OO, so ఔషధం and ఒౌషధం get different terms

TeluguNormalizer (package-private since 10.0; used by TeluguNormalizationFilter, the teluguNormalization factory and TeluguAnalyzer) is one left-to-right pass over the term. It folds the long vowel AU ఔ (U+0C14) to OO ఓ (U+0C13) (TeluguNormalizer.java:59-62), a rule an upstream test pins for the filter alone, and it joins the two-character spelling ఒ + ౌ (U+0C12 U+0C4C) into ఔ (:102-106, comment (ఒ + ౌ) -> au (ఔ)). The joined ఔ is written at the current position, which the loop never visits again, so it is never folded: ఔషధం ("medicine") becomes ఓషధం while ఒౌషధం becomes ఔషధం. The class's other pair, ఒ + ౕ, writes ఓ directly. Chains that run the Telugu filter without IndicNormalizationFilter in front, such as the factory javadoc's example, give the two spellings different terms at index and query time; the stock TeluguAnalyzer composes the pair earlier and is unaffected from 9.10.0 on (before that, only its normalize() path was affected). Every release from 8.10.0 to 10.5.2 and main has it (10 releases and main were run; the releases in between have the same rule by the code history). The one-line fix (write ఓ at :104) changes the terms of words spelled with the pair, so those fields need a reindex.

• ఔ LETTER AU U+0C14 and ఓ LETTER OO U+0C13: independent vowels. The normalizer folds the long vowels to short ones, among them ఔ to ఓ (// long -> short vowels, TeluguNormalizer.java:59-62).
• ఒ + ౌ (LETTER O U+0C12 + VOWEL SIGN AU U+0C4C): a two-character spelling of AU, which the class joins into ఔ (:102-106). It is not a Unicode canonical equivalent (NFC and NFD keep both spellings); IndicNormalizer's decomposition table, documented as following Unicode 5.2 and a page of graphical decompositions (IndicNormalizer.java:36-37, :71-74), maps it to AU for Telugu (:143-144).
• ఒ + ౕ (LETTER O + LENGTH MARK U+0C55): a two-character spelling of OO, which the class joins into ఓ (:98-101); an upstream test calls these "two ways to write oo letter" (it runs the full TeluguAnalyzer, where IndicNormalizationFilter joins the pair first; the filter alone also gives ఓ, as the probe shows).
• TeluguNormalizer: a left-to-right rewrite of each token, one switch (s[i]) case per character. It is package-private (public in 8.10.0-9.12.x) and reached through TeluguNormalizationFilter, the teluguNormalization factory (for analysis and normalize()) and TeluguAnalyzer. Keyword-marked tokens skip it.

1. case 'ఒ': (TeluguNormalizer.java:96-107): if the next character is ౌ, it writes s[i] = 'ఔ' (:104) and deletes position i + 1.
2. The loop then continues with i + 1. Position i is never visited again, so case 'ఔ' (:60-62) never sees the ఔ written there. For ఒౌషధం: ఒ ౌ ష ధ ం → ఔ ష ధ ం, done.
3. A precomposed ఔ always hits case 'ఔ': ఔషధం → ఓషధం. So in a chain with only this filter, the joined ఔ renames the term but never merges it with the precomposed spelling.
4. The sibling compositions write final forms (ఒ + ౕ → ఓ at :100, ె + ౖ → ై at :92), and the same switch already steps back (i--) after deleting a zero-width joiner (:52-57); the AU branch does neither.
5. TeluguAnalyzer runs IndicNormalizationFilter first (TeluguAnalyzer.java:120-121, :130-131), which joins ఒ + ౌ into ఔ before the Telugu filter folds it, so the stock analyzer gives one term.

Over all 837,931 strings of up to 4 characters on the hunt's 30-character Telugu alphabet, ఔ survives in exactly the 2,760 inputs that contain the adjacent pair ఒౌ; over all 25,137,931 strings of up to 5 characters, in exactly the 110,670 such inputs, and the fixed output is always the stock output with ఔ replaced by ఓ.

• Who is exposed. A CustomAnalyzer, a hand-built Analyzer or a schema with the Telugu filter and no IndicNormalizationFilter before it, for example the factory javadoc's field type (StandardTokenizerFactory + TeluguNormalizationFilterFactory); on 8.10.0-9.12.x also direct callers of the then-public TeluguNormalizer.normalize. No javadoc names Indic normalization as a prerequisite.
• The stock analyzer. TeluguAnalyzer is unaffected from 9.10.0 on. In 8.10.0-9.9.x its normalize() dropped IndicNormalizationFilter (fixed by GITHUB#13059), so its prefix, wildcard and fuzzy terms hit the bug there (executed on 8.10.0, 8.10.1, 8.11.0, 8.11.4, 9.0.0 and 9.9.2).
• Why the chain is plausible. In the open issue gh#14659 about a Telugu rule of IndicNormalizer, a maintainer (rmuir) replied "if you have perfect unicode text ... don't use this filter" (the filter under discussion is IndicNormalizationFilter), and a participant proposed making TeluguAnalyzer leave it out by default. Either path leaves the Telugu filter alone with the pair.
• What the damage looks like. Silent: no error, and identical spellings still match. With one document per spelling (run on main), each spelling's query finds 1 of 2 documents and a prefix query typed with the pair finds only its own spelling; with the fix, 2 of 2. The bug only keeps apart spellings that the fix joins (367,395 pairs over 837,931 short strings) and never joins two that the fix keeps apart.
• How likely. Rare: it needs the custom chain and text that spells AU with two characters. How often that spelling occurs in real text was not measured.
• Scale. One word reproduces it; assertions on and off behave the same; the recommended fix does the same write and shift per pair as the stock code, and a timing probe found no material difference between them.
• Reindex. A fix changes the term of every word spelled with ఒ + ౌ in affected chains, so fields analyzed before the fix keep the old terms until reindexed.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<[ఓ]షధం> but was:<[ఔ]షధం>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.te.TestTeluguComposedAu.testComposedAuFoldsToOo(TestTeluguComposedAu.java:27)

• The class's own rules. The composition comment names the result AU (TeluguNormalizer.java:103), under the heading // composed oo or au -> oo or au (:96); IndicNormalizer has the same mapping for Telugu (IndicNormalizer.java:143-144). The AU -> OO fold is unconditional (:59-62), and TestTeluguNormalizer.testVowels pins it with the filter alone (check("ఔఐఆఈఊ", "ఓఏఅఇఉ"), :35-40).
• The sibling composition. ఒ + ౕ is written as ఓ (TeluguNormalizer.java:98-101), and the filter alone gives ఓ for it (probe). TestTeluguAnalyzer.testBasics names the pair: // two ways to write oo letter. checkOneTerm(a, "ఒౕనమాల", "ఓనమాల") (:33-34, and through normalize() at :59); it runs the full TeluguAnalyzer, where IndicNormalizationFilter already joins ఒ + ౕ (IndicNormalizer.java:145-146), so it does not by itself pin the Telugu filter's branch. In the filter alone AU also has two ways, and only one reaches ఓ.
• No prerequisite documented. The filter "applies TeluguNormalizer to normalize the orthography" (TeluguNormalizationFilter.java:27), and the factory javadoc shows it after the standard tokenizer alone (TeluguNormalizationFilterFactory.java:26-32). If Indic normalization always ran first, the class's own ఒ + ౌ branch would be dead code.
• Tests. No upstream test has ఒ followed by ౌ; the 16 upstream Telugu tests pass with either fix.
• History. Both rules arrived in 544dbbea468 (LUCENE-10083, PR #275, 2021-09-02). Inside the PR, the composition to ఔ was in the first commit (c143ef618e, 2021-08-13); the AU -> OO fold came with 6c0c09933f "added long to short vowel conversion" (2021-09-01), which also changed the result of the ె + ౖ composition but left the AU one writing ఔ. The only review was an approval. That this was an oversight is an inference from the commit order; no record says so.
• What is not claimed. Lucene documents no idempotence rule for its language normalizers, and the class promises to remove only "some differences in spelling variations" (TeluguNormalizer.java:24). The case rests on the class's own rules.

The strongest "intended" case (the devil's-advocate verifier and the intended-lens review by another model family, rated weak to moderate): Lucene's language normalizers are one-pass, best-effort rewrites with no idempotence promise, and an upstream Sorani test pins an output that a second pass would change (TestSoraniNormalizationFilter.testH, ھ → ه); the comment at :103 literally says the pair becomes "au (ఔ)", which the code does; ఒ + ౌ is not a Unicode canonical equivalent; and the stock TeluguAnalyzer runs IndicNormalizationFilter first and gives one term.

Why it fails: in the Sorani case the class's javadoc names two different letters: ھ (06BE) is the joining form of 'h' and becomes ه (0647), while a word-final ه is a form of the vowel 'e' and becomes ە (06D5) (SoraniNormalizer.java:29-30), so one pass keeps apart what that javadoc describes as different letters. Here the class itself says both spellings are the letter AU; the same method folds AU to OO without condition and an upstream test pins it; the sibling compositions write their final forms; the fold was added later in the same PR without updating the composition; and the filter and factory are public and documented without an Indic prerequisite. Verdict: a defect (adjudication: confirmed, high). What the counter-case gets right: the reach is narrow and the impact low.

Options A and B give the same output on all 25,137,931 strings of the bounded check (A = B, and A = the stock output with ఔ replaced by ఓ). Add the ఒ + ౌ case to TestTeluguNormalizer with whichever is chosen. Not covered by any option (side observations, untriaged): ే + ౖ (U+0C47 U+0C56) is not a fixed point either, a visarga or a zero-width (non-)joiner between ఒ and ౌ leaves the pair unjoined in the filter with or without either fix (for the zero-width joiners even in the stock TeluguAnalyzer.normalize()), and the class javadoc says @since 9.0.0 although it shipped in 8.10.0.

Change the write at TeluguNormalizer.java:104 from '\u0C14' to '\u0C13' and update the comment (+2/-2), the same shape as the ఒ + ౕ branch at :100. The finder's fix.

} else if (i + 1 < len && s[i + 1] == '\u0C4C') {
-  // (ఒ + ౌ) -> au (ఔ)
-  s[i] = '\u0C14';
+  // (ఒ + ౌ) -> au (ఔ), which is normalized to oo (ఓ) like a precomposed au
+  s[i] = '\u0C13';
   len = delete(s, i + 1, len);

Trade-offs

• The smallest change; the stock TeluguAnalyzer output is unchanged, because it never reaches this branch with the pair.
• It repeats the knowledge "AU folds to OO" in a second place: if the AU -> OO fold were ever dropped, this line would have to change too.
• Changes the indexed term of every word spelled with ఒ + ౌ in affected chains: a CHANGES entry with a reindex note. Main is the natural target; a 10.x backport is debatable.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test fails 2 of 2 on stock (assertions on and off) and passes with the fix; org.apache.lucene.analysis.te.*: 5 suites, 18 tests (16 upstream + 2 new), 0 failures, with assertions on (seed 5EED0669) and off (seed 5EED066A); :lucene:analysis:common:checkGoogleJavaFormat passes; the whole :lucene:analysis:common:test: 340 suites, 1,911 tests, 18 skipped, 0 failures, with assertions on.
• Standalone runner with the fix shadowed: the new test 2 of 2 and the 4 upstream suites 16 of 16, each with -ea and -da; earlier the finder's test 4 of 4, the independent reproducer's 7 of 7 and the devil's advocate's 8 of 8.

After the composition, step back one position so the loop's existing case 'ఔ' folds the result (+2/-1), the pattern the zero-width-joiner deletion already uses (:52-57).

   s[i] = '\u0C14';
   len = delete(s, i + 1, len);
+  i--;

Trade-offs

Keeps "this pair is AU" as the single rule, so a later change to the AU fold applies to both spellings; less obvious to read. It terminates: the revisited ఔ is rewritten once by case 'ఔ' (the bounded run ends on all 25,137,931 inputs). Same reindex note.

What the checks validated

Gradle: the 5 Telugu suites, 18 tests, 0 failures with assertions on and off; checkGoogleJavaFormat passes. Standalone: the new test 2 of 2 and the upstream suites 16 of 16 (-ea, -da); identical output to option A on all 25,137,931 bounded inputs. The whole module was run with option A only.

Say so in the filter and factory javadoc and the factory example. Not run.

Trade-offs

Changes no indexed terms. But the class's own ఒ + ౌ branch stays inconsistent with its fold (and becomes dead code in documented use), and it runs against a maintainer's advice in gh#14659 to drop IndicNormalizationFilter for clean Unicode text.

Would also make the spellings agree (both ఔ). Not run.

Trade-offs

Breaks TestTeluguNormalizer.testVowels (:39) and changes the term of every ఔ word in every Telugu chain, the stock TeluguAnalyzer included: a much larger reindex for a question this bug does not raise.

Method: bounded-exhaustive translation validation on the stock main jars (the study's "semi-formal" tier). A hunt unit, "script normalizers against their documented rules", 2026-10-27.

• A closure oracle (no character or pair that an unconditional rule rewrites, deletes or joins may survive: the long vowels and long vowel signs, candrabindu, visarga, the zero-width joiners, and the two-character spellings the class joins) ran over all 837,931 strings of length 0 to 4 on a 30-character Telugu alphabet: 6,286 violations. A recount with the same oracle splits them: 2,962 have ఒ and ౌ in the input (2,760 with the adjacent pair, 202 with a visarga or zero-width (non-)joiner between the two); the other 3,324 are side cases (3,138 leave ె + ౖ unjoined, for example U+0C47 U+0C56; 186 leave ఒ + ౕ after a deleted visarga or joiner). Over the same strings, ఔ survives in exactly the 2,760 inputs with the adjacent pair.
• A residual run that excludes inputs with ఒ, ె or ే (ే folds to ె) found 0, which isolates every violation to the compositions that start at ఒ or ె.
• Then an AI finder wrote the public-API JUnit, and independent AI tools reproduced, reviewed and judged it (see the checks).