TeluguAnalyzer never removes 13 of its 43 single-word default stop words, such as కూడా ("also"), because its stop list is not normalized

TeluguAnalyzer normalizes every token before its StopFilter: IndicNormalizationFilter, then TeluguNormalizationFilter, then StopFilter, then TeluguStemFilter. The Telugu normalizer shortens the four long vowel signs ీ ూ ే ో to ి ు ె ొ, but the default stop list te/stopwords.txt holds only the original spellings, and StopFilter is an exact set lookup. So the filter sees కుడా while the list holds కూడా: 13 of the 43 single-word entries (30%), among them కూడా ("also"), అయితే, మళ్ళీ, మాత్రమే and ఇప్పటికే, are never removed and are indexed and searched as ordinary terms, with no error. HindiAnalyzer has the same chain shape, and its list ships the normalized forms, so it removes all of its single-word entries. The list has been like this since the analyzer was added (LUCENE-10083, 8.10.0); every release from 8.10.0 to 10.6.0 (10.6.0 by reading) and main is affected. The fix is a 16-line data change in the Hindi style; it changes indexed terms, so affected fields need reindexing.

• Stop words: common words an analyzer drops before indexing and querying. TeluguAnalyzer() "Builds an analyzer with the default stop words" from te/stopwords.txt (TeluguAnalyzer.java:98-101, :42-43).
• Vowel sign (matra): the mark a Telugu consonant carries for its vowel. ూ (U+0C42) is the long u sign, ు (U+0C41) the short one; కూడా is క + ూ + డ + ా.
• The chain (TeluguAnalyzer.java:114-125): StandardTokenizer -> DecimalDigitFilter -> SetKeywordMarkerFilter (only with a stem-exclusion set) -> IndicNormalizationFilter (:120) -> TeluguNormalizationFilter (:121) -> StopFilter(result, stopwords) (:122) -> TeluguStemFilter (:123).

1. TeluguNormalizer folds the long vowel signs to the short ones on every token that is not marked as a keyword: ీ -> ి, ూ -> ు, ే -> ె, ో -> ొ (TeluguNormalizer.java:76-88, "// long -> short vowels matras"). The default constructor adds no keyword marker.
2. StopFilter.accept() is an exact CharArraySet lookup of the current term (StopFilter.java:96-97).
3. te/stopwords.txt has only the header # Telugu stop word list and the original spellings. Every entry with one of those four vowel signs is therefore unreachable in the stock analyzer (no stem-exclusion set): the filter sees కుడా, the set holds కూడా. The word goes on to the stemmer and is indexed as కుడ.

The file has 50 entry lines, 45 distinct entries (5 duplicate lines), 2 of them two words (lines 31 and 45, which can never match one token). Of the 43 single-word entries, 13 (30%) are never removed: మెచ్చుకో, చేయగలిగింది, ఇప్పటికే, మళ్ళీ, అయితే, కూడా, అందుబాటులో, వేరుగా, మాత్రమే, అందరూ, దూరంగా, చుట్టూ, వ్యతిరేకంగా (lines 2, 11, 12, 14, 18, 20, 21, 22, 37, 39, 43, 50, 51). Each differs from its normalized form only in those vowel signs. The entry గురించి has none of them and is removed.

• Words named in a stem-exclusion set: the keyword marker makes TeluguNormalizationFilter skip them, so the raw entry matches (IndicNormalizationFilter ignores the marker, but it changes none of the 13 entries).
• normalize() (TeluguAnalyzer.java:127-133, used for prefix, wildcard and fuzzy terms): it has no StopFilter.
• Custom stop sets that hold the normalized forms.

A sweep ran every default entry of the 39 stock analysis-common analyzers that have a default stop set through the whole analyzer. Read entry by entry, the leftovers outside Telugu and BengaliAnalyzer are comment lines, hyphenated, elided or multi-word entries, so only these two lose single-word entries to pre-stop normalization (13 each; the Bengali case is a separate, already publicly described bug).

• Who is hit. The default constructor of a stock language analyzer, so every Telugu field that uses it, at index and query time, on every release from 8.10.0. Also code that reuses TeluguAnalyzer.getDefaultStopSet() after the same two normalization filters. Only Telugu fields use this analyzer. No dedicated Elasticsearch or OpenSearch Telugu analyzer was found (the Elasticsearch request #84800 is open), but Solr can configure a Lucene analyzer class by name (<analyzer class="...">, built with its no-arg constructor), so Solr users can use it directly.
• What goes wrong (5-document probe index, stock analyzer vs the fixed list, assertions on and off):
  • the 13 words are indexed: extra postings, scoring noise, and no position gap where a stop word should leave one;
  • a query made only of them still matches: the term query కూడా matches 2 documents, where the fixed analyzer builds no query at all;
  • AND and phrase queries that contain one are stricter: MUST("రాముడు కూడా") matches 1 document instead of 3, the phrase "రాముడు కూడా వచ్చాడు" 1 instead of 2;
  • the stemmer folds the kept word onto an unrelated one: కూడా and కుడి ("right") both give కుడ, so the query కుడి also finds "రాముడు కూడా వచ్చాడు".
• What does not go wrong. Index and query share the chain, so plain OR queries still find their documents. No exception, no corruption, and no unusual cost observed: 20,000 seeded random Telugu texts gave 0 exceptions, and 200,000 copies of కూడా take the same 4,188 index bytes as 200,000 copies of an ordinary word (one codec, one single-document segment). Deterministic; no scale needed.
• How common. Every such index gets these terms whenever its text contains one of the 13 words; their corpus frequency was not measured. Impact class: minor relevance quality. Compared with the intended stop-word behaviour it does lose matches for AND and phrase queries that contain one of the words (above), and the stem collision adds false ones; plain OR queries are not affected.
• Reindex. Any fix changes indexed and query terms, so affected fields need reindexing.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: TokenStream has more tokens than expected (expected count=0); extra token=కుడ
       org.junit.Assert.fail(Assert.java:89)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:402)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:573)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:637)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:792)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:777)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:862)
       org.apache.lucene.analysis.te.TestTeluguDefaultStopword.testDefaultStopwordRemoved(TestTeluguDefaultStopword.java:27)

• Contract (implied, not literal). TeluguAnalyzer() "Builds an analyzer with the default stop words" (:98), the field is the "File containing default Telugu stopwords." (:42-43), and the createComponents javadoc lists "Telugu Stop words" in the chain. No sentence literally promises that every entry is removed, but an entry no input can reach through the default analyzer (no stem-exclusion set) is dead data in a shipped default resource.
• Lucene's other stop files and analyzers state the same rule for their own lists (a convention stated file by file, not a general API contract): a list must match the token form where its StopFilter runs: hi/stopwords.txt:4-7 ("by default this file also contains forms normalized by HindiNormalizer for spelling variation ... such that it can be used whether or not you enable that feature. When adding additional entries to this list, please add the normalized form as well."); ArabicAnalyzer.java:135 ("the order here is important: the stopword list is not normalized!", so its StopFilter runs before normalization) and ar/stopwords.txt:4; PersianAnalyzer.java:136-137 and fa/stopwords.txt:4-5 ("used after normalization"); ckb/stopwords.txt:2 ("normalized with our scheme"); el/stopwords.txt:2-3.
• A copied template. The Telugu createComponents javadoc is HindiAnalyzer's with Hindi replaced by Telugu and LowerCaseFilter left out, and the chain has the same shape (HindiAnalyzer.java:122-129; Hindi also lowercases first); the Hindi list rule was not copied.
• No recorded choice. te/stopwords.txt has exactly one commit (the one that added the analyzer). The list, the four folds and the filter order were all in PR #275's first commit; the PR has one review ("LGTM") and no line comments; #11121, CHANGES and MIGRATE say nothing about Telugu stop words. TestTeluguAnalyzer never checks that a stop word is removed.
• Precedent. LUCENE-1966 (#3041, 2009) caught the same mismatch for ArabicAnalyzer in review and settled it by running StopFilter first. Stop-list data fixes have gone under Bug Fixes (GITHUB#14075) and Improvements (GITHUB#12172, GITHUB#16718), so the CHANGES section is the committers' call.
• The createComponents javadoc lists the stop words after TeluguStemFilter while the code runs StopFilter before it, and SetKeywordMarkerFilter after both normalizers while the code adds it before them (HindiAnalyzer has the same two doc nits); the 13 entries are dead in either order.

The intended-behaviour devil's advocate and the cross-model reviewer's intended lens built the strongest case for "not a bug": no javadoc promises that every entry is removed; StopFilter is exact matching and never promises to normalize a supplied set; the list looks like rough data (the devil's advocate's non-native reading: a word-by-word translation of the a-words of an English stop list, which would explain the duplicate pairs, the two-word entries and content words such as "available"); the fold is lossy, Telugu normalization has been questioned upstream (#14659, about IndicNormalizer's separate వు -> మ rule, not these four folds), and upstream might prefer to change the fold; and with a stem-exclusion set the raw entries do match.

Why it fails: Lucene's other stop files and analyzers state the rule for their own lists, and TeluguAnalyzer follows neither form of it; a rough list explains how the error happened, but nobody chose to ship entries that can never fire in the default analyzer; "change the fold" is a choice of fix, not intent (the fold is documented: "long -> short vowels matras"); the keyword path needs a user-supplied exclusion set naming the stop word, which the default analyzer never has. Every checker's verdict: real; the adjudication's class: minor.

Two fixes were applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF); the stop-filter-first option and the workaround were not run.

Append a 3-line comment and the 13 forms that the analyzer's own pre-stop filters produce for the 13 entries. The raw spellings stay, so the list still works in a chain that filters stop words before normalizing. Add the repro (or its first two tests) to the Telugu tests as the regression test; it states a broader rule than any javadoc does, so a test focused on the 13 forms is a less brittle choice if a committer prefers.

--- a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/te/stopwords.txt
+++ b/lucene/analysis/common/src/resources/org/apache/lucene/analysis/te/stopwords.txt
@@ -49,3 +49,19 @@
 వద్ద
 అందుబాటులో
 దూరంగా
+# TeluguAnalyzer applies this list after IndicNormalizationFilter and TeluguNormalizationFilter,
+# which shorten long vowel signs. The forms below are the normalized forms of the entries above
+# that those filters change. When adding entries, please add the normalized form as well.
+చెయగలిగింది
+మళ్ళి
+వ్యతిరెకంగా
+అందరు
+మాత్రమె
+ఇప్పటికె
+కుడా
+అయితె
+వెరుగా
+మెచ్చుకొ
+చుట్టు
+అందుబాటులొ
+దురంగా

Trade-offs

• Reindex. Indexed and query terms change for text with the 13 words (old documents keep కుడ, new queries drop the word): needs a CHANGES entry and a MIGRATE note.
• Over-stopping: ask a Telugu speaker to review the 13 forms. The fold can merge distinct words, so a normalized form may also be an ordinary word: చుట్టు, the normal form of చుట్టూ ("around"), is also an ordinary verb form. The Hindi list accepts the same cost; a reviewer may drop individual forms.
• getDefaultStopSet() grows from 45 to 58 entries; custom chains that reuse it after the two normalizers are fixed too. No API change.
• The list stays consistent only by convention; the comment asks future editors to add normalized forms.

What the checks validated

• Stock main a9779ef3c193 (gradle, seed DEADBEEF): the repro fails 3 of 4 with assertions on and off; the four upstream Telugu test classes pass (16 tests).
• With the fix: the repro and the four Telugu classes pass (20 tests, 0 failures) with assertions on, and again with assertions off; the repro alone passes with assertions off (4 tests); all of :lucene:analysis:common:test with the repro added passes (1,913 tests, 0 failures, 18 skipped); ecjLintTest and checkGoogleJavaFormat pass with the final test file.
• Run order, for the record: the stock gradle runs and the module-wide run used the test before checkGoogleJavaFormat asked for a different line wrapping; the final file differs only in layout, and the fix run with it was repeated.
• Through the classpath runner: the repro passes 4 of 4 with -ea and -da with the fixed list first on the classpath; the default set then has 58 entries (45 before).
• fix.diff and fix-with-test.diff (the fix plus the repro as a new test file) pass git apply --check on a clean tree.

In TeluguAnalyzer.DefaultSetHolder, add normalize("", entry) of every entry (the same DecimalDigitFilter, IndicNormalizationFilter and TeluguNormalizationFilter the analyzer runs before its StopFilter) to the loaded set and keep it unmodifiable (+10/-1 lines in TeluguAnalyzer.java).

CharArraySet raw = WordlistLoader.getWordSet(/* te/stopwords.txt, as before */);
// The StopFilter runs after IndicNormalizationFilter and TeluguNormalizationFilter, so also
// add every entry in the form those filters produce (as normalize() does).
CharArraySet set = new CharArraySet(raw, false);
try (TeluguAnalyzer normalizer = new TeluguAnalyzer(CharArraySet.EMPTY_SET)) {
  for (Object entry : raw) {
    set.add(normalizer.normalize("", new String((char[]) entry)).utf8ToString());
  }
}
DEFAULT_STOP_SET = CharArraySet.unmodifiableSet(set);

Trade-offs

Future list edits stay consistent automatically and the data file is unchanged; but it is code in a static initializer that builds a temporary TeluguAnalyzer, Analyzer.normalize treats each entry as one token (no tokenizer), so it also adds the normalized form of a two-word entry (harmless for the stock chain; 59 entries), and the same reindex and over-stopping review apply.

What the checks validated

• With the change (gradle, seed DEADBEEF, assertions on): the repro and the four Telugu classes pass (20 tests, 0 failures); ecjLintMain and checkGoogleJavaFormat pass.
• Through the classpath runner (the changed class shadowed onto the main jars): the repro passes 4 of 4 with -ea and -da; the default set then has 59 entries.
• Not run: the whole module with this change.

Move StopFilter above IndicNormalizationFilter in createComponents.

Trade-offs

Not run. The 13 raw spellings would then match, but the analyzer would stop removing variant spellings that the normalizer folds onto an entry today (for example a word written with a long vowel sign where the list has the short one), and users who worked around the bug with a normalized custom stop set would break. Not recommended.

Pass a stop set that also holds the normalized forms: new TeluguAnalyzer(set) with the default set plus analyzer.normalize("", w) of each entry. The independent reproducer's control shows that a set holding కుడా removes కూడా.

Trade-offs

Changes indexed terms, so it also needs reindexing, and it does not help other users of the default list.

Method: bounded-exhaustive checking of the stock jar against the analyzer's own documented chain (semi-formal tier), in a hunt unit that checked script normalizers against their documented rules.

• For every entry of the default stop set of each analyzer that runs StopFilter after these normalizers, the check ran the entry through exactly the analyzer's pre-stop filters and asked whether the result is still in the set. Telugu: 14 of 45 entries flagged (the 13 single-word entries plus one two-word entry); Bengali 13; Hindi and Persian 0.
• It was confirmed end to end with Analyzer.tokenStream, then reduced to a 2-test public-API JUnit that fails on the stock main jars with assertions on and off.
• Checked by an independent reproducer, a reviewer from another model family (OpenAI GPT-5.6, refute and intended lenses, both executed), an intended-behaviour devil's advocate, an adjudication, a history, prior-report and security follow-up, and an audit. Found 2026-10-27 on main a9779ef3c193.