SerbianAnalyzer.normalize skips SerbianNormalizationFilter, so Cyrillic and diacritic prefix, wildcard and fuzzy queries miss

SerbianAnalyzer indexes StandardTokenizer -> LowerCaseFilter -> StopFilter -> [SetKeywordMarkerFilter] -> SnowballFilter(SerbianStemmer) -> SerbianNormalizationFilter (SerbianAnalyzer.java:118-127). The last filter maps Cyrillic to Latin and removes the accents on Latin letters (č and ć become c, š becomes s, ž becomes z, đ becomes dj). So Београд is stored as beograd and Čačak as cack. SerbianAnalyzer.normalize(), which the classic and flexible query parsers use for prefix terms, wildcard chunks, fuzzy terms, regexps and range bounds (the simple parser for its prefix and fuzzy terms), returns only new LowerCaseFilter(in) (:129-132). Analyzer#normalize is documented to give "the representation that it would have in the index", without tokenizing or stemming (Analyzer.java:203-207). The result: Беогр*, Čač* and Београд~1 find nothing while Beogr* and Cac* find the document, and normalize("Ђак") returns ђак while the index holds djak even with stemming switched off. Term and phrase queries are unaffected. Every release since the class was added (9.0.0) to 10.5.2 and main has it. The fix applies the filter in normalize() too, as RomanianAnalyzer does; it only changes the query side, so no reindex is needed.

• SerbianNormalizationFilter: "Normalizes Serbian Cyrillic and Latin characters to \"bald\" Latin" (SerbianNormalizationFilter.java:25). It maps all 30 lower-case Serbian Cyrillic letters and the Latin letters č ć š ž đ to ASCII (:41-170; ђ and đ become dj, љ becomes lj, њ becomes nj, џ becomes dz). It is the last filter of SerbianAnalyzer's chain (SerbianAnalyzer.java:125), so no indexed term of the field contains any of those 35 letters.
• Analyzer.normalize(field, text): what the query parsers call for a partial word, because the full analyzer would tokenize or stem it. It runs the analyzer's normalize(String, TokenStream) override, which should contain the normalizing filters and leave out the tokenizer and the stemmer (Analyzer.java:42-60, :203-212).
• Multi-term queries: prefix (Беогр*), wildcard (Бео?рад), fuzzy (Београд~1), regexp (/бео.*/) and range ([a TO š]). Term and phrase queries run the full analyzer and are not affected.

1. SerbianAnalyzer.normalize() is return new LowerCaseFilter(in); (SerbianAnalyzer.java:129-132). SerbianNormalizationFilter is missing.
2. The classic parser passes the prefix through it (QueryParserBase.java:712, BytesRef term = getAnalyzer().normalize(field, termStr);), and likewise the fuzzy term (:729), the regexp (:683), each wildcard chunk (:646, :656) and the range bounds (:550, :556; an open bound is skipped). The flexible parser's processors do the same (range bounds only when they do not parse as dates), and the simple parser does it for its only multi-term syntax, prefix (SimpleQueryParser.java:605) and fuzzy (:573).
3. Беогр* therefore becomes body:беогр* and Čač* becomes body:čač*. The index holds beograd and cack, so neither prefix can match any term.

One document Београд Čačak, default SerbianAnalyzer: Беогр*, Čač* and Београд~1 each find 0 documents; Beogr* and Cac* find 1. With stemming switched off through the stem exclusion set, normalize("Ђак") = ђак (index djak) and normalize("Čačak") = čačak (index cacak). The hunt's bounded scan found 5,334 of 14,794 one-token words that disagree with stemming off; every one of the 3,223 checked disagreeing words was missed by classic and flexible prefix, wildcard, regexp and range queries, while fuzzy queries still hit 1,804 of them within the edit distance.

• Who is exposed. Applications that use the stock SerbianAnalyzer, the default constructor included, and pass user text through QueryParser, StandardQueryParser or SimpleQueryParser, or call Analyzer.normalize themselves. Analyzer wrappers call the wrapped analyzer's normalize, and Lucene's own wrappers add nothing to it (AnalyzerWrapper.java:81-92, :124-126; DelegatingAnalyzerWrapper.java:57-60, the base of PerFieldAnalyzerWrapper), so wrapped uses inherit the gap unless a custom wrapper adds the filter in wrapTokenStreamForNormalization. A CustomAnalyzer built from the same factories is not affected: SerbianNormalizationFilterFactory.normalize() returns create(input) and CustomAnalyzer.normalize applies it.
• Default, not opt-in. Default analyzer and default parser settings. The only choice involved is using the Serbian analyzer and multi-term syntax.
• Ordinary input. Serbian is written in both Cyrillic and Latin; Latin spelling uses č ć š ž đ. Any multi-term query with one of these 35 letters is hit. The same query typed in plain Latin (Beogr*, Cac*) works, which is a workaround, not a fix.
• What the damage looks like. Prefix, wildcard and regexp terms that contain one of the letters cannot match any indexed term: 0 hits, no error. Ranges compare their bounds in the wrong form and can drop or add terms ([a TO š] admitted 6 of 6 probe documents; the indexed-form [a TO s] admits 2). Fuzzy terms can still hit within the edit distance, so say "miss", not "never match".
• Scale. One document and one query are enough. In the hunt's scan, with stemming off, 36% of the one-token words over a Serbian alphabet normalize differently from how they are indexed.
• Downstream (code reading only, nothing run). Elasticsearch exposes this class as its built-in serbian analyzer (elastic/elasticsearch#100921, 8.12.0), and its query_string parser normalizes prefix, wildcard, fuzzy and range terms with the field's analyzer, so it would inherit the miss.
• Limit. After the fix, prefixes that run past the stem stay best effort, as with every stemming analyzer (Čača* becomes caca, while the index holds cack). This bug is only the script and accent mismatch.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[djak]> but was:<[ђак]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.sr.TestSerbianAnalyzerNormalize.testNormalizeReturnsIndexedForm(TestSerbianAnalyzerNormalize.java:28)

• The contract. Analyzer.java:203-207: "Normalize a string down to the representation that it would have in the index. This is typically used by query parsers in order to generate a query on a given term, without tokenizing or stemming". The class example (:53-58) keeps a filter that is "about normalization" and drops one that is "about stemming".
• Lucene classes this filter as normalization. TokenFilterFactory.java:101-105: "filters that should be applied at normalization time can delegate to create"; SerbianNormalizationFilterFactory.java:71-74 does exactly that, and the factory has been multi-term aware since it was added (LUCENE-6053, 2014). A CustomAnalyzer built from the same factories normalizes correctly.
• Siblings. RomanianAnalyzer.java:128-133 applies LowerCaseFilter and RomanianNormalizationFilter in normalize(). Of the stock analyzers that override normalize() and use a filter whose factory overrides normalize(), every other one applies that filter there (German, Romanian, Persian, Arabic, Hindi, Bengali, Telugu, Thai, Greek, Turkish and others); SerbianAnalyzer is the only exception.
• Tests. No upstream test asserts the current behaviour: TestSerbianAnalyzer's four tests cover resources, stemming and a stop word, the exclusion set and random strings; none asserts what normalize() returns.
• History. The lower-case-only body came with the class (c7029b12541d, LUCENE-9313) from the contributor's patch; the issue thread never mentions normalize, wildcards or prefixes. The contract (6.2.0) and the factory's normalize() -> create() (8.0.0) are older. Precedents fixed the same shape: #13059 ("DecimalDigitFilter and IndicNormalizationFilter were mistakenly omitted", CHANGES "Bug Fixes") and #13233 (a committer added the Romanian filter to both the chain and normalize(), for "Analyzer.normalize() (normalizing the characters for e.g. wildcard searches)" and for "Users who make use of stemExclusionSet"); #16717 (Thai) did the same.

The strongest "intended" case (the devil's-advocate verifier and the cross-model reviewers): normalize() deliberately excludes stemming; the Snowball SerbianStemmer itself converts Cyrillic to Latin as its first step; and SerbianNormalizationFilter sits after the stemmer, so it could be read as stemmer clean-up. Also, a comment in the classic parser's wildcard code says "best effort" (QueryParserBase.java:638), and its getPrefixQuery javadoc says a prefix term does not go "through the default Analyzer" (:695-697).

Why it fails: the stem-excluded, Latin-only case Čačak involves no stemmer and still disagrees (index cacak, normalize čačak); #13233 is the same situation, a stemmer that already normalizes, and the committer still added the filter to normalize() for wildcard searches and stem-excluded words; the Analyzer javadoc decides by what a filter does, not where it sits; the "best effort" comment is about keeping wildcard and escape characters out of normalize(), and the prefix javadoc's "will not go through the default Analyzer" can at most mean the full analysis chain: the code a few lines below passes the prefix through the analyzer's normalize() (:712). Every reviewer rated the intended case weak as evidence of intent. The counter-case is right that multi-term queries on a stemmed field stay approximate after the fix and that the fix changes how some regexps and fuzzy terms are read (see the fix's limits); those are limits to state in the report, not signs that the gap is intended. Verdict: a defect.

One fix was built and checked in a scratch worktree of main ea57360912ed; the other options are optional extras or workarounds.

Two lines in SerbianAnalyzer.normalize(), the RomanianAnalyzer pattern. The filter "expects lowercased input" (SerbianNormalizationFilter.java:30), which the LowerCaseFilter before it provides. A Telugu-style regression test would be assertEquals(new BytesRef("djak"), a.normalize("dummy", "Ђак")) in TestSerbianAnalyzer, or the repro's test class.

@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
  TokenStream result = new LowerCaseFilter(in);
  return new SerbianNormalizationFilter(result);
}

Trade-offs

• Query side only: the index already holds the plain-Latin form, so no reindex.
• Behaviour change: the tested Cyrillic and accented prefix, wildcard, fuzzy, regexp and range queries now use the indexed form and hit (executed), which is the intent.
• Cost: one character-mapping filter per normalized query term.
• Applies unchanged to every affected branch: the method body is identical from 9.0.0 to main.
• Prefixes that run past the stem stay best effort, as with every stemming analyzer (Čača* becomes caca*, 0 hits, the index holds cack; executed).
• Escaped letters in a wildcard query skip normalize() (QueryParserBase.java:637-659): Бео\г?ад parses to body:beo\г?ad, 0 hits, with or without the fix (executed).
• The whole regexp string goes through normalize() (QueryParserBase.java:683), so classes and repeats are rewritten in Latin: /[đž]ak/ becomes /[djz]ak/ and matches zak, dak and jak but not djak; /đ+ak/ becomes /dj+ak/ (executed).
• Fuzzy distances are counted on the Latin form, where đ ђ љ њ џ are two letters: Ђак~1 becomes djak~1 and matches djak, dak and jak but not ak, one edit from ђак in Cyrillic (executed).
• Ranges compare the UTF-8 bytes of the indexed terms, not Serbian alphabetical order (TermRangeQuery.java:27-28), with or without the fix.
• Analyzer.normalize() is public: application code that calls it directly also gets the Latin form after the fix.

What the checks validated

• Scratch worktree at ea57360912ed, gradle, JDK 25: the new test fails 2 of 2 on stock and passes with the fix; org.apache.lucene.analysis.sr.*: 5 suites (4 upstream with 15 tests, plus the new one), 17 tests, 0 failed (seed 1A2B3C4D, and 77AA77AA with assertions off); :lucene:analysis:common:test: 340 suites, 1,911 tests, 18 skipped, 0 failed; checkGoogleJavaFormat passes.
• Standalone runner with the fix shadowed: the new test 2 of 2 (assertions on and off), the QueryParser test 3 of 3 and the security probe 9 of 9 pass; earlier, the independent reproducer's 8-test JUnit passed 8 of 8 (twice, including the adjudication's re-run) and the upstream TestSerbianAnalyzer 4 of 4.

SerbianAnalyzer.java:39 says @since 8.6, but no 8.x release contains the class (8.6.3 and 8.11.4: the repro does not compile) and CHANGES lists it under 9.0.0.

Trade-offs

Javadoc only; unrelated to the behaviour, so it could also be a separate change.

The hunt's scan did exactly this across all stock analyzers and found sibling gaps.

Trade-offs

Larger change: stemming and tokenization must be kept out per analyzer, and some sibling gaps would fail it today, so each needs its own fix first.

Subclass SerbianAnalyzer (it is not final) and override normalize() as above; or build the chain with CustomAnalyzer from the factories, whose normalize() already applies serbianNormalization; or type multi-term queries in plain Latin (Beogr*, Cac*).

Trade-offs

The subclass and CustomAnalyzer routes need no reindex either; asking users to type plain Latin is not a real fix.

Method: a bounded-exhaustive metamorphic scan on the stock main jars (the study's "semi-formal" tier). A hunt unit, "analyzer normalize consistency", 2026-10-26.

• Relation N1: Analyzer.normalize(f, w) equals the single full-span token that indexing produces for w. Inputs: every word of length 1-2 over a 110-code-point alphabet that includes Serbian Cyrillic and Latin letters, plus 4,000 random words each of length 3 and 4, times 99 analyzer configurations (stemming switched off through stem exclusion sets where the analyzer has one): 1,437,218 comparisons. SerbianAnalyzer with stemming off disagreed on 5,334 of 14,794 words.
• Relation N2 (parser reachability): each disagreeing word, used as a prefix, wildcard, fuzzy, regexp and range query through the classic, flexible and simple parsers, must still find its document. All 3,223 checked disagreeing words were missed by prefix, wildcard, regexp and range; fuzzy still hit 1,804.
• A second pass read every normalize() override in Lucene side by side with its createComponents; SerbianAnalyzer was the only one leaving out a filter that its factory declares as normalization.
• Then an AI finder wrote the public-API JUnit, and independent AI tools reproduced and judged it (see the checks).