UkrainianMorfologikAnalyzer has no normalize(): prefix, wildcard, regexp, range and fuzzy terms skip its normalization

UkrainianMorfologikAnalyzer normalizes every word before it is indexed: its initReader maps the apostrophe variants U+2019, U+2018, U+02BC, backtick and U+00B4 to ', removes the stress mark U+0301 and the soft hyphen U+00AD and maps ґ to г, and its chain lower-cases before the dictionary lemmatizer. So м’ясо Lucene is indexed as м'ясо and lucene. For prefix, wildcard, regexp, range and fuzzy terms the classic and flexible query parsers call Analyzer#normalize (the simple parser for prefix and fuzzy terms), documented to return "the representation that it would have in the index". The class overrides neither initReaderForNormalization nor normalize(String, TokenStream), and both default to the identity, so normalize("Lucene") returns Lucene and the parsed queries м’яс*, Luc*, Luce?e and [Lucene TO Lucene] find nothing, while м'яс* and luc* find the document. Term and phrase queries and the index are unaffected. LUCENE-7355 created normalize in 2016 and added overrides to 42 named stock analyzers (plus CustomAnalyzer), but skipped this class, committed a week before the change was authored: Analyzer#normalize is wrong in every release from 6.2.0 to 10.5.2 and main, and the classic parser's upper-case symptom starts in 7.0.0. Two overrides fix it on the query side, without reindexing. The character-mapping one has no side effect; the lower-casing one makes prefixes and ranges of capitalised dictionary lemmas such as Київ* stop matching and lower-cases regexp escapes (\D to \d), as every stock analyzer that lower-cases in normalize already does.

• Analyzer chain: the steps that turn text into indexed terms. This analyzer's chain is a MappingCharFilter (initReader), then StandardTokenizer -> LowerCaseFilter -> StopFilter -> (SetKeywordMarkerFilter) -> MorfologikFilter, the dictionary lemmatizer (UkrainianMorfologikAnalyzer.java:151-154, :165-177).
• NORMALIZER_MAP (:47-64): U+2019, U+2018, U+02BC, backtick and U+00B4 to '; U+0301 (stress mark) and U+00AD (soft hyphen) removed; ґ/Ґ to г/Г.
• Multi-term query: prefix (Luc*), wildcard (Luce?e), regexp (/Luc.*/), range ([a TO b]) and fuzzy (Lucene~1). Their term is a pattern or a partial word, so the parsers do not tokenize or stem it.
• Analyzer#normalize(field, text) (Analyzer.java:202-213): "Normalize a string down to the representation that it would have in the index. ... This method uses initReaderForNormalization ... to apply necessary character-level normalization and then normalize(String, TokenStream) in order to apply the normalizing token filters." Both hooks default to the identity (:122-129, :279-286).

1. Indexing runs initReader and createComponents: м’ясо Lucene becomes the terms м'ясо and lucene.
2. The parsers send multi-term terms through normalize: classic QueryParserBase.java:550, :556 (range), :646, :656 (wildcard), :683 (regexp), :712 (prefix), :729 (fuzzy); simple SimpleQueryParser.java:573, :605; flexible WildcardQueryNodeProcessor.java:66, :76, TermRangeQueryNodeProcessor.java:138-139, FuzzyQueryNodeProcessor.java:59, RegexpQueryNodeProcessor.java:43.
3. The class overrides only initReader and createComponents, so normalize returns its input: м’яс* stays body:м’яс*, Luc* stays body:Luc*.
4. No indexed term can contain a mapped character: every pass-through token goes through the char filter, and an exhaustive scan of all 3,485,758 entries of the bundled dictionary found none in any lemma or form. Pass-through tokens are lower-cased; a dictionary word is replaced by one or more lemmas, some capitalised. So the unchanged м’яс and Luc cannot match; a capitalised term matches only a capitalised lemma (Київ* finds Київ).

normalize itself, with each word's indexed form checked first: Lucene -> Lucene (index lucene), м’ясо -> м’ясо (index м'ясо), м'я\u0301со -> unchanged (index м'ясо), сек\u00ADрет -> unchanged (index секрет), ґрунт -> ґрунт (index грунт).

• Who is exposed. Applications that index with the stock UkrainianMorfologikAnalyzer and pass multi-term syntax through the classic, flexible or simple query parser, or call Analyzer#normalize themselves. By code reading only, the analysis-ukrainian plugins of two downstream search servers (Elasticsearch and OpenSearch) construct the stock class unchanged; their query paths were not tested.
• Default or opt-in. Default constructor, default parser settings, default IndexWriterConfig, one document. The opt-in is the language analyzer itself (the optional analysis-morfologik module, Ukrainian-language indexes) and multi-term syntax.
• Triggers. A prefix, wildcard, regexp or range term that keeps one of the five apostrophe forms the analyzer maps (U+2019, U+2018, U+02BC, backtick, U+00B4), a stress mark, a soft hyphen or ґ (no indexed term contains one), or that has an upper-case letter where the word is indexed in lower case (every pass-through word and every lower-case lemma). A capitalised term still matches a capitalised dictionary lemma (stock Київ*, Гет*, Чайков*: 1 hit each). The apostrophe sits inside common words (м’ясо, п’ять, сім’я); ґ is a letter of the alphabet, so a correctly spelled ґрунт* cannot match грунт.
• Scale (measured on synthetic strings). A scan over every 1- and 2-character string of a 110-character multilingual alphabet plus random 3- and 4-character strings: of the 14,854 strings that index as one token (lemmatization almost entirely switched off), normalize disagreed with the indexed token on 7,983: 7,979 indexed as the lower-cased, mapped input and 4 as the capitalised lemma Гб, so every disagreement is a case or mapped-character difference. All 4,786 disagreeing strings checked were missed by classic, flexible and flexible multi-field prefix, wildcard, regexp and range queries and by simple-parser prefix queries; fuzzy still found 3,606. All 5,246 agreeing strings were found.
• What you see. Fewer hits, no error. Term and phrase queries and the index are unaffected; for ranges an upper-case bound compares in the wrong representation and can also admit extra terms.
• No workaround inside the class. It is final, so users cannot subclass it; an AnalyzerWrapper works (Suggested fix, option E).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<м[']ясо> but was:<м[’]ясо>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.uk.TestUkrainianAnalyzerNormalizeApostrophe.testNormalizeMapsTypographicApostrophe(TestUkrainianAnalyzerNormalizeApostrophe.java:27)

• Contract. Analyzer#normalize: "Normalize a string down to the representation that it would have in the index", applying "necessary character-level normalization" through initReaderForNormalization (Analyzer.java:203, :209-211), which is "typically a subset of the CharFilters that are applied in initReader" (:280-282). The class's char map is named NORMALIZER_MAP (UkrainianMorfologikAnalyzer.java:47). The Analyzer class javadoc example keeps the normalization filter in normalize and drops only the stemming filter (Analyzer.java:45-60).
• MIGRATE.txt (LUCENE-7355, 7.0): "CommonQueryParserConfiguration.setLowerCaseExpandedTerms removed ... This option has been removed as expanded terms are now normalized through Analyzer#normalize."
• Siblings. RussianAnalyzer and PolishAnalyzer override normalize with LowerCaseFilter; ThaiAnalyzer and JapaneseAnalyzer also override initReaderForNormalization with their char filters. This is the only stock search analyzer whose chain lower-cases and char-maps and that overrides neither hook. Two analyzers follow the convention only in part: JapaneseCompletionAnalyzer lower-cases at index time but overrides only initReaderForNormalization (JapaneseCompletionAnalyzer.java:52, :62), and PersianAnalyzer char-filters in initReader (PersianAnalyzer.java:158-160) without initReaderForNormalization, which the devil's advocate judged a tokenization aid (ZWNJ to space).
• History. The class was committed on 2016-06-21 (LUCENE-7287); LUCENE-7355 (authored 2016-06-28, committed 2016-07-12) added overrides to 42 named stock analyzers (plus CustomAnalyzer, the test framework's MockAnalyzer and Solr's TokenizerChain) and not to it, and no commit since has. On LUCENE-7355 its author described the design as applying "all normalization filters (lowercasing, ascii folding, etc.) but not transformations (stop word removal, stemming, etc.)". LUCENE-7841 added ґ -> г because "the search will benefit". LUCENE-7429, the same omission in DelegatingAnalyzerWrapper, was fixed as a bug; LUCENE-8022 was resolved Invalid because the analyzer must override normalize, and here that analyzer is Lucene's own and final.
• Tests. No upstream test asserts identity normalization; checkRandomData (through checkAnalysisConsistency) only checks that normalize does not throw ("TODO: what can we do besides testing that the above method does not throw?").

The strongest "intended" case (an intended-lens reviewer from another model family rated it moderate; the devil's advocate and the refute review weak; a later cross-model fact check moderate overall, weak for the character mapping, moderate for the lower-casing): the identity defaults are documented and optional; the classic parser's javadoc still says prefix and wildcard terms "will not go through the default Analyzer, however, since normal Analyzers are unlikely to work properly with wildcard templates" (QueryParserBase.java:610-612, :695-697; :666-668 says the same for regexps with "regular expression templates"); 265,482 of the 3,485,758 dictionary entries (7.6%) have a lemma with an upper-case letter (proper nouns such as Київ, also all-caps forms), so a lower-casing normalize makes Київ* stop matching; the dictionary replaces a word with one or more lemmas (MorfologikFilter.java:121-135), so for dictionary words no single string is "the representation that it would have in the index"; and a lower-casing normalize also lower-cases regexp escapes and range bounds.

Why it fails: optional defaults let arbitrary analyzers skip the hooks; they do not say a stock analyzer that lower-cases and char-maps should, and every other such stock analyzer got at least one of the two overrides. The parser javadoc is outdated rather than a statement of intent: it still describes the removed lowercaseExpandedTerms setting ("Depending on settings, ... may be lower-cased automatically") and does not mention that the same methods call getAnalyzer().normalize(...) (:683, :712); MIGRATE.txt says expanded terms "are now normalized through Analyzer#normalize". Its "will not go through the default Analyzer" is still true of the full chain. The capitalised-lemma and one-to-many points concern only the lower-casing half: normalize can be exact only for pass-through words, and those are lower-cased; in the history probe's classic-parser runs the capitalised-lemma match exists only since 7.0.0 (the 6.x classic parser lower-cased the terms and missed Київ* too). The regexp-escape effect is what StandardAnalyzer already does on main (/\D+/ becomes /\d+/). The character-mapping half has no trade-off at all: no indexed term can contain a mapped character.

Suggested route: lead with option B and offer option A's lower-casing for discussion (the adjudication's advice and the cross-model fact check's); proposing A in one change with its trade-offs stated is the other route. Option A was validated in a scratch worktree at ea57360912ed (removed afterwards); option B through gradle (the repro, and the whole module in the fact checker's run) and the runner; the option-B variant of testNormalize through the runner; option E on main through the runner. The trade-off tables come from two classic-parser probes run against stock, A and B.

initReaderForNormalization reuses the existing NORMALIZER_MAP; normalize applies LowerCaseFilter, as RussianAnalyzer and PolishAnalyzer do. Query side only: indexing does not call these hooks. The code below is the whole file, which git apply applies to ea57360912ed. fix-with-test.diff adds testNormalize to TestUkrainianAnalyzer (the PR shape for this route). Add a CHANGES entry under Bug Fixes, and fix the outdated QueryParserBase javadoc in the same change (option C).

diff --git a/lucene/analysis/morfologik/src/java/org/apache/lucene/analysis/uk/UkrainianMorfologikAnalyzer.java b/lucene/analysis/morfologik/src/java/org/apache/lucene/analysis/uk/UkrainianMorfologikAnalyzer.java
index 3915d264d7f..c3b8468b666 100644
--- a/lucene/analysis/morfologik/src/java/org/apache/lucene/analysis/uk/UkrainianMorfologikAnalyzer.java
+++ b/lucene/analysis/morfologik/src/java/org/apache/lucene/analysis/uk/UkrainianMorfologikAnalyzer.java
@@ -153,6 +153,16 @@ public final class UkrainianMorfologikAnalyzer extends StopwordAnalyzerBase {
     return new MappingCharFilter(NORMALIZER_MAP, reader);
   }
 
+  @Override
+  protected Reader initReaderForNormalization(String fieldName, Reader reader) {
+    return new MappingCharFilter(NORMALIZER_MAP, reader);
+  }
+
+  @Override
+  protected TokenStream normalize(String fieldName, TokenStream in) {
+    return new LowerCaseFilter(in);
+  }
+
   /**
    * Creates a {@link org.apache.lucene.analysis.Analyzer.TokenStreamComponents} which tokenizes all
    * the text in the provided {@link Reader}.

Trade-offs

Prefixes of capitalised dictionary lemmas stop matching, because the index holds them capitalised and the query term is now lower-cased (classic parser; documents м’ясо Lucene, Київ, Цих Чайковського і Ґете.):

A prefix typed in lower case (київ*) finds nothing on stock, A or B; in the history probe's classic-parser runs the capitalised-lemma match exists only since 7.0.0 (the 6.x classic parser lower-cased the terms). 265,482 of the 3,485,758 dictionary entries (7.6%) have a lemma with an upper-case letter. SOLR-11835 (2018) says the dictionary holds proper names in lower case since 6.6; on main the analyzer emits capitalised lemmas (testCapsTokenStream asserts Чайковське, Чайковський and Гете; the probes show Київ), so reconcile the two before describing the side effect. Cost: one MappingCharFilter and one LowerCaseFilter over one query term; nothing at index time.

The whole regexp is lower-cased, escapes included, so \D, \S, \W become \d, \s, \w (the opposite classes) and [^L] becomes [^l]; a range bound on a capitalised lemma stops matching. StandardAnalyzer, whose normalize already lower-cases, does the same to regexps today; with its lower-cased index the range still matches (parsed query and hits; the probe also indexes the document 2026):

The regexp and range cases are in no repro or security-probe test; they come only from this measurement probe. The lower-casing half also turns a deny prefix written with a capitalised lemma (*:* -Київ*) into a no-op, so prefix rules on this field stay best effort either way.

What the checks validated

• Scratch worktree at ea57360912ed: the repro passes with assertions on and off (it fails on stock); TestUkrainianAnalyzer 8 tests (9 with testNormalize), the whole analysis-morfologik suite (21 tests with the repro, 22 with testNormalize, assertions on and off) and :lucene:analysis:morfologik:check (google-java-format, ecj, forbidden APIs) pass. testNormalize fails on stock at its first assertion.
• As a shadow source on main: the parser-level test 2 of 2, the independent reproducer's test 4 of 4, the intended-lens probe 2 of 2, the security probe 17 of 17. Back-ported to 7.0.0 every history-probe group passes; on 6.2.0 and 6.6.6 the normalize and simple-parser groups.

The character-mapping half alone: initReaderForNormalization returning new MappingCharFilter(NORMALIZER_MAP, reader). The snippet shows the added lines.

+  @Override
+  protected Reader initReaderForNormalization(String fieldName, Reader reader) {
+    return new MappingCharFilter(NORMALIZER_MAP, reader);
+  }

Trade-offs

No side effect: no indexed term contains a mapped character, so nothing that matches today stops matching; Ґет* starts to find Гете; regexps and ranges keep their stock behaviour (option A's tables). Upper-case terms (Luc*, Luce?e, [Lucene TO Lucene]) still miss lower-cased words; the lower-casing can follow as a separate, discussed change.

What the checks validated

The repro then fails only on the upper-case entries ([Lucene -> Lucene (index holds lucene)], [Luc* -> body:Luc* hits=0]), through gradle and the runner; the fact checker's gradle run of the whole module with option B: 21 tests, 2 failures, only those repro entries, the 19 existing module tests pass. A test change for this route (testNormalize without its upper-case assertion, 5 assertions) passes git apply --check; through the runner TestUkrainianAnalyzer with it fails 1 of 9 on stock (м’ясо keeps U+2019) and passes 9 of 9 with option B and with option A, assertions on and off. It was not run through gradle check (it is the checked testukrainiananalyzer-test.diff minus one assertion line, with a shorter javadoc line). Trade-off columns B in option A's tables.

QueryParserBase.java:610-612, :666-668, :695-697 still say that, "depending on settings", the term "may be lower-cased automatically" (the lowercaseExpandedTerms setting removed in 7.0) and that it "will not go through the default Analyzer" (still true of the full chain); they do not say that the term goes through Analyzer#normalize. Documentation only; goes with A or B. Not built.

Would remove the capitalised-lemma trade-off of option A.

Trade-offs

Changes the indexed terms (reindex), breaks the upstream assertion testCapsTokenStream (Чайковське, Чайковський, Гете) and changes term queries. Not run.

Wrap the stock analyzer in an AnalyzerWrapper whose wrapReaderForNormalization applies a copy of the nine-entry map (the original is private) and whose wrapTokenStreamForNormalization applies LowerCaseFilter; use it for indexing and parsing (indexing is unchanged).

Trade-offs

Same capitalised-lemma trade-off as option A; the copied map must be kept in sync by hand.

What the checks validated

On main with the classic parser: stock м’яс*, Luc*, Luce?e find 0; with the wrapper they, lucene and the phrase "м’ясо Lucene" find the document (1 each).

Method: bounded-exhaustive metamorphic testing, a hunt unit (analyzer normalize consistency), 2026-10-26 (the hunt's "semi-formal" tier).

• Inputs: 20,207 strings, every 1- and 2-character string over an explicit 110-code-point multilingual alphabet plus random 3- and 4-character strings (fixed seed).
• Relation N1: for every stock analyzer and every input w that indexes as exactly one full-span token, Analyzer.normalize(f, w) equals that token. UkrainianMorfologikAnalyzer disagreed on 7,983 of 14,854 compared strings with lemmatization (almost entirely) switched off through the stem exclusion set (7,971 of 14,838 with the default constructor); the disagreements are case and mapped-character differences (7,979 indexed as the lower-cased, mapped input, 4 as the capitalised lemma Гб; the fact checker's re-run).
• Relation N2 (parser reachability): each string, indexed alone, must be found by its own prefix, wildcard, regexp, range and fuzzy query through the classic, flexible, flexible multi-field and simple parsers. All 4,786 checked disagreeing strings were missed by every non-fuzzy path; fuzzy found 3,606; all 5,246 agreeing strings were found.
• Then: a public-API JUnit; a review by an independent reviewer from another model family (OpenAI GPT-5.6, read-only: "not real, low", procedural, because it could not run its own test; it agreed with the root cause); an independent reproducer; an intended-behaviour review by the same model family (read-only: real, medium); a devil's advocate (with an exhaustive scan of the bundled dictionary); an adjudication; a history sweep; a prior-report search; a security assessment; a registry re-run; an audit. Neither reviewer from the other model family ran code; separate runs re-ran the finder's test, which failed.
• Afterwards: a fact check of the brief and this doc (it re-ran the repro, the fix variants and the probes) and a read-only fact check by the other model family corrected overstatements about upper-case terms and added the regexp and range trade-offs.