The kuromoji small-kana filter factories have no normalize(), so CustomAnalyzer prefix, wildcard, fuzzy, regexp and range queries with small kana can miss documents

The kuromoji token filters japaneseKatakanaUppercase and japaneseHiraganaUppercase (added in 9.11.0) map the 28 small katakana and 12 small hiragana (sutegana, 捨て仮名) to full-size kana, so a document キャット is indexed as キヤツト. Lucene's query parsers build prefix, wildcard, fuzzy, regexp and range queries from Analyzer.normalize, which should return "the representation that it would have in the index". A CustomAnalyzer builds normalize from each factory's normalize(), and these two factories override only create(), so they inherit the identity default and normalize keeps the small kana. The term query キャット finds the document, but キャ* and [キャット TO キャット] find nothing, with no error. It has been there since the filters were added in 9.11.0 (executed on five releases through 10.6.0 and on main; the factory files are byte-identical in all 19 release tags); a small normalize() override (+5 lines) in each factory, as CJKWidthFilterFactory has, fixes it.

1. At index time CustomAnalyzer.createComponents chains each factory's create() (CustomAnalyzer.java:161-167 on main 645d4fe14a20). JapaneseKatakanaUppercaseFilter rewrites each small katakana through a 28-entry map and the pair ㇷ゚ (U+31F7 U+309A) to プ (JapaneseKatakanaUppercaseFilter.java:43-70, :86-97); JapaneseHiraganaUppercaseFilter maps 12 small hiragana one to one (JapaneseHiraganaUppercaseFilter.java:43-54, :69-74). Both read only the term text.
2. The classic QueryParserBase sends range bounds (:550, :556), wildcard chunks (:646, :656), the whole regexp (:683), prefixes (:712) and fuzzy terms (:729) through Analyzer.normalize; the flexible parser's processors (TermRangeQueryNodeProcessor.java:138-139, FuzzyQueryNodeProcessor.java:59, WildcardQueryNodeProcessor.java:66, :76, RegexpQueryNodeProcessor.java:43) and SimpleQueryParser (fuzzy :573, prefix :605) do the same.
3. CustomAnalyzer.normalize calls normalize() on each of its token filter factories (CustomAnalyzer.java:170-177).
4. Both sutegana factories override only create() (JapaneseKatakanaUppercaseFilterFactory.java:45-48, JapaneseHiraganaUppercaseFilterFactory.java:45-48), so TokenFilterFactory.normalize runs: "Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate to {@code create} method." (TokenFilterFactory.java:101-108). CJKWidthFilterFactory delegates (:62-65).
5. So normalize keeps every small kana, while indexed terms never contain one. A bounded-exhaustive check on the real jars, over all 1,112,064 Unicode scalar values, finds that normalize differs from the indexed term for exactly the 28 small katakana and 12 small hiragana the filters map, and for none with the fix.

Main 645d4fe14a20, -ea and -da, one document per row, CustomAnalyzer.builder().withTokenizer("whitespace").addTokenFilter(<filter>), classic QueryParser, hit counts:

With the chain of the upstream factory test (japanese tokenizer + japaneseKatakanaUppercase), the document ストップウォッチ (indexed as ストツプウオツチ) gets 1 hit for the term query and 0 for ストップ*, スト?プウォッチ and ストップウォッチ~1, in both the classic and the flexible StandardQueryParser. The fuzzy query misses although its text is the document text: unnormalized, it is 3 edits away from the indexed term.

Not every multi-term query with a small kana returns nothing. With the documents キャット and コーヒー (same chain, classic parser, -ea and -da): キャット~1 gets 0 hits but キャット~2 gets 1 (2 substitutions from キヤツト); /キャ.*/ gets 0 but /キ(ャ|ヤ)ツト/ gets 1 through its other branch; and [ア TO ヶ] gets 2, one more than with the fix ([ア TO ケ]), because ヶ (U+30F6) sorts after コ (U+30B3) while its mapped form ケ (U+30B1) sorts before it. Literal prefix and wildcard chunks with a mapped small kana always miss; fuzzy, regexp and range results depend on edit distance, pattern structure and bound order.

• Who is hit. A CustomAnalyzer (or another chain built from the factories by SPI name) with japaneseKatakanaUppercase or japaneseHiraganaUppercase, used with the classic, complex-phrase, flexible or simple query parser for range, fuzzy, prefix, wildcard or regexp terms (each parser for the syntax it has; SimpleQueryParser has only prefix and fuzzy). Executed: classic prefix, range and regexp; classic and flexible prefix, wildcard and fuzzy; and, in the independent fact check, complex-phrase prefix and simple prefix and fuzzy (0 hits each for both filters, against 1 for cjkWidth).
• Opt-in. No stock analyzer uses these filters; JapaneseAnalyzer's normalization applies CJKWidthCharFilter (JapaneseAnalyzer.java:130-133) and LowerCaseFilter (:119-123).
• Ordinary input once opted in. Small kana are everyday Japanese (キャ, ちょっ, the ッ of most loanwords), and the filters' documented purpose is to "search against old style Japanese text such as patents, legal, contract policies, etc." with modern spellings (JapaneseKatakanaUppercaseFilter.java:32-33). A literal prefix or wildcard pattern typed with a mapped small kana then matches nothing in that field.
• How each query type goes wrong. The query text keeps its small kana, while no indexed term contains a mapped one. Prefix and wildcard: a literal chunk with a mapped small kana never matches. Fuzzy: the kept small kana add to the edit distance, one substitution each here (キャット~1 0 hits, キャット~2 1). Regexp: a literal mapped small kana misses, but another branch can still match (/キ(ャ|ヤ)ツト/ 1). Range: an unmapped bound sorts elsewhere, so a range can miss documents ([キャット TO キャット] 0) or return extra ones ([ア TO ヶ] 2 against 1 with the fix).
• Silent. No exception and no warning; the query just matches fewer documents (or, for some range bounds, more). A query typed in the indexed (full-size) form still works, which is easy to miss in tests.
• No scale or unusual parameter. One token, deterministic, with assertions on and off.
• Workaround. Subclass each factory, add normalize() returning create(input), and pass the subclass to CustomAnalyzer.builder().addTokenFilter(Class) (CustomAnalyzer.java:347); checked on stock main. CustomAnalyzer is final (:99), so its normalize cannot be overridden; an AnalyzerWrapper whose wrapTokenStreamForNormalization adds the two filters works by code reading (not run).
• Downstream (not run). By code reading, Solr's TokenizerChain.normalize also calls each factory's normalize(), so a Solr field type with these factories most likely misses the same way. Two downstream projects wrap or copy the filters (elastic/elasticsearch#106553, azagniotov/solr-lucene-analyzer-sudachi#110).

The first assertion (indexing maps キャちょ to キヤちよ) passes; the second (normalize returns the indexed form) fails. The query consequence (0 hits for キャ*, [キャット TO キャット] and the others) is not part of the test.

The file passes the kuromoji module's checkGoogleJavaFormat, ecjLint and forbiddenApis as is. Name tests.asserts explicitly: the build randomizes it per seed otherwise. The standalone runner on the 10.6.0 release jars (-ea and -da) and on 9.11.0 (-da) prints the same failure.

TestSuteganaFactoriesNormalize > testNormalizeMapsSmallKana FAILED
    org.junit.ComparisonFailure: expected:<キ[ヤちよ]> but was:<キ[ャちょ]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:AECD5D7A0C4BB052]:0)
        at app/junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:117)
        at app/junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
        at app//org.apache.lucene.analysis.ja.TestSuteganaFactoriesNormalize.testNormalizeMapsSmallKana(TestSuteganaFactoriesNormalize.java:34)

1 test completed, 1 failed

• The contract. Analyzer.normalize: "Normalize a string down to the representation that it would have in the index. This is typically used by query parsers in order to generate a query on a given term, without tokenizing or stemming, which are undesirable if the string to analyze is a partial word (eg. in case of a wildcard or fuzzy query)." (Analyzer.java:202-207). The Analyzer class javadoc gives the rule: "Assuming FooFilter is about normalization and BarFilter is about stemming, only FooFilter should be applied" (Analyzer.java:55-56).
• These filters are normalizers. "A {@link TokenFilter} that normalizes small letters (捨て仮名) in katakana into normal letters." (JapaneseKatakanaUppercaseFilter.java:29; the hiragana filter says the same). They are character-level maps, not stemmers, and read no tokenizer attributes: the hiragana filter maps one character to one, and the katakana filter also looks one character ahead to join ㇷ + U+309A into プ (JapaneseKatakanaUppercaseFilter.java:86-90). #12915's own description says they "normalize sutegana to normal (uppercase) characters".
• The opt-in exists for exactly this. "filters that should be applied at normalization time can delegate to {@code create} method" (TokenFilterFactory.java:101-104). CJKWidthFilterFactory (:62-65) and kuromoji's own JapaneseIterationMarkCharFilterFactory (:72-75) delegate. Of the factories added since LUCENE-8497 (2018), the character-level normalizers are CJKWidthChar, NorwegianNormalization, TeluguNormalization, RomanianNormalization (merged 10 days after #12915), HangulCompositionChar, CaseFolding, ThaiChar and ThaiNormalization: all 8 override normalize(); the sutegana pair is the only exception.
• Nothing documents an exclusion. No javadoc, CHANGES.txt or MIGRATE.md entry, test or code comment says these filters are skipped by normalize; the upstream factory tests call only create(). #12915's body, conversation and inline review comments use "normalize" only as a verb for what the filters do, and never mention the normalize() method, Analyzer.normalize, multi-term queries or query parsers.
• Identity contradicts the contract here. Indexed terms never contain the 40 mapped characters, so a literal prefix or wildcard chunk that keeps one can never match a document in that field, and fuzzy, regexp and range terms are compared against text that is not in the indexed form.
• Labeling precedent is mixed. Similar fixes were listed under "Bug Fixes" (LUCENE-7692 in 6.5.0; GITHUB#13059, a missing step in TeluguAnalyzer.normalize, in 9.10.0) and under "Improvements" (LUCENE-7353 in 6.2.0), so this fix could be listed either way.

The strongest case for "intended" (the intended-behaviour checker and the GPT-5.6 intended-lens reviewer): normalization is opt-in and the identity default is documented, and TestCustomAnalyzer asserts that factories that do not opt in are skipped; the katakana filter is not purely per character, because ㇷ゚ becomes プ and a wildcard can split the pair; LUCENE-8022 was resolved Invalid because a user's own Analyzer must override normalize; and upstream has filed similar factory fixes as improvements.

Why it fails: the opt-in is for filters "that should be applied at normalization time", and by the Analyzer javadoc's own rule a normalizing (not stemming) filter is one; that test checks the plumbing with dummy factories, not which real factories should opt in. CJKWidthFilter composes half-width voiced marks the same way and its factory opts in; the hiragana filter is a pure one-to-one map. With the fix the exhaustive check over all 71,289 two-character kana strings leaves exactly one prefix exception (ㇷ + ゚) for katakana and none for hiragana, against 59 for cjkWidth; on stock main it finds 7,476 and 3,204. CustomAnalyzer is final and builds normalize only from its factories, so the responsibility LUCENE-8022 puts on a hand-written Analyzer sits in the factory here. How upstream labels the fix does not change the behaviour. Verdict: not intended, a defect (every checker agrees).

One fix was built and validated; one alternative is described. Workaround until a fix ships: subclass each factory with normalize() returning create(input) and pass the subclasses to CustomAnalyzer.builder().addTokenFilter(Class) (checked on stock main), or wrap the analyzer in an AnalyzerWrapper that adds the two filters in wrapTokenStreamForNormalization (not run).

Add the same normalize() override (+5 lines) to JapaneseKatakanaUppercaseFilterFactory and JapaneseHiraganaUppercaseFilterFactory (+10 lines in total), the pattern of CJKWidthFilterFactory and of the other character-level normalizer factories. The factory files are byte-identical in every release since 9.11.0, so the diff applies unchanged to any backport line.

@Override
public TokenStream normalize(TokenStream input) {
  return create(input);
}

Trade-offs

• One digraph at a wildcard boundary. With the fix normalize equals the indexed term for every whole term, but prefix and wildcard chunks are normalized separately: a chunk ending in ㇷ before * gives フ, while a document with ㇷ゚ is indexed as プ. The exhaustive prefix check finds exactly this one case (and 59 such cases for the accepted cjkWidth).
• Behaviour change. Multi-term queries typed with small kana start to match; queries typed in full-size kana match as before. No index change and no reindex: the index already holds full-size kana.
• Regexps are normalized as whole strings. The classic parser passes the whole regexp to normalize (QueryParserBase.java:683; its own comment notes this "may not work with some analyzers"), so with the fix small kana inside regexp syntax are mapped too. Literal kana then match, but a character class that names a small kana changes meaning: /[ァ-ヶ].*/ becomes /[ア-ケ].*/ and finds 1 of the probe's two documents instead of 2. Every filter whose factory opts in gets the same text-level treatment.
• Cost: none at index time; normalize builds one more filter per call.

What the checks validated

• Scratch worktree at 645d4fe14a20, gradle, JDK 25: the test fails on stock with assertions on and off; with the fix it passes together with the four upstream sutegana filter and factory suites and the nightly TestFactories (6 suites, 17 tests), and again with assertions off (16 tests).
• The whole :lucene:analysis:kuromoji test task passes with assertions on and off: 31 suites, 196 tests, 0 failures, 7 skipped. :lucene:analysis:kuromoji:check -x test passes with the fix and the test.
• Earlier (hunt): 0 normalize mismatches over all Unicode scalar values with the fix; the independent reproducer's and the GPT reviewers' own tests pass with the same change.

State in both factories' javadocs that they are not applied by Analyzer.normalize.

Trade-offs

No code change, but the current behaviour stays and every user who opts in needs the workaround; it contradicts the Analyzer.normalize contract and the filters' own description as normalizers.

Method: non-formal (code reading plus executed probes; the finder's later bounded-exhaustive check is semi-formal).

• The hunt looked for Analyzer.normalize gaps across sibling factories, inventoried factories without a normalize() override, and witnessed this pair with a public-API differential probe on stock main 645d4fe14a20 (2026-10-30): the index holds キヤツト and ちよつと, normalize returns キャット and ちょっと, the term query gets 1 hit and the prefix and range queries 0, while the cjkWidth control gets 1 each. A separate re-run confirmed it.
• A hunt unit reduced it to one JUnit (later minimized to the one test above) and added a bounded-exhaustive check on the real jars: normalize against indexing for all 1,112,064 Unicode scalar values and all 71,289 two-character kana strings, plus a prefix property, stock and with the fix, with cjkWidth as the control.
• The finder also screened two siblings and did not claim them: PatternReplaceFilterFactory (already open upstream as #12613) and CapitalizationFilterFactory (position-dependent, by design).