SmartChineseAnalyzer.normalize skips the tokenizer's full-width folding, so full-width prefix, wildcard, fuzzy and range queries can miss

SmartChineseAnalyzer does its case and width handling inside the tokenizer: HMMChineseTokenizer's SegTokenFilter turns full-width Latin letters and digits into ASCII and lower-cases A-Z (SegTokenFilter.java:39-62), and createComponents has no LowerCaseFilter (SmartChineseAnalyzer.java:124-134). So 我们使用Ｌｕｃｅｎｅ is indexed as 我们, 使用, lucen. SmartChineseAnalyzer.normalize(), which the classic and flexible query parsers use for prefix terms, wildcard chunks, fuzzy terms, regexps and range bounds (the simple parser for its prefix and fuzzy terms), returns only new LowerCaseFilter(in) (:137-140), although Analyzer#normalize is documented to give "the representation that it would have in the index", without tokenizing or stemming (Analyzer.java:203-208). The result: Ｌｕｃ*, Ｌｕｃｅｎ~1 and [Ｌｕａ TO Ｌｕｚ] find nothing while Luc* and the term query Ｌｕｃｅｎｅ find the document, and normalize("ＡＢＣ") returns ａｂｃ while the index holds abc. A secondary, narrower direction: normalize lower-cases non-ASCII capitals such as É that the index keeps. Parsed term and phrase queries and the indexed data are unaffected. Every release from 6.2.0 to 10.6.0 (10.6.0 by reading) and main has it. The fix makes normalize() apply the tokenizer's folding; it changes only the query side, so no reindex is needed.

• Full-width forms: the letters Ａ-Ｚ, ａ-ｚ (U+FF21-U+FF3A, U+FF41-U+FF5A) and digits ０-９ (U+FF10-U+FF19), each as wide as a Chinese character. Chinese input methods produce them in full-width mode, giving mixed text such as 我们使用Ｌｕｃｅｎｅ or ２０２６年.
• SegTokenFilter: part of HMMChineseTokenizer, so it runs inside the tokenizer, not as a token filter. For full-width tokens it subtracts 0xFEE0 from each character (full-width to ASCII) and then lower-cases A-Z; for ASCII tokens it lower-cases A-Z (SegTokenFilter.java:39-62; javadoc "Full-width latin will be converted to half-width, then all latin will be lowercased"). createComponents adds no LowerCaseFilter: "LowerCaseFilter is not needed, as SegTokenFilter lowercases Basic Latin text" (SmartChineseAnalyzer.java:128).
• Analyzer.normalize(field, text): what the query parsers call for a partial word, because the full analyzer would tokenize or stem it. It runs the analyzer's normalize(String, TokenStream) override, which should hold the normalizing steps and leave out the tokenizer and the stemmer (Analyzer.java:53-58, :203-212).
• Multi-term queries: prefix (Ｌｕｃ*), wildcard (Ｌｕ?ｅｎ), fuzzy (Ｌｕｃｅｎ~1), regexp (/ｌｕｃ.*/) and range ([Ｌｕａ TO Ｌｕｚ]). Parsed term and phrase queries run the full analyzer and are not affected.

1. SmartChineseAnalyzer.normalize() is return new LowerCaseFilter(in); (SmartChineseAnalyzer.java:137-140). Because the folding lives in the tokenizer, which normalize skips, it is lost.
2. The classic parser passes the prefix through it (QueryParserBase.java:712, BytesRef term = getAnalyzer().normalize(field, termStr);), and likewise the fuzzy term (:729), the regexp (:683), each wildcard chunk (:646, :656) and the range bounds (:550, :556; an open * bound is skipped, :547-557). The flexible parser's processors do the same (range bounds only when they do not parse as dates, TermRangeQueryNodeProcessor.java:100-139). The simple parser has only prefix and fuzzy syntax and normalizes both (SimpleQueryParser.java:573, :605).
3. Ｌｕｃ* therefore becomes body:ｌｕｃ*. The index holds lucen, so the prefix cannot match any term. LowerCaseFilter also applies Character.toLowerCase to every code point, while the index lower-cases only A-Z, so normalize("É") is é but the index keeps É. This secondary direction matters for single-letter terms, because the tokenizer emits such letters one per token, and for range bounds: [Аz TO Яz] (Cyrillic capitals) built from normalize admits б instead of the indexed Б.
4. The sibling analyzers fold width in normalization: CJKAnalyzer.normalize is CJKWidthFilter + LowerCaseFilter (CJKAnalyzer.java:101-106), and JapaneseAnalyzer applies CJKWidthCharFilter in initReaderForNormalization (JapaneseAnalyzer.java:130-133).

One document 我们使用Ｌｕｃｅｎｅ, default SmartChineseAnalyzer, classic QueryParser: Ｌｕｃ* and Ｌｕｃｅｎ~1 find 0 documents; Luc*, Lucen~1 and the term query Ｌｕｃｅｎｅ find 1. normalize("ＡＢＣ") = ａｂｃ (index abc), normalize("１２３") = １２３ (index 123). The hunt's scan found 76 of 285 single-token words that disagree; every one of the 73 that the parser scan checked (the other 3 were not in its word set) was missed by classic and flexible prefix, wildcard, regexp and range queries and by simple-parser prefix queries, while fuzzy still hit 61 within the edit distance. An exhaustive scan of every BMP character indexed as one token found 1,141 of 61,527 disagreeing (the 62 full-width letters and digits and 1,079 non-ASCII capitals) and all 2,704 two-letter full-width words.

• Who is exposed. Applications that use the stock SmartChineseAnalyzer, the default constructor included, and pass user text through QueryParser, StandardQueryParser or SimpleQueryParser (prefix and fuzzy only), or call Analyzer.normalize themselves. Analyzer wrappers delegate normalize to the wrapped analyzer (AnalyzerWrapper.java:123-127), so wrapped uses inherit it unless the wrapper adds its own normalization step (as the workaround below does). The class targets "Chinese or mixed Chinese-English text" (SmartChineseAnalyzer.java:33).
• Default, not opt-in. Default analyzer and default parser settings. The only choices involved are the smartcn analyzer (an optional module, marked @lucene.experimental) and multi-term syntax.
• What it takes. Full-width letters or digits in the query term next to an ASCII *, ?, ~, / or range syntax: mixed-width input, or an application that appends * or builds a range around user text. By the classic grammar, a full-width ＊ is an ordinary character, so Ｌｕｃ＊ typed entirely in full-width mode becomes the plain term query body:luc, not a prefix query, and does not match lucen (QueryParser.jj:183-187, :208-211; executed). Documents written in full width are still found by half-width queries that fit the stemmed term: Luc* finds both widths, Lucene* neither, because the index holds lucen. The registry rates the likelihood rare; no report was found in the searched public sources.
• What the damage looks like. Prefix terms, and wildcard or regexp literals, that require a full-width letter or digit cannot match any indexed term: 0 hits, no error, whichever width the document used. Range bounds compare in the wrong form and can drop or add terms ([a TO ｚ] admitted 7 of 8 probe documents; the indexed-form [a TO z] admits 3). Fuzzy terms can still hit within the edit distance, so say "miss", not "never match".
• Scale. One document and one query are enough. All 62 full-width letters and digits and all 2,704 two-letter full-width words normalize differently from how they are indexed.
• Downstream (code reading only, nothing run). Elasticsearch's analysis-smartcn plugin registers the stock analyzer, and its query_string parser normalizes range, fuzzy, prefix and wildcard terms with the field's analyzer, so it would inherit the miss; another downstream server's smartcn plugin also registers the stock class.
• Limit. After the fix, prefixes that run past the Porter stem stay best effort, as with every stemming analyzer (Ｌｕ?ｅｎｅ misses even with the fix, because Lucene is indexed as lucen). This bug is only the width and case mismatch.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[abc]> but was:<[ａｂｃ]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.cn.smart.TestSmartChineseNormalizeFullWidth.testNormalizeMatchesIndexedTerm(TestSmartChineseNormalizeFullWidth.java:28)

• The contract. Analyzer.java:203-208: "Normalize a string down to the representation that it would have in the index. This is typically used by query parsers in order to generate a query on a given term, without tokenizing or stemming". The class example (:53-58) keeps a filter that is "about normalization" and drops one that is "about stemming".
• The analyzer documents its folding as index behaviour. SegTokenFilter's javadoc ("Full-width latin will be converted to half-width") and the SmartChineseAnalyzer.java:128 comment; TestSmartChineseAnalyzer.testFullWidth ("Full width alphas and numerics are folded to half-width", :160-170) asserts it.
• Design intent. On LUCENE-7355 (#8409), jpountz, who added normalize, described it as applying "all normalization filters (lowercasing, ascii folding, etc.) but not transformations (stop word removal, stemming, etc.)".
• Siblings. The same commit gave CJKAnalyzer.normalize CJKWidthFilter + LowerCaseFilter and JapaneseAnalyzer width folding (today through initReaderForNormalization). SmartChineseAnalyzer, whose folding is hidden in the tokenizer, got only LowerCaseFilter.
• Tests. No smartcn test calls normalize; BaseTokenStreamTestCase only checks that it does not throw (:1478). Nothing pins the current output.
• Precedents fixed as bugs. LUCENE-8186 (#9234, fixed in 7.4 and 8.0): CustomAnalyzer with a lower-casing tokenizer did not normalize multi-term queries, fixed so that normalize applies the tokenizer's normalization; LUCENE-7606 (#8657, fixed in 6.4 and 7.0): CustomAnalyzer.normalize applied only the last token filter. Separately, in LUCENE-8022 (#9070, closed as Invalid) a maintainer told a user that their analyzer must override normalize() to mirror its index-time lower-casing.

The strongest "intended" case (the cross-model reviews rated it moderate; the independent reproducer rated it weak and the devil's-advocate verifier rejected it): SmartChineseAnalyzer is @lucene.experimental; normalize deliberately skips tokenization, and here the folding lives in the tokenizer; the classic parser's javadoc says "normal Analyzers are unlikely to work properly with wildcard templates" (QueryParserBase.java:611, :696; :667 says the same of "regular expression templates"), and a comment in its wildcard code calls keeping wildcard and escaped characters away from normalize a "best effort" (:638); and LowerCaseFilter keeps roughly what the parsers did before 6.2 (lowercaseExpandedTerms).

Why it fails: "experimental" marks API stability, not licence for query and index to disagree; the javadoc excludes tokenizing and stemming, not character folding, and LUCENE-8186 made normalize mirror a tokenizer's normalization; the wildcard javadoc predates LUCENE-7355 and is about splitting templates, while the prefix code a few lines below calls normalize (:712); back-compat with the old lower-casing makes the behaviour long-standing, not intended, since LUCENE-7355 aimed beyond lower-casing and added width folding wherever the index chain visibly folds width. No test, discussion or decision supporting the current output was found. Verdict: a defect. What the counter-case gets right: a fix changes long-standing query behaviour; the recommended fix copies the tokenizer's private folding rules into a second place; full-width text next to ASCII operators is unusual input; and the old parser lower-casing (String.toLowerCase(locale)) is close to, not identical with, LowerCaseFilter (per-code-point Character.toLowerCase; they differ for locale-sensitive and contextual mappings such as the Turkish dotless i and the Greek final sigma). These argue for a careful, tested change, not for keeping the mismatch. For the secondary É direction one could argue the index side deviates (SegTokenFilter's javadoc says "all latin will be lowercased" while the code lower-cases only A-Z), so present that direction as a question.

Two fixes were built and checked in a scratch worktree of main ea57360912ed; A mirrors the index exactly and is recommended. The other options are a larger refactoring and a workaround.

Replace new LowerCaseFilter(in) with a private filter that does what SegTokenFilter does at index time, restricted to the three ranges that Utility.getCharType puts in the full-width classes (U+FF10-U+FF19, U+FF21-U+FF3A, U+FF41-U+FF5A). The LowerCaseFilter import goes; TokenFilter and CharTermAttribute are imported (+36/-2, formatted by tidy). Same logic as the finder's fix sketch. A regression test: the repro's class, or its assertions added to TestSmartChineseAnalyzer next to testFullWidth.

@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
  return new SegTokenNormalizationFilter(in);
}

/**
 * Applies the case and width folding that SegTokenFilter applies at index time: full-width Latin
 * letters and digits become half-width, then only Basic Latin A-Z is lower-cased.
 */
private static final class SegTokenNormalizationFilter extends TokenFilter {
  private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);

  SegTokenNormalizationFilter(TokenStream in) {
    super(in);
  }

  @Override
  public boolean incrementToken() throws IOException {
    if (input.incrementToken() == false) {
      return false;
    }
    final char[] buffer = termAtt.buffer();
    for (int i = 0; i < termAtt.length(); i++) {
      char c = buffer[i];
      if ((c >= 0xFF10 && c <= 0xFF19)
          || (c >= 0xFF21 && c <= 0xFF3A)
          || (c >= 0xFF41 && c <= 0xFF5A)) {
        c -= 0xFEE0;
      }
      if (c >= 'A' && c <= 'Z') {
        c += 0x20;
      }
      buffer[i] = c;
    }
    return true;
  }
}

Trade-offs

• Query side only: the index already holds the folded form, so no reindex.
• Behaviour change: the tested full-width multi-term queries now hit (the intent); a full-width ＊ or ． stays a literal character (Ｌｕ＊? becomes lu＊?, 0 hits; executed). Non-ASCII capitals are no longer lower-cased in normalize, so É* matches the indexed É instead of é; consistent with the index, but worth raising as a question with maintainers.
• Cost: one character loop per normalized query term.
• Maintenance: the filter copies the tokenizer's private folding rules (Utility.getCharType, SegTokenFilter), so the two can drift apart again unless a test pins them together.
• The override has not changed since 2016, so the same patch should apply to maintained branches; only main was built and tested.
• Prefixes that run past the Porter stem stay best effort, as with every stemming analyzer.

What the checks validated

• Scratch worktree at ea57360912ed, gradle, JDK 25: the new test fails 2 of 2 on stock and passes with the fix; :lucene:analysis:smartcn:check passes (checkGoogleJavaFormat, ecjLint, forbiddenApis, and 3 suites / 23 tests: TestSmartChineseAnalyzer 19, TestHMMChineseTokenizerFactory 2, the new 2); the module tests pass with assertions off (seed 77AA0565), and the new test 20 iterations (40 tests).
• Standalone runner with the fix shadowed: the new test 2 of 2 (assertions on and off), the QueryParser test 3 of 3, the independent reproducer's 5 of 5, the security probe 11 of 11; the exhaustive scan finds 0 of 61,527 single-character tokens and 0 of 2,704 two-letter full-width words disagreeing, and a scan of U+FF00-U+FFEF 0 of 202 (not scanned: supplementary characters and mixed, escaped, wildcard or regexp inputs). The finder's equivalent sketch passed the upstream TestSmartChineseAnalyzer (19 of 19) earlier.
• These standalone runs shadowed the fix before tidy re-wrapped its if condition (same logic). The fact check re-ran the new test (assertions on and off), the QueryParser test and the security probe with the final source (brief/fix/src-a); all pass.

TokenStream result = new CJKWidthFilter(in); result = new LowerCaseFilter(result); return result; (+4/-1). The repro passes and :lucene:analysis:smartcn:check passes (23 tests).

TokenStream result = new CJKWidthFilter(in);
result = new LowerCaseFilter(result);
return result;

Trade-offs

Smaller and reuses an existing filter, but it does not mirror the index: with it, the exhaustive scan still finds 1,138 of 61,527 single-character tokens disagreeing. That is the 1,079 non-ASCII capitals as before, plus 59 new ones in U+FF65-U+FF9F, which the index keeps half-width: CJKWidthFilter maps the half-width katakana and middle dot to their full-width forms, and the two half-width sound marks U+FF9E and U+FF9F to the combining marks U+3099 and U+309A, or merges them into a preceding katakana (CJKWidthFilter.java:67-76). It also folds full-width punctuation (U+FF01-U+FF5E), so a full-width ＊ or ． inside a wildcard or regexp term becomes an ASCII operator: Ｌｕ＊? becomes lu*? and /Ｌｕｃ．＊/ becomes /luc.*/, and both match lucen (executed). The independent reproducer's É test still fails.

The direction of an unmerged 2009 patch on LUCENE-2023 (#3098), written for speed: split SegTokenFilter's lower-casing and full-width conversion into filters, which normalize could then reuse.

Trade-offs

By inference (not run): it could change indexed terms for some inputs and would then need a back-compat story (reindex), so it is a larger change than the query-side fix.

SmartChineseAnalyzer is final, so wrap it: an AnalyzerWrapper whose getWrappedAnalyzer returns the stock analyzer and whose wrapTokenStreamForNormalization returns new CJKWidthFilter(in) (it runs after the stock LowerCaseFilter). Indexing is unchanged. On the stock main jars, with assertions on and off, Ｌｕｃ*, Ｌｕｃｅｎ~1 and [Ｌｕａ TO Ｌｕｚ] then find the document. Or type multi-term queries in half-width (Luc*).

Trade-offs

No reindex. It carries the CJKWidthFilter caveats of the previous option (half-width katakana; full-width ＊ and ． become operators) and leaves the É direction as it is, because the stock LowerCaseFilter runs first (É* becomes é*, 0 hits; executed).

Method: a bounded-exhaustive metamorphic scan on the stock main jars (the study's "semi-formal" tier). A hunt unit, "analyzer normalize consistency", 2026-10-26.

• Relation N1: Analyzer.normalize(f, w) equals the single full-span token that indexing produces for w, checked over bounded word sets for 99 analyzer configurations (every public stock Analyzer with a no-arg constructor in the module jars, 5 hand-built ones, and stemmer-free variants of 38 language analyzers). SmartChineseAnalyzer disagreed on 76 of 285 single-token words: full-width Latin letters and digits, and non-ASCII capitals such as É, Ø, İ, ẞ.
• Relation N2 (parser reachability): each disagreeing word, used as a prefix, wildcard, fuzzy, regexp and range query through the classic, flexible and simple parsers, must still find its document. All 73 checked disagreeing words were missed by classic and flexible prefix, wildcard, regexp and range queries and by simple-parser prefix queries; fuzzy still hit 61. Every agreeing word was found. The flexible parser's multi-field mode, which has a separate bug of its own, differs: there prefix and wildcard hit 19 of the 73 and missed 79 agreeing words.
• Code reading then located the cause: the folding lives in the tokenizer's SegTokenFilter, which normalize skips.
• An AI finder wrote the public-API JUnit; independent AI tools then reproduced and judged it, and an exhaustive scan over every BMP character confirmed the scope (see the checks).