CustomAnalyzer.normalize skips the filters inside when()/whenTerm() blocks and ProtectedTermFilterFactory, so multi-term queries miss documents

Query parsers build prefix, wildcard, fuzzy, regexp and range queries from Analyzer.normalize, which applies only the filters that opt in to normalization, such as lowercase and asciifolding. In a CustomAnalyzer, a conditional block (when(...)/whenTerm(...) ... endwhen(), or ProtectedTermFilterFactory with wrapped filters) is a single ConditionalTokenFilterFactory. It runs its inner filters at index time but has no normalize() of its own, so at query time it inherits the identity default and the inner filters are skipped. A document Apache Lucene indexed as apache, lucene is then missed by Apa*, Apach?, APACHE~1, /Apa.*/ and [Apache TO Apache] (0 hits each, 1 without the condition), with no error. It has been there since conditional filters were added in 7.4.0, in every release through 10.6.0 and on main; a normalize() override that wraps the inner factories' normalize() under the same condition fixes the always-true / full-term case.

1. CustomAnalyzer.normalize calls normalize() on each of its top-level token filter factories (CustomAnalyzer.java:170-177 on main 645d4fe14a20).
2. endwhen() adds the whole conditional block to the chain as one top-level factory (CustomAnalyzer.java:689-694); whenTerm(...) builds an anonymous ConditionalTokenFilterFactory (:564-581), and ProtectedTermFilterFactory extends that class.
3. ConditionalTokenFilterFactory.create() chains the inner factories' create() inside a ConditionalTokenFilter (ConditionalTokenFilterFactory.java:61-74), and inform() is forwarded too (:76-85). There is no normalize() override, so TokenFilterFactory.normalize runs: "While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate to create method." (TokenFilterFactory.java:101-108).
4. The inner LowerCaseFilterFactory.normalize, which delegates to create() (LowerCaseFilterFactory.java:61-63), is never called. The block contributes no normalization: in this chain (whitespace tokenizer, then the block) normalize returns the query text unchanged, while indexing applied the inner filters.

Callers that reach it: the classic QueryParserBase for range bounds (:550, :556), wildcard chunks (:646, :656), the whole regexp pattern (:683), prefix (:712) and fuzzy (:729), inherited by ComplexPhraseQueryParser through QueryParser; the flexible parser's range, fuzzy, wildcard and regexp processors; SimpleQueryParser for prefix and fuzzy, the only two of these syntaxes it has. Executed in the fact check: on stock the complex-phrase and flexible parsers miss all five multi-term queries and SimpleQueryParser misses Apa* and APACHE~1; all hit with the fix.

Document Apache Lucene, classic QueryParser, stock main, hit counts:

With asciifolding inside the block the same happens to accents: the index holds elan, normalize("Élan") returns Élan, and Éla*, Él?n and [Élan TO Élan] get 0 hits (1 each without the condition).

Not affected: plain term and phrase queries (they use the full analysis chain), query text already typed in the indexed form (apa*, apach? and [apache TO apache] get 1 hit on stock), and every stock analyzer (none builds a conditional block). What misses is query text the skipped filter would have changed, here a capital letter.

• Who is hit. A CustomAnalyzer (or a chain built from the factories by SPI name) with a normalizing filter such as lowercase, asciifolding or germanNormalization inside a conditional block, used with the classic, complex-phrase, flexible or simple query parser for range, fuzzy, prefix, wildcard or regexp terms (each parser for the syntax it has; SimpleQueryParser has only prefix and fuzzy). 28 filter factories in lucene/analysis override normalize(), and any of them loses it inside a block.
• Opt-in, but documented. No stock analyzer and no default builds a conditional block. But the ProtectedTermFilterFactory javadoc example wraps truncate and lowercase in one (its Solr example uses wrappedFilters="truncate,lowercase"), and on #14416 a committer (rmuir) told a user to use ProtectedTermFilter around a normalizer in a CustomAnalyzer.
• Silent. No exception or warning; queries just match fewer documents. With ProtectedTermFilterFactory the protected words keep working, while multi-term queries on other words miss whenever the query text differs from the indexed form (here, any capital letter). That is easy to overlook when tests type queries in lower case.
• No scale or odd parameter. One document, one token, deterministic, with assertions on and off.
• Workaround. Put the normalizing filter outside the block, as the CustomAnalyzer class javadoc example does (executed in the fact check: normalize("Apache") = apache, all six queries hit). That changes what the block protects, so it is not always acceptable. CustomAnalyzer is final, so normalize cannot be overridden on it, but an AnalyzerWrapper can add the missing normalization in wrapTokenStreamForNormalization (not run).
• Downstream (not run). By code reading of apache/solr main 9ba1a7d42e4b (2026-10-30), Solr's TokenizerChain.normalize also calls each factory's normalize() (TokenizerChain.java:140-143), so a Solr field type built this way most likely behaves the same.

The condition is always true, so the block must behave exactly like a plain lowercase filter. The query consequence (0 hits for Apa*, [Apache TO Apache] and the others, table above) is not part of the test. What the test does not settle: it covers only an always-true condition with lowercase, not what an arbitrary or attribute-reading condition, a protected-term list, or prefix, wildcard and regexp text should give (the policy question under Suggested fix).

The module's check task rejects the file as is: ecjLintTest reports "The value of the lambda parameter t is not used" and checkGoogleJavaFormat flags the comment line (over 100 columns). The PR copy changes only those two lines (whenTerm(t -> t.length() > 0) // holds for every token and a shorter comment); it fails the same way on stock, passes with the fix, and :lucene:analysis:common:check -x test passes.

Runs of this test: gradle on main 645d4fe14a20 (-Ptests.asserts=true and false: 1 test, 1 failed); the standalone runner on main (seed 5EED0471) and on the 10.6.0 release jars, -ea and -da: fails each time; with the fix: passes. An earlier two-assertion form (it also indexes a document and runs [Lucene TO Lucene]) fails the same way on main, 10.6.0, 10.5.2, 10.0.0 and 9.12.3. Name tests.asserts explicitly: the build randomizes it per seed otherwise.

TestConditionalFilterNormalize > testNormalizeAppliesFiltersInsideWhenTerm FAILED
    org.junit.ComparisonFailure: expected:<[l]ucene> but was:<[L]ucene>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:5C3E91A926CACBC]:0)
        at app/junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:117)
        at app/junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
        at app//org.apache.lucene.analysis.custom.TestConditionalFilterNormalize.testNormalizeAppliesFiltersInsideWhenTerm(TestConditionalFilterNormalize.java:33)

1 test completed, 1 failed

• The contract. Analyzer.normalize: "Normalize a string down to the representation that it would have in the index. This is typically used by query parsers in order to generate a query on a given term, without tokenizing or stemming, which are undesirable if the string to analyze is a partial word (eg. in case of a wildcard or fuzzy query)." (Analyzer.java:202-207). With an always-true condition the block must behave like the plain filter, and it does not.
• The inner filters opted in. TokenFilterFactory.normalize: "filters that should be applied at normalization time can delegate to create" (TokenFilterFactory.java:101-104); LowerCaseFilterFactory does. The conditional factory forwards create() and inform() to its inner factories (ConditionalTokenFilterFactory.java:61-85), not normalize().
• The documentation invites the configuration. The ProtectedTermFilterFactory javadoc puts lowercase inside the block. On #14416 rmuir wrote "Just use ProtectedTermFilter which works with any tokenfilter without requiring modification to its code?" and "If you want to change the behavior of GermanAnalyzer then just make a CustomAnalyzer with ProtectedTermsFilter.", about wrapping GermanNormalizationFilter, whose factory overrides normalize().
• Nothing documents the exclusion. No javadoc, CHANGES.txt or MIGRATE.md entry, test or code comment says conditional blocks are skipped; TestCustomAnalyzer's conditional tests check only index-time output, and its normalization tests use unconditional chains. Neither the feature's review (#9320) nor the normalize() redesign (#9543) discusses it.
• How the gap arose. LUCENE-8273 added the factory in 7.4.0, when CustomAnalyzer.normalize applied only MultiTermAwareComponent factories, and this one was not. LUCENE-8497 (8.0.0) replaced that with TokenFilterFactory.normalize() and converted the classes that were already multi-term aware, so this one stayed out.
• Precedents fixed as bugs (under "Bug Fixes" in lucene/CHANGES.txt): LUCENE-7606 "Normalization with CustomAnalyzer would only apply the last token filter" (6.4.0), LUCENE-8186 "LowerCaseTokenizerFactory now lowercases text in multi-term queries" (7.4.0), LUCENE-7429 "AnalyzerWrapper can now modify the normalization chain too and DelegatingAnalyzerWrapper does the right thing automatically" (6.3.0).

The strongest case for "intended" (the intended-behaviour checker and the GPT-5.6 intended-lens reviewer): normalization is opt-in and the default is identity, and TestCustomAnalyzer.testNormalization asserts that components that are not multi-term aware "should not be applied"; a condition cannot be judged faithfully on a partial word (Luc* when Lucene is protected), and a custom condition may read attributes (type, keyword) that the one-token normalize stream does not set, so never applying the block could be safer; and whenTerm(len > 2){lowercase} looks contrived.

Why it fails: the block is a container whose inner filters did opt in, and it drops that silently, like the DelegatingAnalyzerWrapper case fixed in LUCENE-7429. Nothing records a decision to exclude blocks, and the stock behaviour is no coherent policy: it skips the block for full terms too (range bounds, fuzzy terms), even under an always-true condition, where the condition would judge the whole query word just as it judges that word at index time. The fragment question is about how to fix partial words, not a defence of today's behaviour. The realistic trigger is a ProtectedTermFilterFactory wrapping lowercase, as its javadoc example does (there after truncate). Verdict: a defect.

One fix was built and validated; two alternatives are described. Workaround until a fix ships: put normalizing filters outside the conditional block (.addTokenFilter("lowercase").when(...)...endwhen()), as the CustomAnalyzer class javadoc example does (executed in the fact check: all six queries hit), or wrap the analyzer in an AnalyzerWrapper that adds the missing normalization in wrapTokenStreamForNormalization (not run).

Add normalize(TokenStream) to ConditionalTokenFilterFactory: the same body as create(TokenStream), calling each inner factory's normalize() instead of create(), wrapped by the subclass's create(input, inner), so the condition decides as at index time. +19 lines with a javadoc comment; the finder's sketch has the same logic. Offer it as a tested sketch for the case the test covers (an always-true condition, full terms), not as the settled policy for every condition and every partial word: maintainers may prefer a narrower opt-in or an explicit fragment policy to merging it unchanged.

@Override
public TokenStream normalize(TokenStream input) {
  if (innerFilters == null || innerFilters.size() == 0) {
    return input;
  }
  Function<TokenStream, TokenStream> innerStream =
      ts -> {
        for (TokenFilterFactory factory : innerFilters) {
          ts = factory.normalize(ts);
        }
        return ts;
      };
  return create(input, innerStream);
}

Trade-offs

• What the condition sees. The condition now runs on the text normalize gets: the user's query spelling, not the document's. For a full term (a range bound, a fuzzy term) it judges the whole query word just as it would judge that word at index time, so the result is what indexing that spelling produces (the normalize contract). Under an always-true condition (the test), or any condition that does not depend on the spelling, that is the document's indexed term. A spelling-sensitive condition can take a different branch: under whenTerm(t -> "Apache".contentEquals(t)), with the fix normalize("APACHE") stays APACHE, so APACHE~1 gets 0 hits while the indexed Apache was lower-cased, just as the term query APACHE gets 0 (executed).
• Fragments. Prefix, wildcard and regexp text is not a whole word, and the three differ: a prefix query normalizes the stem before * (Luc of Luc*), a wildcard query each literal chunk between wildcards separately (Él and n of Él?n), a regexp query the whole pattern, operators included (Apa.* of /Apa.*/). With ProtectedTermFilterFactory protecting Lucene, stock Luc* matches the unlowercased protected term and the fixed version (which lower-cases the unprotected stem Luc) does not, while unprotected prefixes such as Apa* start to match. Under whenTerm(len > 2){lowercase, asciifolding} the fix makes Éla* and [Élan TO Élan] hit, but Él?n still gets 0 hits, because the chunk Él fails the length condition.
• Only normalize() is forwarded, not create(). An inner filter without a normalize() override, such as truncate, is still skipped, as it is outside a block. For the exact ProtectedTermFilterFactory javadoc example (truncate with prefixLength 4, then lowercase), the fix restores Apa* and /Apa.*/ (0 to 1 hit), but [Apache TO Apache], APACHE~1 and Apach? stay at 0, as they do for the same two filters without the block.
• Attributes. A custom condition that reads tokenizer-set attributes (type, keyword) sees default values in the one-token normalize stream; the two stock conditions read only the term text. Worth a javadoc sentence.
• Cost. None at index time; normalize builds one more wrapper per call.

What the checks validated

• Scratch worktree at 645d4fe14a20, gradle, JDK 25: the test fails on stock with assertions on and off; with the fix it passes together with TestCustomAnalyzer, TestConditionalTokenFilter, TestProtectedTermFilterFactory and TestProtectedTermFilter (5 suites, 45 tests), assertions on and off.
• The whole :lucene:analysis:common test task: 340 suites, 1910 tests, 0 failures, 18 skipped, assertions on and off; :lucene:analysis:common:check -x test passes with the PR copy of the test.
• Earlier: the independent reproduction's own test and the GPT-5.6 reviewer's tests pass with the same logic; with it the hit table's conditional row is all 1s.
• The diff applies cleanly to the 10.6.0 source (patch --dry-run); only main was built and tested.

Ignore the condition at normalization time and chain the inner factories' normalize() directly.

Trade-offs

Simpler, and fragments then behave like the plain filter. But it breaks full protected terms: with Lucene protected (indexed as Lucene), [Lucene TO Lucene] normalizes to lucene and misses, where stock finds it today. Executed in the fact check: for both protected-term forms [Lucene TO Lucene] and Luc* drop from 1 hit to 0, while every Apache query hits.

State in the when()/whenTerm() and ProtectedTermFilterFactory javadocs that filters inside a conditional block are not applied by Analyzer.normalize, and change the ProtectedTermFilterFactory example.

Trade-offs

No code change, and the silent misses stay for every existing configuration.

Method: non-formal (code reading plus executed probes; the finder tagged it semi-formal, and the first signal decides).

• The hunt's round-18 planners took a family-siblings lens on Analyzer.normalize delegation gaps, one level above other normalize rows (analyzers whose normalize omits a filter), and read that ConditionalTokenFilterFactory has no normalize().
• A public-API probe on stock main 645d4fe14a20 (2026-10-30) witnessed it: inside whenTerm(len > 2){lowercase, asciifolding} the index holds elan, normalize("Élan") returns Élan, and Éla*, Él?n and [Élan TO Élan] get 0 hits while the plain chain gets 1 each; ProtectedTermFilterFactory behaves the same. A skeptic re-ran it.
• A hunt unit built the differential (term, range, fuzzy, regexp, prefix and wildcard queries, with and without the wrapper, stock and with a fix), wrote a public-API JUnit, ran it on releases, and the no-noise gate reduced it to the one-assertion test above.