StandardQueryParser with setMultiFields normalizes field-less wildcards with the wrong analyzer

The flexible StandardQueryParser runs a parsed query through a fixed list of processors. The first one, WildcardQueryNodeProcessor, normalizes the literal parts of prefix and wildcard terms with Analyzer#normalize(field, text); the second one, MultiFieldQueryNodeProcessor, gives a clause without a field the fields set by setMultiFields. So a field-less Luc* is normalized for field null, which a PerFieldAnalyzerWrapper answers with its default analyzer instead of the expanded field's. With a case-keeping default and a lower-cased title, Luc* becomes title:Luc* and misses the document titled "Lucene" that title:Luc* (parsed as title:luc*) finds; with the wrapper's javadoc setup (lower-casing default, keyword name fields) Jo* is lower-cased and misses "John"; with a Map.of per-field map every such term with a literal part throws NullPointerException from parse(). Fuzzy, regexp and range terms through the same expansion are normalized per field and plain terms analyzed per field, and the classic MultiFieldQueryParser normalizes wildcard terms per field. Introduced by LUCENE-7355 together with LUCENE-7429 (both 2016), first released in 7.0.0, still on main; moving MultiFieldQueryNodeProcessor before WildcardQueryNodeProcessor fixes it and passes the whole queryparser module.

• Multi-field mode: StandardQueryParser.setMultiFields(fields) plus parse(query, null). A clause without a field is expanded into one clause per field (StandardQueryParser.java:102-105, :357-362).
• Normalization: Analyzer#normalize(field, text) maps a partial word to "the representation that it would have in the index" (lower-casing, folding) without tokenizing; query parsers use it for wildcard and fuzzy terms (Analyzer.java:202-213). It depends on the field.
• Per-field analyzer: PerFieldAnalyzerWrapper (or any DelegatingAnalyzerWrapper) picks an analyzer by field name with fieldAnalyzers.get(fieldName), else the default (PerFieldAnalyzerWrapper.java:74-77).

1. The pipeline adds WildcardQueryNodeProcessor first and MultiFieldQueryNodeProcessor second (StandardQueryNodeProcessorPipeline.java:48-49); each processor walks the whole tree before the next one (QueryNodeProcessorPipeline.java:75-81).
2. WildcardQueryNodeProcessor calls analyzeWildcard(analyzer, fqn.getFieldAsString(), text) (WildcardQueryNodeProcessor.java:107-111), which calls a.normalize(field, chunk) on each literal chunk (:66, :76). For a field-less clause getFieldAsString() is still null (FieldQueryNode.java:99-102).
3. The wrapper is asked for field null (AnalyzerWrapper.java:146): PerFieldAnalyzerWrapper falls back to its default analyzer, and a Map.of map throws NullPointerException on get(null).
4. Only then does MultiFieldQueryNodeProcessor set the field and clone the node per field (MultiFieldQueryNodeProcessor.java:70-105). Nothing normalizes the text again; the builder makes new PrefixQuery(new Term(field, text), ...) (PrefixWildcardQueryNodeBuilder.java:48).
5. Fuzzy, regexp and range terms are normalized by later processors with the expanded field (FuzzyQueryNodeProcessor.java:59, RegexpQueryNodeProcessor.java:43, TermRangeQueryNodeProcessor.java:138-139), plain terms are analyzed by AnalyzerQueryNodeProcessor (pipeline :58). Only prefix and wildcard terms see null.

• Who is hit. Callers of StandardQueryParser with setMultiFields and parse(query, null) whose fields use analyzers that normalize differently, through PerFieldAnalyzerWrapper or another DelegatingAnalyzerWrapper. Typical setups: the wrapper's javadoc example (lower-casing default, keyword name or ID fields), or a keyword or whitespace default with analyzed text fields.
• Which queries. A field-less prefix or wildcard term is affected when the default analyzer and the field's analyzer normalize one of its literal chunks differently (whether hits change then depends on the indexed data): an upper-case letter in every hand-written repro and security case; in the finder's scan also accent and width folding (ASCII-folding and ICU-folding chains).
• Default vs opt-in. Multi-field mode and per-field analyzers are application choices; given those, no other setting is needed. With one field-independent analyzer nothing visible changes.
• Both directions. Missing normalization (Luc* on a lower-cased field: 0 hits) and wrong normalization (Jo* lower-cased on keyword fields: 0 hits; ABC* selects abc-2 instead of ABC-1).
• Crash variant. A per-field map that rejects null keys (Map.of, executed; by the JDK contract also ConcurrentHashMap, not executed) makes every such term with a literal part throw an unchecked NullPointerException from parse() instead of a QueryNodeException.
• Scale. None: one document, a one-word query. Silent: fewer or different hits.
• Reach. Solr, Elasticsearch and OpenSearch do not call setMultiFields (read-only code search of their default branches, not exhaustive); Lucene's Luke uses the parser without it. Applications that embed the flexible parser directly can be affected; they were not surveyed. No user report was found in the nine years since 7.0.0; the likelihood is rated plausible.
• Fix cost. Query side only: no reindexing.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<title:luc*> but was:<title:Luc*>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.queryparser.flexible.standard.TestMultiFieldWildcardNormalize.testPrefixUsesExpandedFieldAnalyzer(TestMultiFieldWildcardNormalize.java:35)

• The contract. StandardQueryParser: "If the field specification is omitted, the query parser will expand the clause over all fields specified by a call to setMultiFields(CharSequence[])" (:102-105); setMultiFields: "Set the fields a query should be expanded to when the field is null" (:357-362). Analyzer#normalize: "Normalize a string down to the representation that it would have in the index ... used by query parsers ... in case of a wildcard or fuzzy query" (Analyzer.java:202-213). PerFieldAnalyzerWrapper "can be used like any other analyzer, for both indexing and query parsing" (:42-43).
• The 7.0 migration note. "CommonQueryParserConfiguration.setLowerCaseExpandedTerms removed (LUCENE-7355). This option has been removed as expanded terms are now normalized through Analyzer#normalize" (lucene/MIGRATE.txt:94-97 at the 7.0.0 tag).
• The parser disagrees with itself. Fuzzy, regexp, range and plain clauses through the same expansion get per-field forms (title:lucene~1); the classic MultiFieldQueryParser expands prefix and wildcard terms per field before normalizing (MultiFieldQueryParser.java:251-274).
• Tests and history. No upstream test combines multi fields, per-field normalization and wildcards; the whole queryparser module (533 upstream tests) passes with the fix. LUCENE-7355 removed LowercaseExpandedTermsQueryNodeProcessor, which ran after the expansion and lower-cased without looking at the field, and put the new normalize() call into the processor that already ran first; its other calls run after the expansion. None of the issue's 34 comments mentions the multi-field path.

The strongest case for "intended" (a devil's-advocate verifier, reviews by OpenAI GPT-5.6 and a fact check by the same model family, all of which rated it weak): the pipeline javadoc says "The order they are assembled affects the results" (StandardQueryNodeProcessorPipeline.java:32); MultiFieldQueryNodeProcessor expands terms "so the query looks for the same term in different fields" (:31-33), which could mean one identical, already-normalized term; TestMultiFieldQPHelper.testAnalyzerReturningNull says bla*, bla~ and [a TO c] "are not affected as their terms are not analyzed anyway" (:308-315); the NullPointerException could be blamed on the map; and normalizing once before the expansion is cheaper than once per expanded field.

Why it fails: the order warning is generic; the same parser gives fuzzy, regexp, range and plain clones per-field forms, so "same term" means the same input text, analyzed per field; the test comment comes from LUCENE-1567 (2009), seven years before Analyzer#normalize, lumps in fuzzy and range terms that are now normalized per field, and passes in either order; with multi fields set no other clause type passes null; neither normalize nor getWrappedAnalyzer says whether null is allowed, so the API is silent, and the case rests on the disagreement with the explicit-field form; the classic MultiFieldQueryParser already normalizes once per field. Verdict: a defect. Before 7.0 this parser lower-cased expanded wildcard terms without looking at the field, so the regression is the null field (the field-less form disagrees with the explicit-field form, and Map.of maps throw), not the loss of a per-field feature.

The root cause is the order of two processors. The swap was built and run in a scratch worktree at ea57360912ed (removed afterwards) and with the standalone runner; the other options are not built.

Move add(new MultiFieldQueryNodeProcessor()) above add(new WildcardQueryNodeProcessor()). MultiFieldQueryNodeProcessor works on any FieldableNode (a FieldQueryNode is one), so it clones the raw Luc* per field, and WildcardQueryNodeProcessor then normalizes each clone for its own field. Results: Luc* gives title:luc*, Jo* gives firstname:Jo* lastname:Jo*, and the Map.of map no longer throws.

public StandardQueryNodeProcessorPipeline(QueryConfigHandler queryConfig) {
  super(queryConfig);

  add(new MultiFieldQueryNodeProcessor()); // was second: set the fields first
  add(new WildcardQueryNodeProcessor());   // was first: now normalizes per field
  add(new FuzzyQueryNodeProcessor());
  // ... the rest unchanged

Trade-offs

• Behaviour change for affected users (the point of the fix): field-less prefix and wildcard terms now match like the explicit-field form; queries that matched only through the default analyzer's normalization see different hits. Worth a CHANGES entry.
• Pipeline positions change. StandardQueryNodeProcessorPipeline is a public List; code that edits it by index (for example get(0) expecting WildcardQueryNodeProcessor) sees the two processors swapped.
• Cost. A field-less wildcard term is then normalized once per expanded field instead of once, as the classic MultiFieldQueryParser already does; not measured, and only noticeable with many fields or an expensive normalize().
• Failure order for invalid configurations. Without setMultiFields, parse(query, null) makes MultiFieldQueryNodeProcessor throw IllegalArgumentException (MultiFieldQueryNodeProcessor.java:79-82); after the swap it throws before the wildcard processor runs, so a Map.of map then gives that exception instead of the NullPointerException. With an empty multi-field list the field stays null in either order.
• Checked: nested AND/OR/NOT, clause boosts, field boosts (setFieldsBoost), a leading wildcard, quoted text, ranges and fuzzy-on-wildcard all agree with the explicit expansion after the swap (14 shapes x 2 configurations). Not checked: custom pipelines built on this class.

What the checks validated

• gradle, :lucene:queryparser:test with this doc's test and the scope probe: stock 538 tests, 4 failures (exactly those 4 tests), 1 skipped; with the fix 538 tests, 0 failures, 2 skipped (the upstream skip and the workaround test, which skips itself once the order is fixed), for seed DEADBEEF with assertions on and off and a random seed 36B0F102C8846336. That is the 533 upstream queryparser tests plus 5.
• :lucene:queryparser:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis main and test) passes with the fix and the test.
• Standalone runner with the fix shadowed: this test 3 of 3 (assertions on and off), the PerFieldAnalyzerWrapper variant 3 of 3, the scope probe 28 of 28 pairs agree (stock: 20 of 28 differ). Earlier, the finder's test, verifiers' tests, the security probe and the 7.0.0 history probe passed with the same swap.

Leave WildcardQueryNodeProcessor first but skip analyzeWildcard while the field is null and multi fields are set, then normalize in a processor that runs after MultiFieldQueryNodeProcessor.

Trade-offs

• Keeps pipeline positions (and the current failure order) stable; more code, and two places that handle wildcard text. If maintainers want to keep the public pipeline positions, this option deserves equal consideration.

Removes the NullPointerException with Map.of maps; the prefix and wildcard terms stay wrong.

Trade-offs

• It would also bypass a null key that a null-tolerant map holds on purpose (undocumented, but possible). Only useful as hardening next to the swap.

After creating the parser, move the wildcard processor behind the multi-field processor; check the order first, so the line does nothing on a fixed release (the probe makes the same check). Or write the fields explicitly, or use the classic MultiFieldQueryParser.

QueryNodeProcessorPipeline p = (QueryNodeProcessorPipeline) qp.getQueryNodeProcessor();
if (p.get(0) instanceof WildcardQueryNodeProcessor && p.get(1) instanceof MultiFieldQueryNodeProcessor) {
  p.add(1, p.remove(0)); // WildcardQueryNodeProcessor now runs after MultiFieldQueryNodeProcessor
}

What the checks validated

• On main, 10.5.2 and 9.12.3 the scope probe's 28 pairs all agree with the explicit-field form.

Method: semi-formal (a hypothesis from code reading, then witnessed by a bounded tool run).

• A hunt unit (analyzer normalize consistency, 2026-10-26). The finder, an AI tool, checked a metamorphic relation: a word's Analyzer#normalize form should equal the token the index holds, and each parser should build its prefix, wildcard, fuzzy, regexp and range terms from that form.
• While building the parser-reachability scan it read StandardQueryNodeProcessorPipeline and suspected the order. The bounded scan on the stock main jars (2,921,246 query checks: 18 analyzers, words of length 1-3, 4 parser setups, 5 query kinds) used a PerFieldAnalyzerWrapper with a KeywordAnalyzer default and the scanned analyzer on the expanded field.
• Result: in multi-field mode the flexible parser missed 38-92% of the prefix and wildcard checks for 15 of the 16 analyzer configurations in the comparison, counted over the words for which the analyzer's own normalize() form equals its indexed token (StandardAnalyzer 4,548 of 10,032, an ASCII-folding chain 7,772 of 10,032, an ICU-folding chain 11,167 of 12,198); the 16th, a Ukrainian chain, missed none. With an explicit default field there were no such misses, and fuzzy, regexp and range in multi-field mode had none either. The scan covers a finite set of analyzers and words.
• A public-API JUnit test then failed on main, 10.5.2 and 9.12.3; an independent reviewer from another model family (OpenAI GPT-5.6) reviewed it, verifiers and an adjudication confirmed it, and the history, prior-report and security checks followed.