MultiFieldQueryParser under AND pairs up different fields' clauses by position number

MultiFieldQueryParser analyzes the query text once per field, takes clause i of each field's analyzed BooleanQuery, ORs those across the fields into group i, and under AND_OPERATOR requires every group. That is only right when clause i covers the same query word in every field. With fields that split text differently (title on whitespace, body on non-letters), wi-fi router parses to +(title:wi-fi body:wi) +(title:router body:fi) +(body:router): it misses the document titled "wi-fi router", which QueryParser("title") finds, and matches a document with title "router" and body "wi router". That is in every release from 7.0.0 under default settings (6.2.0 with setSplitOnWhitespace(false)). Since #16446 (10.6.0, main and branch_10_6) the alternatives of a token graph are paired the same way, so with autoGeneratePhraseQueries a multi-word synonym's alternatives are all required and even a single field disagrees with QueryParser; #16446 also widened the per-field case to single words. No error is raised.

• Per-field analysis: an analyzer that treats fields differently, e.g. PerFieldAnalyzerWrapper or any DelegatingAnalyzerWrapper. Here title splits on whitespace (wi-fi stays one token) and body splits on every non-letter (wi, fi).
• Token graph: analysis that offers alternatives for the same words, e.g. a multi-word synonym ("guinea pig" = cavy) or WordDelimiterGraphFilter with catenation ("wi fi" or wifi). With autoGeneratePhraseQueries, QueryBuilder.analyzeGraphPhrase turns it into one phrase per path, ORed.

1. With the default splitOnWhitespace=false, the grammar joins wi-fi router into one getFieldQuery(null, "wi-fi router", false) call (QueryParser.jj:376-409).
2. MultiFieldQueryParser.getFieldQuery(String,String,boolean) (MultiFieldQueryParser.java:142-209) builds each field's query: title +title:wi-fi +title:router (2 clauses), body +body:wi +body:fi +body:router (3 clauses).
3. It takes clause termNum of each field's query, dropping its occur (:160-172), and ORs them into one group per number (:185-192): (title:wi-fi body:wi), (title:router body:fi), (body:router).
4. getMultiTermPositionQuery (:220-234, added by #16446) adds every group with MUST under AND (:227-228). The code comment calls the groups "one group per analyzed TERM POSITION" (:199-204).
5. For a token graph the clauses are alternatives, not positions: analyzeGraphPhrase adds one SHOULD phrase per path (QueryBuilder.java:565-583), so ("guinea pig" OR cavy) becomes +(body:"guinea pig") +(body:cavy).

Before #16446 the groups of a one-word call were combined with SHOULD: correct for graphs, an over-match for per-field wi-fi (the known #16443). Multi-word per-field text was already required group by group since LUCENE-2605 (2016).

For these inputs the flexible parser builds the by-word expansion that the classic javadoc's example suggests: each query word ORed over the fields, whatever each field's analysis makes of it.

• Who is hit. Callers of the classic MultiFieldQueryParser with AND_OPERATOR. Per-field half: the searched fields use analyzers that split the same text differently; the typical trigger is a word one field keeps whole and another splits, such as wi-fi. Graph half: autoGeneratePhraseQueries=true (with splitOnWhitespace=true) and a multi-word synonym or WordDelimiterGraphFilter with catenation; it hits even one field with one analyzer.
• Default vs opt-in. AND and per-field analysis are application choices; given those, the two-word case needs no other setting, because splitOnWhitespace=false is the default since 7.0.0. The graph half needs the opt-in autoGeneratePhraseQueries.
• Both directions. The per-field half loses matches and can add wrong ones (doc 1). The graph half only loses matches: requiring every alternative can never match more than one of them.
• Scale. None: one or two documents and a one- or two-word query. Silent: fewer or different hits.
• Reach. Solr, Elasticsearch and OpenSearch do not reference the class (code search of their default branches, not exhaustive); applications that embed Lucene directly can. The recorded search found no user report in the roughly nine years since the default changed (7.0.0, 2017); it was not exhaustive. The likelihood is rated plausible.
• Release timing. The graph half and single-word per-field input first shipped in 10.6.0 (released 2026-10-29) and are on main and branch_10_6. The branch_10_6 backport of #16446 adds no CHANGES entry there.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: body:"guinea pig" body:cavy vs +(body:"guinea pig") +(body:cavy) expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.queryparser.classic.TestMultiFieldQueryParserSynonymGraph.testMultiWordSynonymUnderAnd(TestMultiFieldQueryParserSynonymGraph.java:65)

• The contract (javadoc of both constructors, MultiFieldQueryParser.java:35-55, 61-76): the query becomes (title:term1 body:term1) (title:term2 body:term2), under AND +(title:term1 body:term1) +(title:term2 body:term2), "In other words, all the query's terms must appear, but it doesn't matter in what fields they appear." The example uses the same terms in both fields and does not say what a term is when the fields' analyzers differ. Natural consistency properties suggested by it, not explicit API guarantees: with one field the class matches what QueryParser on that field matches; when every field keeps every query word, adding a field does not lose a match; and a document that lacks a query word in every field does not match.
• Per-field analysis is supported for query parsing. PerFieldAnalyzerWrapper "can be used like any other analyzer, for both indexing and query parsing" (PerFieldAnalyzerWrapper.java:42-43). No javadoc, CHANGES, MIGRATE or package text restricts the class to one analysis for all fields.
• Graph clauses are alternatives. analyzeGraphPhrase builds "phrase queries" from "all the finite strings from the graph" and adds each as SHOULD (QueryBuilder.java:571-579).
• The class disagrees with itself. A quoted "wi-fi" takes getFieldQuery(String,String,int) (MultiFieldQueryParser.java:82-108), which ORs each field's whole query and keeps the alternatives as SHOULD; the flexible StandardQueryParser builds +(title:wi-fi (+body:wi +body:fi)) +(title:router body:router) and finds doc 0.
• Tests and history. #16446's tests (TestMultiFieldQueryParser.java:403-457, TestMultiFieldQueryParserBoostAndOperator) use one shared analyzer with aligned positions; none uses autoGeneratePhraseQueries or fields that split text differently. The #16446 discussion (25 comments, 10 reviews) never considers either. The pairing came with LUCENE-2605 (2016); in 2017 the same author guarded the same pairing in Solr's edismax with allSameQueryStructure (SOLR-9185), which Lucene's parser never got.

The strongest case for "intended" (the devil's-advocate verifier, the adjudication and reviews by OpenAI GPT-5.6): #16446 deliberately requires every analyzed position under AND, and its new javadoc names "synonym expansion" among the causes (MultiFieldQueryParser.java:211-219); an upstream test pins required groups; a 2009 java-user mail by uschindler said one "can use MultiFieldQueryParser, but not if the analyzer for both fields is different", because the different analyzers "will produce different tokens"; and the per-field half is nine years old, so it could be called a known limitation.

Why it fails: graph clauses are alternatives, not positions, and requiring a word and its synonym together contradicts QueryParser on the same field; #16443/#16446 and their tests cover aligned positions from one shared analyzer only; nothing in the docs restricts the class to uniform analysis, and the 2009 remark, though on point, is a list answer, not documentation, and predates the 2016 code that zips by clause index; a group like (title:router body:fi) has no meaning under any reading of the contract; and Solr guarded the same hazard. Verdict: a defect, clearly for the graph half new in 10.6.0; for the per-field half, which has shipped since 2016, whether to change it or keep it as a limitation is the maintainers' compatibility decision.

The root cause is one method: pairing is only valid when clause i of every field's query covers the same words. Two options were built and run in a scratch worktree at ea57360912ed (removed afterwards); the others are untested.

Builds on the finder's structure check (option A) and adds sameTermSpans: it re-analyzes the text for each field with getAnalyzer().tokenStream(field, text) and compares the (start offset, end offset) of every position; stacked tokens widen the previous span. If either check fails, the method ORs each field's whole query. Results: wi-fi router gives (+title:wi-fi +title:router) (+body:wi +body:fi +body:router), hits [0]; wi-fi gives title:wi-fi (+body:wi +body:fi), [0].

if (maxTerms > 1
    && (termPositionsAligned(fieldQueries) == false   // same size, all clauses MUST under AND
        || sameTermSpans(fieldQueries, queryText) == false)) { // same text span per position
  for (int i = 0; i < fields.length; i++) {
    if (fieldQueries[i] != null) {
      clauses.add(applyBoost(fieldQueries[i], fields[i])); // keep each field's query whole
    }
  }
  return getMultiFieldQuery(clauses);
}
// ... the existing loop that pairs clause i of every field, unchanged

Trade-offs

• Fallback = the union of the per-field queries. When analyses differ, a document matches only if one field alone holds all the words. A document with title wi-fi and body router is not found for wi-fi router, though the javadoc's expansion would find it (stock main does not find it either).
• Stop-word behaviour changes. If title keeps "the" and body drops it, the matrix now matches a body-only "the matrix" document, as QueryParser("body") does; the literal expansion +title:the +(title:matrix body:matrix) requires title:the, and stock main gives +(title:the body:matrix) +(title:matrix), which requires title:matrix; both miss that document. Worth a CHANGES line.
• Normalisation-only differences keep the pairing. Lowercased vs case-sensitive fields with the same token boundaries still give +(title:router body:Router) +(title:cheap body:Cheap).
• Cost. One extra analysis per field, only for multi-position text whose structure already lines up.
• It trusts the offsets (read, not run). termSpans gets the OffsetAttribute with addAttribute, which creates it when the chain has none (AttributeSource.java:209-220), and OffsetAttributeImpl starts and clears at (0, 0) (:27-28, :63-67). So fields whose chains set no offsets give equal (0, 0) spans, and equal-length but misaligned queries are still zipped, as on stock main. WordDelimiterGraphFilter's 3-argument constructor turns adjustInternalOffsets off (WordDelimiterGraphFilter.java:249-250), so its parts keep the incoming token's offsets (:263, :401-403).
• It assumes repeatable analysis and flattens graphs (read, not run). It analyzes the text a second time, so a stateful or nondeterministic analyzer could give a different stream from the one that built the query; it merges tokens with position increment 0 into one span and ignores larger increments and PositionLengthAttribute. More design work is needed before proposing it upstream.

What the checks validated

• Gradle, :lucene:queryparser:test with this doc's test and five evidence tests from the hunt (the finder's, verifiers', two security probes): 548 tests, 0 failures, 1 skipped. Stock: 548 tests, 15 failures, exactly the 15 bug tests. With only this doc's test: 536 tests, 0 failures.
• The classic package with -Ptests.asserts=false -Ptests.seed=1234ABCD: 112 tests, 0 failures (with this doc's test and the 12 evidence tests from the hunt; 100 tests, 0 failures with only this doc's test, the fact check's run).
• checkGoogleJavaFormat, forbiddenApisMain and forbiddenApisTest for the module pass.
• Standalone runner with the fix: this doc's test 3 of 3 (assertions on and off), and every verifier's and security probe's test, including the WordDelimiterGraphFilter and equal-length over-match tests, pass.

Pair only if every per-field query is a BooleanQuery of the same size whose clauses all carry the default-operator occur; else OR the whole per-field queries (+39 lines). Solr's allSameQueryStructure is the same kind of check.

Trade-offs

Equal-length lists with different words are still paired: the wi-fi with title [the, wi-fi] and a stop-word body [wi, fi] gives +(title:the body:wi) +(title:wi-fi body:fi), which admits a document with no "the" and no "fi". SOLR-16594 discusses the same weakness of structure-only checks. Otherwise as B.

What the checks validated

Gradle, same worktree: 548 tests, 1 failure, the equal-length over-match test. This doc's test passes 3 of 3.

Analyze each field with offsets and put every field's clauses for the same words into one group: +(title:wi-fi (+body:wi +body:fi)) +(title:router body:router). That is what the flexible StandardQueryParser already builds for this input.

Trade-offs

Keeps cross-field matches and follows the javadoc exactly, but needs more code, and token graphs and stacked tokens need care.

Fall back to whole per-field queries when a field's BooleanQuery has SHOULD clauses under AND (the occur part of A).

Trade-offs

Removes the #16446 regression only; the per-field half stays as in the releases.

Drop the pairing entirely.

Trade-offs

Breaks #16446's test, which asserts +(field1:foo field2:foo) +(field1:bar field2:bar) for a shared analyzer (read, not run), and drops cross-field matching everywhere.

Method: semi-formal (a hypothesis formed while modelling, then witnessed by a tool).

• A hunt unit (the classic query parser, 2026-10-26). The finder, an AI tool, wrote a Z3 model of getFieldQuery's per-field grouping: 1 to 3 fields, per-field query shapes none, one term, k positions or k alternatives (k = 2..3), both operators, shared and per-field analyzers. The lead came from reading #16446 while writing the model.
• The model enumerated the (shape, operator) classes where the code disagrees with its reading of the documented per-term expansion: with one shared analyzer only graph alternatives under AND; with per-field analyzers also misaligned shapes under AND. The enumeration ran to exhaustion for up to 2 fields and for 3 fields with a shared analyzer; for 3 fields with per-field analyzers it stopped at its cap of 50 classes.
• A bounded-exhaustive harness on the stock jar (every query of at most 4 symbols, 6 analyzer families, 24 parser configurations) found the oracle "one field equals QueryParser" failing 10,640 times, all multi-word synonyms with AND and autoGeneratePhraseQueries, and "adding a field never loses a match" failing 12,030 times, all per-field analyzers with AND. A public-API JUnit test then confirmed both.
• An independent reviewer from another model family (OpenAI GPT-5.6) reviewed it, verifiers and an adjudication confirmed it, and the history, prior-report and security checks followed.