ComplexPhraseQueryParser skips its same-field check for prefix and regexp clauses inside a phrase

ComplexPhraseQueryParser parses the text inside quotes a second time and, as its javadoc says, checks that every clause in it is for the phrase's own field. The check runs for term, wildcard, fuzzy and range clauses but not for prefix (id:2*) or regular-expression (id:/2/) clauses, because the class never overrides getPrefixQuery or getRegexpQuery. So "jo* id:2*" parses and the search then throws IllegalArgumentException: Clauses must have same field., "jo* id:9*" silently matches nothing, and the one-clause phrase name:"id:2*" searches field id. The prefix gap is as old as the class (2.9.0, 2009), the regexp gap dates from 4.0.0-alpha, and both are still on main; two small overrides fix it.

• The class javadoc: the second pass parses phrase content "checking all embedded clauses are referring to the same field and therefore can be rewritten as Span queries" (ComplexPhraseQueryParser.java:51-53).
• checkPhraseClauseIsForSameField (:153-164) throws ParseException("Cannot have clause for field ... nested in phrase for field ..."). It is called only from newTermQuery (:141-151, wrapped in a RuntimeException because that method cannot throw a checked exception), getWildcardQuery (:166-172), getRangeQuery (:174-182) and getFuzzyQuery (:198-205).
• The classic parser's handleBareTokenQuery sends x* words to getPrefixQuery and /re/ words to getRegexpQuery (QueryParserBase.java:747-754). Both build their query directly (:709-715, :679-686), and ComplexPhraseQueryParser overrides neither, so their field is never checked. No commit ever added or removed such an override (git log -S).

• One clause: a lone TermQuery, MultiTermQuery or SynonymQuery is returned unchanged (:261-265). id:2* is a MultiTermQuery, so name:"id:2*" runs as id:2* on field id.
• Several clauses: each clause is rewritten (:285). A foreign prefix or regexp that matches terms becomes a span clause on field id: a span term when it matches one term (id:2* rewrites to id:2, :317-318), a span OR when it matches several (:294-300, :371-414). Then new SpanNearQuery(...) (:333) throws IllegalArgumentException("Clauses must have same field.") (SpanNearQuery.java:119-126).
• A foreign clause that matches nothing is replaced by a placeholder term on the phrase's field (:299-315, "Dummy clause because no terms found - must match nothing"): 0 hits, no error.

With a synonym analyzer, a plain word for another field becomes a SynonymQuery through QueryBuilder.newSynonymQuery instead of newTermQuery, so "smith id:dogs" (with dogs -> dogs|dog) also parses and fails at search, while "smith id:cats" is rejected. The two overrides do not cover it; fix B below does.

• Who is hit: code that parses user queries with ComplexPhraseQueryParser, default parser settings, documented classic syntax, public API. The trigger is a field qualifier on a prefix or regexp word inside a quoted phrase, for example title:"jo* author:sm*". No index size, setting or concurrency is involved.
• Several clauses, the foreign clause matches a term: the rejection moves from parse() (a checked ParseException) to IndexSearcher.search (an unchecked IllegalArgumentException). Code that catches ParseException to report a bad query does not see it.
• Several clauses, the foreign clause matches nothing: no error and 0 hits.
• One clause: no error; name:"id:2*" searches field id. That field is one the user typed, and the classic syntax lets them search it at top level anyway (id:2*), so nothing new is reachable.
• Assertions: -ea and -da behave the same.
• Likelihood: rare. A malformed query shape in an opt-in parser; no report was found in the 17 years since 2.9.0. An open proposal (#16820) would deprecate this parser for Lucene 11.

new ComplexPhraseQueryParser("name", new MockAnalyzer(random())) parses "jo* id:2*"; the test expects a ParseException, the type the wildcard, fuzzy and range checks already throw and the type both fixes throw here. Deterministic, single thread; it needs Lucene's test framework (LuceneTestCase as the base class, MockAnalyzer, expectThrows) and no package-private API.

The test shows the accepted parse only. The search-time IllegalArgumentException, the regexp form, the silent 0 hits and the one-clause search on field id are in the table above (a separate probe, not part of the repro).

Regression test: this same class (or its one method added to TestComplexPhraseQuery, next to the checkBadQuery lines); it passes with fix A and with fix B.

cp $REPRO/TestComplexPhrasePrefixClauseOtherField.java $LUCENE/lucene/queryparser/src/test/org/apache/lucene/queryparser/complexPhrase/
cd $LUCENE && ./gradlew :lucene:queryparser:test --tests TestComplexPhrasePrefixClauseOtherField -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false: same failure)
   >     junit.framework.AssertionFailedError: Expected exception ParseException but no exception was thrown
   >         at __randomizedtesting.SeedInfo.seed([DEADBEEF:65C37B6BF24F3147]:0)
   >         at app/org.apache.lucene.test_framework@11.0.0-SNAPSHOT/org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1432)
   >         at app/org.apache.lucene.test_framework@11.0.0-SNAPSHOT/org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
   >         at app//org.apache.lucene.queryparser.complexPhrase.TestComplexPhrasePrefixClauseOtherField.testPrefixClauseForOtherFieldIsRejected(TestComplexPhrasePrefixClauseOtherField.java:28)

:lucene:queryparser:test (FAILURE): 1 test, 1  failure

# standalone, seed DEADBEEF: main 645d4fe14a20 and 10.6.0, -ea and -da, fail with the same message

• The class promises the check. ComplexPhraseQueryParser.java:51-53 (quoted above); the pass-1 comment says pass 2 runs "with different set of syntax restrictions (i.e. all fields must be same)" (:119-121).
• The original test pins it at parse time. TestComplexPhraseQuery.java:87: checkBadQuery("\"jo* id:1 smith\""); // mixing fields in a phrase is bad; checkBadQuery (:127-135) expects qp.parse(...) itself to throw. It dates from the class's creation (LUCENE-1486).
• Every other clause kind is checked. QueryParserBase.java:688-693 calls getPrefixQuery "similar to getWildcardQuery" and a prefix "a special case of generic wildcard term"; :706-707 documents "throw in overridden method to disallow" as the way to reject input.
• Later changes kept the rule. LUCENE-5530 (#6593, 4.8.0) made author:"fred* smith" work and kept the rejection of other-field clauses. LUCENE-7614 (#8665, 6.4.0) added the one-clause fast path but did not lift the check: name:"id:2?" and name:"id:2" are still rejected (probe).
• Precedent. In #5943 (LUCENE-4878) the issue body quotes Simon Willnauer: "This is in-fact a bug in the MultiFieldQueryParser [...] MultifieldQueryParser should override getRegexpQuery but it doesn't"; it was fixed with the missing override in 4.2.1.
• The recorded searches found nothing that says otherwise. No javadoc, test, CHANGES or MIGRATE entry, and no issue, PR or list thread among the recorded searches, presents prefix or regexp clauses as exempt. The adjudication, the verifiers and the cross-model reviews rated the intended-behaviour argument weak; a cross-model fact check agreed for intent but rated it moderate as a "not worth changing" compatibility argument, so a behaviour-changing fix is something to discuss.

Strongest case for "intended": (1) the classic parser's docs say wildcard searches work "within single terms (not within phrase queries)" (classic/package-info.java:162-163); (2) the input is malformed, and the multi-clause form still fails, only later; (3) the parser always validated loosely: in #2560 (LUCENE-1486) its author, Mark Harwood, introduced the list of supported in-phrase operators with "these are not necessarily definitive answers or guarantees of correctly implemented support", and "john -percival" is a known non-working case commented out in TestComplexPhraseQuery.java:81-82; (4) LUCENE-7614's CHANGES entry says the parser "ignores double quotes around single token prefix, wildcard, range queries", so name:"id:2*" could be read as id:2*; (5) #16820 proposes deprecating the parser.

Why it fails: (1) is the generic limitation that ComplexPhraseQueryParser exists to lift, and its own javadoc states the same-field rule. (2), (3) and (5) are about severity and priority, not intent; (3) is about operators, while the field rule has a dedicated check and a test. (4) proves too much: the same entry names wildcard and range queries, yet name:"id:2?" and name:"id:[1 TO 2]" are still rejected.

Fix A is the finder's patch; an independent verifier wrote the same two overrides without reading it. Fix B adds one more override for the same-root synonym path. Both were validated with gradle in a scratch worktree of 645d4fe14a20. Both change behaviour (queries that parse today are rejected), so they are a choice: the narrow fix A, which leaves the synonym path open, or the complete fix B, which also turns the other-field term-clause error from a wrapped RuntimeException into a plain ParseException.

Two overrides next to the four existing checks, 1 file, +16 lines.

  @Override
  protected Query getPrefixQuery(String field, String termStr) throws ParseException {
    if (isPass2ResolvingPhrases) {
      checkPhraseClauseIsForSameField(field);
    }
    return super.getPrefixQuery(field, termStr);
  }

  @Override
  protected Query getRegexpQuery(String field, String termStr) throws ParseException {
    if (isPass2ResolvingPhrases) {
      checkPhraseClauseIsForSameField(field);
    }
    return super.getRegexpQuery(field, termStr);
  }

Trade-offs

• The smallest change, in the style of the existing checks. A query that used to parse and then fail (or return nothing, or search the other field) now fails in parse(); anyone relying on name:"id:2*" searching field id would get a ParseException (worth a CHANGES line).
• Same-field prefixes and regexps ("jo* smi*", "jo* name:smi*", "jo* /smi.h/") keep working.
• Leaves the synonym sibling ("smith id:dogs" with a synonym analyzer) open; fix B closes it.
• Solr's {!complexphrase} would get the check: on apache/solr main its getPrefixQuery override calls super and getRegexpQuery is inherited (read, not run).

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3: the repro fails on stock with assertions on and off and passes with the fix, on and off.
• org.apache.lucene.queryparser.complexPhrase.* (12 tests: the repro 1, upstream TestComplexPhraseQuery 11) passes with seeds DEADBEEF and CAFEBABE; the whole :lucene:queryparser:test passes with seed DEADBEEF (31 test classes, 534 tests, 1 skipped, 0 failures).
• An independent verifier's 9-test probe: 8 of 9 pass; only the synonym case ("smith id:dogs") still fails, as expected.
• A probe of 13 queries with the fix: every other-field prefix and regexp clause, including "jo* id:9*" and name:"id:2*", is rejected by parse() with the usual ParseException; the same-field controls still find john smith.
• Earlier, with the fix shadowed: the finder's, the independent reproduction's and separate tests and upstream TestComplexPhraseQuery all pass.

Fix A plus one more override, +9 lines; every plain word passes through it before QueryBuilder builds a term, synonym, boolean or phrase query.

  @Override
  protected Query getFieldQuery(String field, String queryText, boolean quoted)
      throws ParseException {
    if (isPass2ResolvingPhrases) {
      checkPhraseClauseIsForSameField(field);
    }
    return super.getFieldQuery(field, queryText, quoted);
  }

Trade-offs

• Also rejects "smith id:dogs" with a synonym analyzer at parse time.
• Side effect: a term clause for another field ("jo* id:2") is then rejected with a plain ParseException instead of a RuntimeException wrapping one; the newTermQuery check becomes redundant on that path but is harmless.
• A broader change than the bug strictly needs; the synonym path's history was not traced.

What the checks validated

• Gradle, same worktree: the repro passes; the independent verifier's 9-test probe, including the synonym case, passes 9 of 9; the complexPhrase package (12 tests) and the whole :lucene:queryparser:test (31 classes, 534 tests, 1 skipped, 0 failures) pass with seed DEADBEEF.
• The 13-query probe with fix B: every other-field clause, term clauses included, is rejected with ParseException; the same-field controls still find john smith.

State that prefix and regexp clauses are not checked.

Trade-offs

Leaves the late unchecked exception, the silent 0 hits and the other-field search, and contradicts the class javadoc and the existing checkBadQuery test.

Non-formal: a hunt unit on the complex-phrase and extendable query parsers read ComplexPhraseQueryParser's pass-2 overrides. Its bounded-exhaustive harness did not generate clauses for another field and missed the gap; code reading found it, probes witnessed it, and a 4-test public-API JUnit (3 fail, the control passes) reproduced it. Independent checks, two of them by a reviewer from another model family (OpenAI GPT-5.6), then confirmed it.