Flexible StandardQueryParser: (the fox)@1 with a stop-word analyzer, or the boosted group (fox dog)^2@1, throws ClassCastException

The flexible StandardQueryParser (and PrecedenceQueryParser, which shares its builders) documents a min-should-match operator for disjunction groups, (blue crab fish)@2, and boosts on grouped clauses. MinShouldMatchNodeBuilder builds the @N node by casting the group's query to BooleanQuery without a check. That query is not a BooleanQuery when a stop-word analyzer leaves one clause in the group (the parser then replaces the group's one-clause Boolean node by that clause), when the group carries a boost (the grammar puts the boost inside the @N node), or when the group holds a single term, phrase or prefix. parse() then throws java.lang.ClassCastException instead of returning a query or its declared QueryNodeException.

The failure is immediate and deterministic, with assertions on and off and with either default operator; there are no wrong results and no index effect. Every release with the operator is affected, 9.1.0 (2022) to 10.6.0, and main. A one-file fix in the builder (move the boost outside, accept a lone query as one SHOULD clause) passes the repro and all 534 queryparser tests.

1. Grammar. GroupingExpr is <LPAREN> q = Query(field) <RPAREN> (q = Boost(q))? ("@" minShouldMatch = <NUMBER>)? and then builds new MinShouldMatchNode(parseInt(minShouldMatch), new GroupQueryNode(q)) (StandardSyntaxParser.jj:358-361). The boost is applied first, so in (fox dog)^2@1 the @N node wraps the boost; (fox dog)@1^2 is not in the grammar (a ParseException).
2. Analysis. AnalyzerQueryNodeProcessor (StandardQueryNodeProcessorPipeline.java:58) turns a term that analyzes to no token into a NoTokenFoundQueryNode (AnalyzerQueryNodeProcessor.java:162-163), a DeletedQueryNode that RemoveDeletedQueryNodesProcessor removes (pipeline :63).
3. One-child collapse. BooleanSingleChildOptimizationQueryNodeProcessor (pipeline :65) "removes every BooleanQueryNode that contains only one child and returns this child" (:28-34, code :42-58). It is the Boolean node inside the group that is replaced, so the group keeps only the field node body:fox: GroupQueryNode [BooleanQueryNode [FieldQueryNode(body:fox)]] becomes GroupQueryNode [FieldQueryNode(body:fox)] (the diagram probe's trace).
4. Group builder. GroupQueryNodeBuilder returns the child's query unchanged (GroupQueryNodeBuilder.java:36-39): a TermQuery for (the fox), a BoostQuery for (fox dog)^2, a PhraseQuery for ("quick fox").
5. The cast. MinShouldMatchNodeBuilder.build reads that query (:38), does BooleanQuery booleanQuery = (BooleanQuery) q; (:40), and copies its clauses into a new BooleanQuery with the minimum set (:41-44). Its javadoc: "Builds a BooleanQuery from a MinShouldMatchNode" (:27).
6. Escape. QueryTreeBuilder.processNode calls the builder directly (QueryTreeBuilder.java:165), so the ClassCastException leaves StandardQueryParser.parse (:277-279), which declares only QueryNodeException (:274; QueryParserHelper.java:211-213).

PrecedenceQueryParser extends StandardQueryParser and only swaps the processor pipeline (PrecedenceQueryParser.java:35, :41, :50), so it fails the same way.

The table is a selection: the independent reproducer's probe also saw the cast for ((fox dog)^2)@1 (BoostQuery), (fox~)@1 (FuzzyQuery), (*:*)@1 (MatchAllDocsQuery), ([a TO c])@1 (TermRangeQuery) and (body:fox)@1 (TermQuery), with OR and AND. Any group whose query is not an unboosted BooleanQuery fails.

(the a)@1 never reaches the builder: NoChildOptimizationQueryNodeProcessor (pipeline :62) turns the emptied Boolean node into a MatchNoDocsQueryNode, a DeletedQueryNode (NoChildOptimizationQueryNodeProcessor.java:43-62, MatchNoDocsQueryNode.java:23), and RemoveDeletedQueryNodesProcessor (pipeline :63) deletes the group and the @N node above it and returns a match-no-docs root (RemoveDeletedQueryNodesProcessor.java:38-43, :50-72). With default operator AND, BooleanQuery2ModifierNodeProcessor (pipeline :61) makes the clauses required; @N is documented for OR groups only, so the AND row shows only that the cast fails there too.

The no-argument StandardAnalyzer() has no stop words, so the stop-word trigger needs a stop-word analyzer (an explicit stop set, EnglishAnalyzer or StopAnalyzer); the boost and one-clause triggers need none.

• What goes wrong. A valid query is refused with an undeclared ClassCastException. Code that catches only QueryNodeException to answer "invalid query" sees an unexpected runtime exception instead.
• When. Whenever the group does not build a BooleanQuery: a stop word leaves one clause (users cannot know which words the analyzer drops), the group carries a boost, or a query generator renders a one-element list as (t)@1.
• Limits. Deterministic and immediate; nothing is searched, so no wrong results; no hang, no index effect, no scale needed. ((a b)@1)^2 is a workaround for the boost case only.
• Reach is narrow. @N is the flexible parser's own syntax (the classic QueryParser has no such operator), and no public report was found since 9.1.0, released in March 2022.

The test sits in the parser's test package, as Lucene's own parser tests do, but uses no package-private member. It is also the suggested regression test, unchanged, for the recommended fix (with the QueryNodeException alternative, (the fox)@1 would still throw and the test would have to expect that exception); it passes checkGoogleJavaFormat, ecjLint and forbiddenApis as is.

The second lead trigger, the boosted group (fox dog)^2@1, has the same root cause and the same fix; it was run as a separate probe (ClassCastException: BoostQuery cannot be cast to BooleanQuery, same line) and is left out of the test.

The same file fails on every release from 9.1.0 to 10.6.0 (assertions on and off); 9.0.0 has no @N operator and needs the pre-9.1 test-framework import.

TestMinShouldMatchStopwordGroup > testStopwordLeavesOneClauseInGroup FAILED
    java.lang.AssertionError: parse("(the fox)@1") must return a query but threw java.lang.ClassCastException: class org.apache.lucene.search.TermQuery cannot be cast to class org.apache.lucene.search.BooleanQuery (org.apache.lucene.search.TermQuery and org.apache.lucene.search.BooleanQuery are in module org.apache.lucene.core@11.0.0-SNAPSHOT of loader 'app')
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:8450CFC8EF670AD6]:0)
        at org.apache.lucene.queryparser.flexible.standard.TestMinShouldMatchStopwordGroup.testStopwordLeavesOneClauseInGroup(TestMinShouldMatchStopwordGroup.java:33)

        Caused by:
        java.lang.ClassCastException: class org.apache.lucene.search.TermQuery cannot be cast to class org.apache.lucene.search.BooleanQuery (...)
            at org.apache.lucene.queryparser.flexible.standard.builders.MinShouldMatchNodeBuilder.build(MinShouldMatchNodeBuilder.java:40)
            at org.apache.lucene.queryparser.flexible.standard.builders.MinShouldMatchNodeBuilder.build(MinShouldMatchNodeBuilder.java:28)
            at org.apache.lucene.queryparser.flexible.core.builders.QueryTreeBuilder.processNode(QueryTreeBuilder.java:165)
            ...
            at org.apache.lucene.queryparser.flexible.core.QueryParserHelper.parse(QueryParserHelper.java:222)
            at org.apache.lucene.queryparser.flexible.standard.StandardQueryParser.parse(StandardQueryParser.java:279)
:lucene:queryparser:test (FAILURE): 1 test, 1  failure

# boosted group, separate probe, same line:
java.lang.ClassCastException: class org.apache.lucene.search.BoostQuery cannot be cast to class org.apache.lucene.search.BooleanQuery

• The input is written in the documented form. (the fox)@1 and (the OR fox)@1 are disjunction groups as written, the shape of the javadoc examples (StandardQueryParser.java:195, :198) and of the only upstream test, (f:foo OR f:bar OR f:baz)@2 (TestStandardQPEnhancements.java:55-67). The stop word is removed by the parser's own pipeline; with two survivors, (the quick fox)@2, it works.
• The boost order is the grammar's own. The same commit wrote GroupingExpr so that (a b)^2@1 is the only accepted boost-plus-@ order, and the class javadoc documents boosts on grouped clauses (StandardQueryParser.java:163-164).
• The declared failure is QueryNodeException (:274). The grammar already rejects unsupported @ uses that way: (fox dog)@1^2, fox@1 and (fox dog)@x are ParseExceptions. This is a robustness argument, not a broken contract: Java lets any method throw an unchecked exception, and the javadoc does not say what @N does once analysis leaves one clause or when a boost sits inside it. But nothing documents a ClassCastException, and the parser refuses every other unsupported @ use with the declared exception.
• Nobody discussed it. No test, javadoc, CHANGES or MIGRATE entry, issue or review comment mentions the cast or non-Boolean groups; the only test uses a three-clause unboosted group, which is why it never failed.
• Precedent. LUCENE-4103 (#5175, fixed 2012), a ClassCastException in the same flexible pipeline, was fixed as a bug.

The strongest case for "intended" (moderate; raised by every verifier): @N is documented only for "a disjunction Boolean query (a query with only "OR"-subclauses)" (StandardQueryParser.java:190-191) and "Boolean groups" (standard/package-info.java:33), so direct one-clause or one-phrase groups such as (fox)@1 or ("quick fox")@1 are arguably outside the contract, and their meaning is unspecified.

Why it fails: it does not cover the lead triggers, two-clause OR groups reduced by the parser's own analysis and a documented group boost, and even for inputs outside the contract the declared failure is QueryNodeException; nothing documents a ClassCastException. It is a fair reason to accept a QueryNodeException for the direct one-clause shapes.

One fix was built and checked through gradle in a scratch worktree of 645d4fe14a20; the alternatives come from the verifiers and were not run.

Unwrap any BoostQuery around the group's query and re-apply the boost outside the result; copy a BooleanQuery's clauses as today; otherwise add the query as the one SHOULD clause. BooleanQuery already defines a minimum larger than the SHOULD count (no match), so (the fox)@2 gives (body:fox)~2, like (fox dog)@3 today. Also update the builder javadoc (:27) and add a CHANGES entry.

float boost = 1f;
while (q instanceof BoostQuery bq) {
  boost *= bq.getBoost();
  q = bq.getQuery();
}
BooleanQuery.Builder builder = new BooleanQuery.Builder();
builder.setMinimumNumberShouldMatch(mmNode.minShouldMatch);
if (q instanceof BooleanQuery booleanQuery) {
  booleanQuery.clauses().forEach(builder::add);
} else {
  builder.add(q, BooleanClause.Occur.SHOULD);
}
Query result = builder.build();
return boost == 1f ? result : new BoostQuery(result, boost);

Trade-offs

• Gives direct one-clause groups such as (fox)@1 a meaning the javadoc does not define (the natural one: at least N of 1 clause).
• A lone user + is not kept: (+fox)@1 gives (body:fox)~1, as (+fox) without @ gives body:fox today.
• Every input that built a query before builds the same query: the old code succeeded only when the group's query was an unboosted BooleanQuery, and the new code takes the same path for it (by reading; the shape probe agrees for the controls, (the quick fox)@2, (-fox)@1, (the a)@1 and AND groups).

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: stock, the repro fails with assertions on and off; with the fix it passes with assertions on and off.
• With the fix: org.apache.lucene.queryparser.flexible.* 162 tests, 1 skipped, 0 failures (all 23 TestStandardQPEnhancements tests); the whole :lucene:queryparser:test 534 tests, 1 skipped, 0 failures (an independent gradle re-run in the fact check counted the same 534 tests in 31 classes from the JUnit XML); :lucene:queryparser:check -x test passes with the repro test in place.
• Standalone runner: the repro, the boosted-group probe, the 22-input shape probe and the 9.1.0 release pass with the fixed builder shadowed in.

Keep the cast for a BooleanQuery and throw QueryNodeException otherwise (the builder interface allows it, QueryBuilder.java:35).

Trade-offs

Smallest change and turns the crash into the declared failure, but it still refuses (the fox)@1 and (fox dog)^2@1, which are written in documented syntax; acceptable for the direct one-clause shapes only, so it needs the boost unwrap anyway. The builder sees the same TermQuery for (the fox)@1 as for (fox)@1, so it refuses both, and the suggested regression test would have to expect the exception. Not run.

Build Boost(MinShouldMatchNode(...)) in GroupingExpr.

Trade-offs

Fixes only the boost trigger, needs the JavaCC parser regenerated, and still needs a builder change for the stop-word and one-clause triggers. Not run.

Make BooleanSingleChildOptimizationQueryNodeProcessor skip the group of a MinShouldMatchNode.

Trade-offs

Fixes the stop-word trigger only, couples a generic processor to one node type, and leaves the boost and phrase shapes. Not recommended; not run.

Method: semi-formal (a hunt unit, the flexible parser's analysis processors composed with the rest of the processor pipeline).

• A probe parsed 57 edge-case query strings (stop words, synonyms, word delimiters, CJK, boosts, phrases, @N groups) under 6 analyzers, each with StandardQueryParser and PrecedenceQueryParser and default operator OR and AND (1,368 parses), and recorded the query or exception of each.
• The only unchecked exception class in the output was this ClassCastException, on (the fox)@2 with the three analyzers that drop "the" (all four parser variants), while (the fox dog)@2 built normally.
• An AI tool then read MinShouldMatchNodeBuilder and the grammar and found the analysis-independent triggers (the boosted group and direct one-clause groups), and wrote the public-API JUnit test.
• So the exploration targeted the analysis pipeline in general; the @N strings were among its edge cases, not a hypothesis about this cast.