Classic QueryParser: since #16446 a stopword right before AND/OR changes which documents match

With the default splitOnWhitespace=false, the classic QueryParser sends a run of consecutive plain words to the analyzer as one string, but keeps a word that is followed by AND/OR out of the run, so that the operator binds to that one word. #16446 (merged 2026-10-22, milestone 10.6.0) changed how the run is added: one clause for the whole run instead of one clause per word position. When the word right before AND/OR is dropped by the analyzer (a stopword, or with the default StandardAnalyzer a lone , or en dash), it adds no clause, and the operator's "make the preceding clause required (or optional)" step lands on the whole run.

So apple banana the AND cherry parses to +(body:apple body:banana) +body:cherry instead of body:apple +body:banana +body:cherry: on five documents it matches 3 instead of 2. Under the AND default, apple banana the OR cherry becomes (+body:apple +body:banana) body:cherry instead of +body:apple body:banana body:cherry, which gains and loses documents. 10.5.2 and splitOnWhitespace=true give the stopword-free parse. MultiFieldQueryParser is affected the same way. No error is raised. Only 10.6.0, main and branch_10_6 contain it; a fix that adds a run's positions as separate clauses again, without changing their flags, passes the whole queryparser suite but has two known gaps (per-field analyzers, and subclasses that return alternatives).

• Run: two or more consecutive plain words with no operator between them. With splitOnWhitespace=false, the default since 7.0.0, the parser analyses a run as one string.
• Dropped token: a word for which the analyzer emits no token (a stopword, or a lone , with StandardAnalyzer), so it produces no clause.
• Default operator: OR unless set to AND; it decides whether plain words are optional (SHOULD) or required (MUST).

1. MultiTerm takes a following word only if the token after it is not :, *, ~, ^, AND or OR (disallowedPostMultiTerm, QueryParser.jj:167-171, lookaheads :389-391). In apple banana the AND cherry the run is apple banana; the stays out because AND follows it.
2. The run is analysed once (QueryParser.jj:398, :404); newFieldQuery passes the default operator's occur, so the result is a BooleanQuery with one clause per position (QueryParserBase.java:405-411).
3. Since #16446 it is added as one clause, addClause(clauses, CONJ_NONE, MOD_NONE, firstQuery) (QueryParser.jj:405). Before, addMultiTermClauses added the positions one by one: unchanged when all were TermQuerys, otherwise each child with the default operator's occur (10.5.2 QueryParserBase.java:421-448).
4. the analyses to nothing: addClause returns at if (q == null) return; (QueryParserBase.java:369) and adds no clause.
5. AND cherry: "If this term is introduced by AND, make the preceding term required" sets the last clause in the list to MUST (QueryParserBase.java:349-355); under the AND default, OR sets it to SHOULD (:357-365). That last clause is now the whole run, so banana loses its own MUST. In 10.5.2 it was banana.
6. MultiFieldQueryParser builds one field disjunction per position and, since #16446, combines them with the default operator (MultiFieldQueryParser.java:205); the run is again one clause, with the same result.

Documents: d0 "apple cherry", d1 "apple banana cherry", d2 "banana cherry", d3 "cherry", d4 "apple banana". && and || behave like AND and OR. A document with only "apple" is lost under the AND default (security probe).

Not affected: a stopword that is not next to the operator, a single word before the stopword (apple the AND cherry), splitOnWhitespace=true, and the operator that equals the default (only the structure changes). A run that analyses to one unit, such as one multi-word synonym, was already bound as a unit on 10.5.2.

• Who is exposed. Users can be affected with default parser settings (splitOnWhitespace=false, either default operator) with any analyzer that drops a token: a stop filter (EnglishAnalyzer, or StandardAnalyzer with a stop set), and the default StandardAnalyzer with an empty stop set for a space-separated , or en dash: apple banana , AND cherry gives +(apple banana) +cherry on main and apple +banana +cherry on 10.5.2.
• What it takes. Two or more plain words that survive analysis, then the dropped token, then AND/&& (OR default) or OR/|| (AND default). A field-qualified or modified dropped word (apple banana title:the AND cherry, apple banana -the AND cherry) gives the same wrong parse for both parsers and both default operators (run by the fact check). Code that joins a user's text with AND produces the shape whenever the text ends with a stopword after two real words, though the package docs say the parser "is designed for human-entered text, not for program-generated text" (package-info.java:82-85).
• What goes wrong. A query of the wrong structure, silently. OR default with AND: it can only add documents. AND default with OR: it can gain and lose documents (both were observed). On the same build, splitOnWhitespace=true and false now disagree for analyzers that emit one token per word.
• Scale. Deterministic; no index size or data shape is involved (40 of 40 runs fail). No prevalence was measured; the registry rates it rare.
• Release status. 10.6.0 (released 2026-10-29), main (11.0.0-SNAPSHOT) and branch_10_6. branch_10x does not contain #16446.
• Downstream (by code reading, not run). Elasticsearch's and OpenSearch's query_string parsers extend the classic parser with the default splitOnWhitespace and would inherit the change on a Lucene with #16446 (both use 10.5.2 today). Solr keeps its own grammar copy and is not affected.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<body:apple +body:banana +body:cherry> but was:<+(body:apple body:banana) +body:cherry>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.queryparser.classic.TestStopwordBeforeAnd.testStopwordBeforeAndDoesNotChangeQuery(TestStopwordBeforeAnd.java:31)

• The code's own rule. addClause: "If this term is introduced by AND, make the preceding term required" (QueryParserBase.java:349) and "If this term is introduced by OR, make the preceding term optional" (:358). The grammar makes sure that preceding clause is never a run of plain words by ending runs before AND/OR (QueryParser.jj:167-171); it can still be a field-qualified term, a phrase or a parenthesized group (:297-372).
• The docs. "The AND operator matches documents where both terms exist" (package-info.java:300); "The OR operator links two terms" (:284). setSplitOnWhitespace only says "Whether query text should be split on whitespace prior to analysis" (QueryParser.jj:152).
• Upstream tests. LUCENE-2605's test: "Operators should interrupt multiword analysis of adjacent words if they associate", with guinea pig AND dogs -> guinea +pig +Synonym(dog dogs) (TestQueryParser.java:656, :679). #16446's testDefaultOperatorAppliesStopwordNextToTerm: +foo bar the must parse like +foo bar (TestMultiFieldQueryParser.java:439-456). No test covers a run, a dropped token, then AND/OR.
• History. #16446 fixed two MultiFieldQueryParser bugs (#16441, #16443). In its thread the changed expectations were called "cosmetic", producing "exactly same results", and the author, after re-running the #10355 boolean precedence cases (A AND B OR C AND D, ...) on 8.11.3, 10.5.1 and the PR, said "the general clause handling isn't affected"; a maintainer noted it would be a breaking change before merging and backporting it to 10.6. CHANGES.txt (under 11.0.0 on main) lists only the removal of addMultiTermClauses and the two MultiFieldQueryParser fixes; MIGRATE.md has nothing, and on branch_10_6 (10.6.0, released 2026-10-29) the backport adds no CHANGES.txt entry at all.

The strongest "works as designed" case (review, the independent reproduction, the intended-behaviour check, adjudication): no document says how AND/OR binds next to a dropped token, and the package docs say "the query parser syntax may change from release to release" (package-info.java:74); addClause's rule (adjust the last surviving clause) is old and unchanged, and main's result equals the long-standing parse of the explicit (apple banana) the AND cherry; the flexible StandardQueryParser already gives a third answer (appl banana +cherri) on both versions; committers have called odd classic AND/OR results expected before (#8535); the package docs say the parser "is designed for human-entered text, not for program-generated text" (package-info.java:82-85), which weakens the case of code that joins text with AND (not a person typing it); and grouping the run is coherent in itself ((apple OR banana) AND cherry), most of all for MultiFieldQueryParser, where #16446 deliberately combines a run's positions into one unit.

Why it fails: the change was described as not changing results and nothing announces it; it makes splitOnWhitespace true and false disagree on one build; the grammar and the upstream tests bind an operator to the word before it; and #16446's own test says a stopword next to a term must not change the parse. Which reading applies depends only on whether a dropped word was typed. The independent reproduction, the intended-behaviour check and the adjudication classify it as real (the intended-behaviour check and the adjudication with medium confidence, because the contract is implicit; the independent reproduction records none); the runs show the change, not intent.

One fix was built and validated in a scratch worktree at ea57360912ed (option A), together with a simpler variant that has a known corner (A1). A probe after the fact check found two shapes option A gets wrong (below), so an upstream fix may need a different test of what a term position is. Any fix must keep #16441 (a per-field disjunction must not be re-flagged MUST under the AND default), #16443 (positions combined with the default operator) and multi-word synonym grouping.

A package-private QueryParserBase.addMultiTermRunClauses replaces the addClause call in MultiTerm (QueryParser.jj:405; QueryParser.java and the checksum file regenerated). If the run's query is a BooleanQuery without minimumNumberShouldMatch whose clauses all have the default operator's occur and all cover the same fields (one clause per position), the clauses are added as they are; any other query is added as one clause, as on main. The same-occur guard keeps #16441 fixed (+foo bar the still parses like +foo bar); the same-fields guard keeps a one-position MultiFieldQueryParser field disjunction in one piece.

  void addMultiTermRunClauses(List<BooleanClause> clauses, Query q) {
    if (q instanceof BooleanQuery bq
        && bq.getMinimumNumberShouldMatch() == 0
        && bq.clauses().isEmpty() == false) {
      Occur defaultOccur = operator == OR_OPERATOR ? Occur.SHOULD : Occur.MUST;
      Set<String> fields = fieldsOf(bq.clauses().get(0).query());
      boolean positions = true;
      for (BooleanClause clause : bq.clauses()) {
        if (clause.occur() != defaultOccur || fields.equals(fieldsOf(clause.query())) == false) {
          positions = false;
          break;
        }
      }
      if (positions) {
        clauses.addAll(bq.clauses());
        return;
      }
    }
    addClause(clauses, CONJ_NONE, MOD_NONE, q);
  }
  // fieldsOf(q): the fields that q.visit(...) reports through QueryVisitor#acceptField

Trade-offs

• Output strings go back to 10.5.2's. Ordinary runs print flat again (motley crue motl* cru* instead of (motley crue) motl* cru*), so the fix reverts #16446's expectation changes in TestQueryParser (the multiTermGroup overrides, 14 synonym and 5 ASCII-folding strings). Matches of those queries do not change. #16446 knowingly accepted and tested the grouping; option B keeps it.
• Two guards decide what a position is, and they can be wrong both ways. Gap 1, per-field analyzers (executed): MultiFieldQueryParser over t and b, where b also drops banana; apple banana the AND cherry gives +((t:apple b:apple) (t:banana)) +(t:cherry b:cherry) on main and with the fix (the positions cover different fields, so the guard declines), while 10.5.2, splitOnWhitespace=true and the stopword-free query give (t:apple b:apple) +(t:banana) +(t:cherry b:cherry): the bug remains. Gap 2, a subclass whose getFieldQuery returns two same-field SHOULD alternatives body:x body:y for the run (executed): main gives +(body:x body:y) +body:z for a b the AND z, the fix body:x +body:y +body:z (as 10.5.2 did), so a document with only x and z is lost against main. Also by reading: fieldsOf ignores MUST_NOT children (QueryVisitor.getSubVisitor returns EMPTY_VISITOR, QueryVisitor.java:79-82) and gives an empty set for a query that reports no field.
• Scores, rewritten queries and query-cache keys of the flattened queries were not compared; the tests and probes compare parses and match sets only.
• Not run on branch_10_6 (the code uses instanceof patterns and BooleanClause record accessors, both on 10.x), on x86, or in a benchmark (the check runs once per run, at parse time).

What the checks validated

• Gradle, scratch worktree at ea57360912ed, JDK 25: the finder's and the independent tests go from 5 of 7 failing to 7 of 7 passing; with the fix as shadow sources over the stock jars they pass with assertions on and off, and the security probe's 4 tests pass.
• Whole lucene/queryparser suite with the fix and its tests: 30 classes, 536 tests, 0 failures, 1 skipped; :lucene:queryparser:check -x test passes (google-java-format, ecj, forbidden APIs, generated-file checksums). regenerateClassicQueryParser gives a QueryParser.java identical to the hand edit. The 2 new tests fail on the stock sources.
• Translation-validation harness: two words and a stopword, every query of up to 5 symbols (434,688 evaluations): 48 + 48 mismatches on stock, 0 with the fix. All six analyzer families up to 4 symbols (6,366,096 evaluations): identical counts with and without the fix.
• Not covered by these runs: per-field analyzers and getFieldQuery overrides; the gap probe (a public-API main() class) ran them on stock main, with the fix as shadow sources and on 10.5.2.

Flatten whenever every clause has the default occur.

Trade-offs

Passes the repro and the whole module suite, but re-breaks a corner that main gets right: MultiFieldQueryParser, OR default, apple the the AND cherry gives t:apple +b:apple +(t:cherry b:cherry) instead of +(t:apple b:apple) +(t:cherry b:cherry), which is what main and splitOnWhitespace=true give (10.5.2 had the same flaw as A1).

What the checks validated

Module suite with the finder's, the independent and a probe test: 33 classes, 541 tests, 0 failures; the corner shown by a parse probe.

Remember the clause MultiTerm added; when addClause is about to apply AND/OR to it (which only happens when the tokens in between were dropped), replace it by its per-position clauses first.

Trade-offs

Keeps every toString() and test expectation of #16446 and changes only the buggy shape; needs parser state across addClause calls and the same position guards. Not built.

Skip the previous-clause adjustment when the clause just before the operator was dropped.

Trade-offs

Tried as an addClause override on main (not as a patch): it gives (apple banana) +cherry, which matches neither 10.5.2 nor splitOnWhitespace=true, and keeps the long-standing drop AND stop AND roll -> +drop +roll, because the first AND still makes drop required. A wider variant that also ignores the AND that introduces a dropped token gives the same (apple banana) +cherry and changes that test to drop +roll. Both live in addClause, so they also change splitOnWhitespace=true (by reading). Not recommended.

Restore addMultiTermClauses.

Trade-offs

Makes the repro pass (executed in the history check) but brings back the re-flagging that #16441 fixed: the old method re-added non-TermQuery clauses with the default occur. Not acceptable.

setSplitOnWhitespace(true) gives the 10.5.2 parse, but changes multi-word synonym handling; or leave out any dropped token (a stopword, a lone ,) before an explicit operator.

Trade-offs

Application-side only.

Method: SMT (Z3) proof attempt, then bounded-exhaustive translation validation on the stock jar (the hunt's "formal" tier: the solver produced the first signal).

• In a hunt unit (classic query parser, 2026-10-26) the finder, an AI tool, transcribed QueryParserBase.addClause and the MultiTerm rule into a Z3 model and asked whether splitOnWhitespace=false and true give the same match set for every clause sequence of up to 6 items, both operators, all modifiers and conjunctions, stopwords anywhere. The model is a single-field Boolean abstraction of bare clauses (no real token streams, synonyms, fields, MultiFieldQueryParser, scoring or subclasses). Within it: proved (unsat) up to length 3; counterexample at length 4: a b <stopword> AND +c under the OR default (the + is redundant after AND, so it equals a b <stopword> AND c, the JUnit's shape).
• Controls: without stopwords the equivalence is proved up to length 7; a mutant without the grammar's "no AND/OR right after a run" rule fails from length 3 (vacuity check).
• An independent reference grammar and match-set semantics, checked against the real QueryParser and MultiFieldQueryParser on every query of up to 5 symbols over two words and a stopword, found only mismatches of the shape w w <stopword> AND|OR x. The finder then wrote the public-API JUnit; the independent checks, history, prior-report search and security assessment followed.