IndriAndQuery silently accepts MUST, FILTER and MUST_NOT clauses, matches them as SHOULD, and explains some of its own hits as non-matches

IndriAndQuery (core, since 8.9.0) takes a List<BooleanClause>, so every clause carries an Occur, and its public constructor accepts any of them without a check. When it matches, it builds a scorer for every clause and iterates their union, whatever the Occur: MUST and FILTER clauses do not narrow the hits, and a MUST_NOT clause is one more disjunct, so it keeps the documents it names instead of removing them (a lone MUST_NOT clause returns exactly them). Its explain, a trimmed copy of BooleanWeight.explain, does treat MUST and FILTER as required, so IndexSearcher.explain reports some returned hits as non-matches. In the repro, +f:a -f:b over doc 0 {f=a} and doc 1 {f=b} returns both documents (a BooleanQuery returns doc 0) and explains doc 1 as "Failure to meet condition(s) of required/prohibited clause(s)". Matching all clauses as a disjunction is the Indri design for SHOULD clauses; what is wrong is that other Occur values are accepted silently and undocumented, and that explain contradicts the hits.

// IndriAndQuery.java
/** A Query that matches documents matching combinations of subqueries. */          // :22
public IndriAndQuery(List<BooleanClause> clauses) { super(clauses); }              // :25-27, no check
// IndriQuery.java: this.clauses = clauses; (:33); toString appends c.occur() (:46) -> "+f:a -f:b"

// IndriAndWeight.java
for (BooleanClause c : query) {                                                    // :41
  Weight w = searcher.createWeight(c.query(), scoreMode, 1.0f);                    // :42, Occur not read
  weights.add(w);
}
private Scorer getScorer(LeafReaderContext context) {                              // :47
  for (Weight w : weights) { /* keep every non-null scorer */ }                    // :50-55
  Scorer scorer = subScorers.get(0);                                               // :60, lone clause as is
  if (subScorers.size() > 1) scorer = new IndriAndScorer(subScorers, scoreMode, boost);  // :61-63
}
// IndriDisjunctionScorer.java:36-41: DisjunctionDISIApproximation over ALL sub-scorers

public Explanation explain(LeafReaderContext context, int doc) {                   // :76
  ...
  if (e.isMatch()) { subs.add(e); }
  else if (c.isRequired()) {                                                       // :86, MUST or FILTER
    subs.add(Explanation.noMatch("no match on required clause (" + c.query() + ")", e));
    fail = true;
  }
  ...
  if (fail) return Explanation.noMatch("Failure to meet condition(s) of required/prohibited clause(s)", subs); // :92-94

• BooleanClause.isRequired() is occur == MUST || occur == FILTER (BooleanClause.java:80-82).
• IndriAndWeight.explain keeps only the isRequired branch of BooleanWeight.explain (BooleanWeight.java:65-108, same failure message at :108); Boolean's MUST_NOT branch (:86) was not copied, which is why a MUST_NOT hit still explains as a match.
• With one clause scorer, getScorer returns that clause's own scorer, so a lone MUST_NOT clause returns exactly the documents it names.

The four documents: doc 0 {f=a, acl=x}, 1 {f=a, acl=y}, 2 {f=a, f=secret, acl=x}, 3 {f=b, acl=x} (StringFields, default similarity). Same results with assertions on and off.

• Who is hit. Applications that build an IndriAndQuery with a MUST, FILTER or MUST_NOT clause. Default codec, writer config, similarity and searcher; any index size; public API only. Queries with SHOULD clauses only are unaffected.
• What they see. MUST and FILTER clauses do not narrow the hits, a MUST_NOT clause is matched as one more disjunct, so it keeps the documents it names instead of removing them (alone, it returns exactly them), and IndexSearcher.explain reports some returned hits as non-matches. No exception, no index damage.
• Why someone passes them. The constructor asks for an Occur on every clause, the class is called "And", and toString prints +, # and -. The Lucindri parser (lemurproject/Lucindri, on lucene-core 8.10.0) builds Lucene's IndriAndQuery with Occur.MUST and Occur.MUST_NOT for its #scoreif / #scoreifnot operators, which its README documents as filters (read, not executed); on Lucene's class they would not filter.
• Diagnostics. Lucene's generic query check asserts, for any query it is given, that each collected hit explains as a match (QueryUtils.check -> CheckHits.checkExplanations -> ExplanationAsserter); IndriAndQuery breaks that agreement through its public constructor.
• Scale. Every release since 8.9.0 (2021). Low priority: a niche API, and the matching half has no stated contract.

One test method, two documents, one segment, single thread, deterministic; public API only. It sits next to TestIndriAndQuery. The test accepts the two outcomes that make the hits follow the clauses: rejecting the clauses (fix A) or honouring them with an explain that agrees (fix B). It still fails under the document-only option C.

Regression test: this same test. It passes with fix A (gradle) and with fix B (standalone runner, with the replacement classes).

catch (IllegalArgumentException _) needs Java 22 or later; on a 9.x or 10.x branch name the variable if that branch's lint allows it. The test does not run on 9.0.0, whose test framework predates the 9.1.0 package rename (a limit of the test setup, not the bug). The gradle test framework randomizes the codec (CheapBastard for seed DEADBEEF, Asserting(Lucene104) for seed 1CEB00DA, same failure); the supporting probe uses the plain defaults (default codec, BM25).

TestIndriAndQueryOccur > testMustAndMustNotClausesAreNotSilentlyIgnored FAILED
    java.lang.AssertionError: expected:<[doc 0 explain isMatch=true]> but was:<[doc 0 explain isMatch=true, doc 1 explain isMatch=false]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:FA465899BAD282F6]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.search.TestIndriAndQueryOccur.testMustAndMustNotClausesAreNotSilentlyIgnored(TestIndriAndQueryOccur.java:61)
:lucene:core:test (FAILURE): 1 test, 1  failure

# supporting probe on the same two documents (not the repro)
BooleanQuery +f:a -f:b -> hits 0 ... count=1
IndriAndQuery +f:a -f:b -> hits 0 1 ... count=2
explain(IndriAndQuery, doc 1):
0.0 = Failure to meet condition(s) of required/prohibited clause(s)
  0.0 = no match on required clause (f:a)
    0.0 = no matching term
  0.31506687 = weight(f:b in 1) [BM25Similarity], result of: ...

• Self-inconsistent. explain applies MUST/FILTER (IndriAndWeight.java:86-94); getScorer and IndriDisjunctionScorer do not (IndriAndWeight.java:47-65, IndriDisjunctionScorer.java:36-41). Whatever the design, one of the two halves is wrong.
• Explain contract. Weight.explain: "An explanation of the score computation for the named document" (Weight.java:97-105); CheckHits.checkExplanations, which QueryUtils.check runs, requires every hit's explanation to "indicate match" (CheckHits.java:607-627).
• No documentation of ignoring. The javadoc says only "A Query that matches documents matching combinations of subqueries" (IndriAndQuery.java:22); the Occur javadoc gives MUST, FILTER and MUST_NOT a meaning (BooleanClause.java:27, :35, :57-61); IndriQuery.toString prints them (:46).
• Tests. TestIndriAndQuery uses SHOULD clauses only (:113-114, :130-131); no test covers other values.
• History. Both behaviours came with the introducing commit 9cc5c9b7983 (LUCENE-9537); the first explain was a copy of BooleanWeight.explain, and the original author's later PR #463 deletes its isRequired branch.

Strongest case for "intended": IndriAndQuery implements the Indri #and belief operator, which combines all its children and has no required or excluded clauses (Indri filters with other operators); the scorer is documented as "The Indri implementation of a disjunction scorer"; in the original review mikemccand wrote "Normally in Lucene AND implies MUST ... But this query is actually disjunctive, right?" (lucene-solr#2097, r532223725) and the name was kept; and BooleanClause is documented as "A clause in a BooleanQuery", so Occur semantics are BooleanQuery's contract.

Why it fails: that shows the union is right for SHOULD clauses, which nobody disputes. It does not justify accepting MUST, FILTER and MUST_NOT silently and undocumented, printing them as if they meant something, or an explain that applies MUST and FILTER while search does not. The intended-behaviour verifier built exactly this case and still called the explain half a real self-inconsistency; an independent reviewer from another model family (OpenAI GPT-5.6) called the search/explain disagreement "especially difficult to defend as intended".

Fix A was validated with gradle in a scratch worktree of 5adff78bb168; B was only checked with two replacement classes put ahead of the core jar in standalone runs; C was not run. The assert advanced == doc line in explain is a separate bug (LFV-875) and stays as it is.

The constructor throws IllegalArgumentException for any clause that is not SHOULD, and its javadoc says so and points to an enclosing BooleanQuery for required, filter and excluded clauses. IndriAndWeight.explain loses the else if (c.isRequired()) branch and the fail flag, exactly as PR #463 does (plus the now unused query field and Iterator import). The diff is formatted with ./gradlew tidy.

public IndriAndQuery(List<BooleanClause> clauses) {
  super(clauses);
  for (BooleanClause clause : clauses) {
    if (clause.occur() != BooleanClause.Occur.SHOULD) {
      throw new IllegalArgumentException(
          "IndriAndQuery only supports SHOULD clauses, got: "
              + clause.occur().name()
              + " "
              + clause.query());
    }
  }
}
// IndriAndWeight.explain: drop the `else if (c.isRequired())` branch and `fail` (as in #463)

Trade-offs

• A behaviour change for callers that pass other Occur values today: they get an exception instead of silently widened hits (by reading, Lucindri's #scoreif / #scoreifnot would start failing loudly). Fine for main; for a 10.x backport a maintainer may prefer C.
• IndriQuery keeps the caller's list as is (IndriQuery.java:33), so mutating that list after construction bypasses the check; List.copyOf would close that, but getClauses() would then return a copy.

What the checks validated

• The repro passes with assertions on and off.
• TestIndri* (TestIndriAndQuery 2 tests and the repro), TestBooleanQuery (30) and TestQueryVisitor (4): 37 tests, 0 failures.
• :lucene:core:check -x test passes (google-java-format, ecj lint, forbidden APIs) with the fix and the new test.

Rewrite a query with non-SHOULD clauses into a BooleanQuery: MUST and FILTER clauses as FILTER, MUST_NOT clauses as MUST_NOT, plus an IndriAndQuery of the SHOULD and MUST clauses (optional when a MUST/FILTER clause exists, as in BooleanQuery), together with the LFV-875 advance check in explain.

Trade-offs

Gives the Indri operator semantics it does not have in Indri, changes the Indri scores of such queries, and goes against the maintainers' direction of narrowing what Indri queries accept (#11194).

What the checks validated

A replacement-class experiment, not a build: the two replacement classes are compiled alone and put ahead of the core jar. This doc's test passes that way. The finder's bounded check of the real classes, run the same way on main a9779ef3c193, no longer reports a widened hit or a hit explained as a non-match over 60,288 checks, but the run as a whole still fails: its model describes the stock behaviour and it reports 45,392 differences from that model, so it is not a clean validation of B. Not run through gradle.

Say in the javadoc that every clause is treated as SHOULD, and drop the isRequired branch of explain (PR #463's change).

Trade-offs

Non-breaking and backportable, and explain then agrees with the hits; but a MUST_NOT clause still matches as one more disjunct instead of excluding, so this doc's test still fails under C.

What the checks validated

Not run.

Method: non-formal first signal, then formal and semi-formal characterization (2026-10-26 to 2026-10-28; the model and the bounded check ran on main a9779ef3c193).

• An executed public-API JUnit test, written while checking the related bug LFV-299 (IndriAndQuery does not rewrite its clauses; a different root and fix site), found [MUST f:a, FILTER acl:x] returning {0, 1, 2, 3} where the same BooleanQuery returns {0, 2}.
• A Z3 model of the matching and explain rules proved that IndriAndQuery equals the BooleanQuery over the same clauses when all clauses are SHOULD, is always a superset, widens exactly when a MUST/FILTER clause misses or a MUST_NOT clause matches, and explains a hit as a non-match only when a MUST/FILTER clause is present.
• A bounded-exhaustive translation validation of the real classes ran 1,884 clause lists (60,288 checks) and found 30,840 widened hits and 18,048 hits explained as non-matches, with 0 mismatches against the model; with the fix-B sketch's replacement classes, it reported neither family (that run still fails as a whole, since its model describes the stock behaviour).