IndriAndQuery explains documents that match no clause as matches, with another document's score, or throws

IndriAndWeight.explain (IndriAndWeight.java:76-106 at 645d4fe14a20) collects the explanations of the matching clauses and returns a no-match only when a required clause fails (:86-94). With SHOULD clauses, the only Occur upstream TestIndriAndQuery uses, every document reaches the tail (:96-100): when at least one clause has a scorer in the segment, it takes the query's scorer, calls scorer.iterator().advance(doc), checks the result only with assert advanced == doc, and returns Explanation.match(scorer.score(), "sum of:", subs). advance(doc) returns the first match at or after doc, so for a document that matches no clause and is not a hit the scorer stands on the next hit and explain reports a match with that hit's score. If no hit follows, score() runs at NO_MORE_DOCS and the norms read throws when the norms vary (on main with the default codec: java.io.EOFException on MMapDirectory and NIOFSDirectory, IndexOutOfBoundsException on a plain ByteBuffersDirectory; the class depends on the Directory and the version); with constant or omitted norms it returns a meaningless match instead. With assertions on, every such call (the scorer past the document) throws AssertionError. BooleanWeight.explain, which this method mirrors, returns noMatch("No matching clauses") before the same tail; the Indri copy lost that guard. This bug does not change hits, counts or scores; every release since 8.9.0 and main are affected.

1. Entry. IndexSearcher.explain(Query, int) (IndexSearcher.java:941-944) calls explain(Weight, int) (:959-968), which takes any valid document id, returns a no-match for a deleted one (:964-966) and otherwise calls weight.explain(ctx, doc) (:967).
2. Clause loop. IndriAndWeight.java:79-91: a matching clause's explanation goes into subs (:84-85); a non-matching clause sets fail only if it is required (:86-90). A non-matching SHOULD clause is dropped.
3. The tail (:96-100), reached whenever fail is false:

Scorer scorer = scorer(context);
if (scorer != null) {
  int advanced = scorer.iterator().advance(doc);
  assert advanced == doc;                                    // :99, the only check
  return Explanation.match(scorer.score(), "sum of:", subs); // :100
}

scorer(context) is null only when no clause has a scorer in the segment (getScorer, :47-65); then explain correctly says no match (:101-104). With one clause the scorer is that clause's scorer; with several it is an IndriAndScorer (:60-63).

1. Where the scorer lands. advance(target) "Advances to the first beyond the current whose document number is greater than or equal to target ... Exhausts the iterator and returns NO_MORE_DOCS if target is greater than the highest document number in the set" (DocIdSetIterator.java:137-139). For a document that matches no clause and is not a hit, that is the next hit, whose score explain then reports, or NO_MORE_DOCS (2147483647), where score() reads norms for a document that does not exist.
2. Everyone else checks. BooleanWeight.explain returns noMatch("No matching clauses", subs) when no clause matched (BooleanWeight.java:109-111) and only then reaches the same advance(doc); assert advanced == doc tail (:126-129). TermWeight.explain uses the advance only if (newDoc == doc) (TermQuery.java:228-229); the default Weight#matches, which IndriAndWeight does not override, returns null when advance(doc) != doc (Weight.java:78-92); ConstantScoreWeight.explain (:52-54) and PhraseWeight.explain (:75) check too.

With assertions on, every row throws AssertionError at IndriAndWeight.java:99. A BooleanQuery with the same SHOULD clause explains every non-hit as a non-match. The exception class past the last hit depends on the Directory and the version: on main EOFException on MMapDirectory and NIOFSDirectory, IndexOutOfBoundsException on a plain ByteBuffersDirectory (and on MMapDirectory too in 8.9.0, 8.11.4 and 9.0.0); with constant or omitted norms the call returns a match with a meaningless value instead (0.0 when every document had the same length).

• Who is hit. Code that builds an IndriAndQuery (public, in org.apache.lucene.search, not marked experimental or internal; no in-tree parser or other code builds it) and explains a document that is not a hit, for example a "why did this document not match?" tool or a check that explains every document.
• Nested queries. The false match spreads: BooleanQuery(MUST IndriAndQuery(george, washington), SHOULD president) explains its non-hits 1 and 3 as matches (0.0 and -1.8996168) with assertions off and throws AssertionError with assertions on.
• Default vs opt-in. Default codec, IndexWriterConfig, searcher and similarity (BM25; also IndriDirichletSimilarity); opt-in only in choosing IndriAndQuery. Any index size.
• Effect. A wrong diagnostic (a non-hit reported as a match with another document's score), or one exception for that call: EOFException or IndexOutOfBoundsException with assertions off, AssertionError with assertions on (as in Lucene's own test runs). No state is left behind.
• Workaround. Check that the document is a hit (or that weight.matches(ctx, doc) is non-null) before calling explain. Both follow the query's scorer, so with two-phase clauses they share the separate over-matching of LFV-874.

The test fixes the directory, the documents, the query and the searcher (a plain IndexSearcher, BM25); the test framework still picks its environment per seed (seed DEADBEEF ran with codec=Asserting(Lucene104), locale=nb); seeds DEADBEEF and C0FFEE gave the same message. The test needs org.apache.lucene.tests.*, so it compiles against 9.1.0 and later.

This test shows the false match. The exception after the last hit comes from the same line; the five-document case in the full explanation above shows it.

Suggested regression test: this same test, as one method in TestIndriAndQuery or as its own class.

# main 645d4fe14a20, gradle, seed DEADBEEF, -Ptests.asserts=true
TestIndriAndExplainNonMatch > testExplainDocumentMatchingNoClause FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:8A07A2C5A9153847]:0)
        at org.apache.lucene.search.IndriAndWeight.explain(IndriAndWeight.java:99)
        at org.apache.lucene.search.IndexSearcher.explain(IndexSearcher.java:967)
        at org.apache.lucene.search.IndexSearcher.explain(IndexSearcher.java:943)
        at org.apache.lucene.search.TestIndriAndExplainNonMatch.testExplainDocumentMatchingNoClause(TestIndriAndExplainNonMatch.java:46)
:lucene:core:test (FAILURE): 1 test, 1  failure

# the same, -Ptests.asserts=false
TestIndriAndExplainNonMatch > testExplainDocumentMatchingNoClause FAILED
    java.lang.AssertionError: expected no match for doc 0, got 0.13076457 = sum of:
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:8A07A2C5A9153847]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertFalse(Assert.java:65)
        at org.apache.lucene.search.TestIndriAndExplainNonMatch.testExplainDocumentMatchingNoClause(TestIndriAndExplainNonMatch.java:48)
:lucene:core:test (FAILURE): 1 test, 1  failure

# 9.12.3 and 10.6.0 release jars, -da: the same 'expected no match for doc 0, got 0.13076457 = sum of:'
# with the advance-check fix on main: 1 test, 0 failures (assertions on and off)

• The contract covers non-hits. Weight#explain: "An explanation of the score computation for the named document" (Weight.java:98). Explanation.noMatch is documented "for a document which does not match" (Explanation.java:50, :55), and isMatch "Indicates whether this Explanation models a match" (:77). IndexSearcher.explain takes any valid document id and gives a deleted one a no-match (IndexSearcher.java:964-966).
• Lucene's test framework requires it. CheckHits.checkNoMatchExplanations (lucene/test-framework/.../CheckHits.java:60-88) asserts that every non-hit up to maxDoc explains with isMatch() == false; it runs in, among others, TestSimpleExplanationsOfNonMatches, TestComplexExplanationsOfNonMatches and TestSpanExplanationsOfNonMatches. Applied unchanged to upstream TestIndriAndQuery's corpus, similarity (IndriDirichletSimilarity) and query, it fails: "Explanation of [[george washington]] for #1 doesn't indicate non-match: 0.0 = sum of:". No upstream test calls explain on an Indri query.
• The assert is a copied invariant, not a precondition. The method mirrors BooleanWeight.explain (same structure, messages and tail). There the tail came with LUCENE-8008 ("Reduce leniency in CheckHits", d27ddcb409d, 2017) and is reached only after the matchCount == 0 guard, which makes the assert true. The Indri copy kept the assert and dropped the guard (an inference from the code text; the PR does not say).
• History. In the introducing PR (apache/lucene-solr#2097) a reviewer asked at these lines whether scorer can be null (r555792974, 2021-01-12); the follow-up added only scorer != null. Nobody discussed where advance(doc) lands.
• Precedent. LUCENE-7065 fixed the same symptom in the global-ordinals join query as a bug: "Before the explain would also indicate that non matching documents would match." (lucene/CHANGES.txt:9496-9498).

Strongest case (from the cross-model reviewer, the intended-behaviour checker and the adjudication): the search package docs frame explain around "a certain matching document" (package-info.java:230-232), so the assert may state a hit-only precondition; Indri smoothing gives every document a score (IndriAndScorer.java:22-25); the feature is lightly maintained (on #10577 jpountz noted in 2021 that all matching documents scored 0); and the only open PR on this code, #463 (stale), keeps the unchecked advance and copies it into a new IndriOrWeight.

Why it fails: no javadoc states a hit-only precondition, and the package sentence describes typical use. The value explain returns is not this document's smoothing score but another document's score: on the upstream corpus doc 0 explains as -2.9882789, the search score of hit doc 2, while TermScorer#smoothingScore(0) is -2.9982293. Search, count, Weight#matches and a BooleanQuery with the same clauses all agree that these documents do not match; only explain disagrees. Throwing EOFException or AssertionError for a valid document id is no documented outcome, and a stale, unmerged PR is no design decision.

The recommended fix was applied to a scratch worktree of 645d4fe14a20 and run through gradle; the alternatives were not run.

Fold the advance into the condition of IndriAndWeight.explain's tail (the diff replaces IndriAndWeight.java:97-99), so a document the scorer skips gets the existing no-match explanation.

Scorer scorer = scorer(context);
if (scorer != null && scorer.iterator().advance(doc) == doc) {
  return Explanation.match(scorer.score(), "sum of:", subs);
} else {
  return Explanation.noMatch(
      "Failure to meet condition(s) of required/prohibited clause(s)", subs);
}

Trade-offs

• The same advance(doc) == doc comparison that TermWeight.explain and the default Weight#matches make; one comparison in an already expensive diagnostic call; hits are explained exactly as before.
• The no-match message is the existing generic one, which reads oddly for a SHOULD-only query; BooleanWeight's "No matching clauses" may read better (next option).
• With several clauses the check is only as exact as IndriAndScorer's iterator, which has its own issue with two-phase clauses (LFV-874, a separate fix site).
• The open, stale PR #463 would need the same check in IndriAndWeight and in its new IndriOrWeight if it is ever revived.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20 with the repro test added, seed DEADBEEF: stock, the repro fails with -Ptests.asserts=true and false; with the fix it passes both ways.
• With the fix: TestIndri* and *Explanations* plus the repro, 8 classes, 238 tests, 0 failures (2 skipped); the whole core org.apache.lucene.search.* test package (with similarities and comparators), 215 classes, 2,340 tests, 0 failures, 0 errors, 49 skipped.
• :lucene:core:checkGoogleJavaFormat passes with the fix.
• Earlier hunt runs with the fix shadowed: the finder's 2 tests, the independent reproduction's 6 tests (also seed C0FFEE), the intended-behaviour check's probe and upstream TestIndriAndQuery pass; on 8.9.0 the history check goes from 8 of 8 wrong explains to 0 of 8.

Before the tail, return Explanation.noMatch("No matching clauses", ...) when no clause explanation matched, and keep the advance check. Not run.

Trade-offs

A clearer message for the common case, and the method then matches BooleanWeight.explain line for line. Do not use the clause test instead of the advance check: explain must agree with the scorer, and where the scorer and the clause explanations disagree (LFV-874's two-phase case) a clause-only test would explain a returned hit as a non-match (by reading).

Leave the code and state a hit-only precondition in the javadoc. Not run.

Trade-offs

No code change, but it contradicts CheckHits.checkNoMatchExplanations, the other Weight#explain implementations and IndexSearcher.explain's handling of non-hits, and explain still throws after the last hit. Not recommended.

Method: semi-formal. The first signal (2026-10-26) came from a bounded-exhaustive translation-validation harness that compares explain with search scores over the stock jars: its IndriAndQuery rows showed 303 non-hits explained as matches and 4 IndexOutOfBoundsException on non-hits; that unit kept it as an unfiled lead. A later unit (2026-10-28, main a9779ef3c193) wrote a Z3 model of IndriAndQuery's matching and explain rules: the wrong explanation is satisfiable with SHOULD-only clauses, needs a clause with a scorer in the segment and no matching clause, and the advance check provably removes it for SHOULD-only clauses. A bounded-exhaustive check of the real core jar against that model ran 60,288 (query, layout, doc) checks with 0 model mismatches; 1,760 non-hits were explained as matches (assertions off) or threw (assertions on), counted in two disjoint rows: 1,480 with at least one MUST, FILTER or MUST_NOT clause and 280 with SHOULD-only clauses. With the advance check shadowed both rows were empty and exactly those 1,760 explains differed from the model of the stock code. A public-API JUnit witnessed it; a later minimisation step reduced it to the one-method test shown here.