IndriAndQuery returns documents that match none of its clauses when a clause is a phrase or doc-values range query

IndriAndQuery should return the documents that match at least one of its clauses ("matches documents matching combinations of subqueries", IndriAndQuery.java:22). When two or more clauses have a scorer in a segment, IndriAndWeight scores them with an IndriAndScorer, an IndriDisjunctionScorer (IndriAndWeight.java:47-65). That scorer wraps each clause scorer in a DisiWrapper, whose approximation for a two-phase clause is twoPhaseView.approximation() (DisiWrapper.java:69-73), merges those approximations in a DisjunctionDISIApproximation (IndriDisjunctionScorer.java:36-41) and returns the merge from iterator() (:44-47) without a twoPhaseIterator(). Nobody ever calls a clause's matches(), so every document in a phrase clause's "has all the words" set or a doc-values range's "has a value" set (for a field without a skip index, the default) becomes a hit. In the three-document repro the range n:[1 TO 10] next to body:c returns [0, 1, 2], count() 3, instead of [0, 2]; all hits score 0.0, and no error is raised. A segment where only one clause has a scorer uses that scorer directly and is exact, so the result depends on which documents share a segment. Present since the Indri classes were added (LUCENE-9537, first released in 8.9.0), in every release to 10.6.0 and on main 5adff78bb168.

1. Two-phase clauses. A TwoPhaseIterator's matches() "needs to be checked in order to know whether the returned doc ID actually matches" (TwoPhaseIterator.java:23-27). PhraseScorer returns one (PhraseScorer.java:51-52); so does the slow doc-values range without a skip index (TwoPhaseIterator.asDocIdSetIterator(DocValuesRangeIterator.forRange(...)), lucene/core/src/java/org/apache/lucene/document/SortedNumericDocValuesRangeQuery.java:152-163).
2. Scorer choice. IndriAndWeight.getScorer collects the non-null clause scorers; a single one is returned as is, two or more are wrapped in new IndriAndScorer(subScorers, ...) (IndriAndWeight.java:47-65), a subclass of IndriDisjunctionScorer (IndriAndScorer.java:26).
3. The defect. The IndriDisjunctionScorer constructor wraps every clause scorer in new DisiWrapper(scorer, false) (IndriDisjunctionScorer.java:36-40); for a two-phase scorer the wrapper's approximation is twoPhaseView.approximation() (DisiWrapper.java:69-73). It merges them with new DisjunctionDISIApproximation(wrappers, Long.MAX_VALUE) (:41), "a disjunction of the approximations of the provided iterators" (DisjunctionDISIApproximation.java:26-28), and returns that from iterator() (:44-47). It does not override twoPhaseIterator(), so it inherits return null (Scorer.java:58-60).
4. Nobody filters. A consumer that sees no two-phase view treats iterator() as exact: DefaultBulkScorer uses scorer.iterator() when twoPhaseIterator() is null (Weight.java:236-243). Executed on the repro's index: the query's scorer is an IndriAndScorer, twoPhaseIterator() is null, and iterator() yields [0, 1, 2].
5. What the core does. DisjunctionScorer builds the same merge but, when any sub-scorer is two-phase, wraps it in a TwoPhase (DisjunctionScorer.java:42-63), returns TwoPhaseIterator.asDocIdSetIterator(twoPhase) from iterator() and exposes it from twoPhaseIterator() (:66-78); its matches() accepts a document only when a sub-scorer without a two-phase view is on it or a two-phase view matches (:118-157).
6. Scores cannot hide it. IndriAndScorer.scoreDoc adds only children that are IndriScorers (IndriAndScorer.java:47-62); term, phrase and doc-values scorers are not, so the score is 0 (:64-65) for real and false hits alike. That all-zero scoring is a separate, already-discussed issue (#10577); here it only means ranking cannot push the false hits down.
7. Segment dependence. Because of step 2, a segment where only the two-phase clause has a scorer is exact. Executed with two segments, {n=50} and {n=50}, {body=c}: the first segment uses the range's own scorer and its n=50 document is not returned; the second uses an IndriAndScorer and its identical n=50 document is returned (hits [1, 2]; BooleanQuery gives [2]).

The finder's four-document test ("a b" n=1, "b a" n=50, "c" n=2, "x" n=100) gives [0, 1, 2], count 3, for [SHOULD body:"a b", SHOULD body:c] and [0, 1, 2, 3], count 4, for [SHOULD n:[1 TO 10], SHOULD body:c], where only [0, 2] match, on every version run from 8.9.0 to main.

• Who is hit. Code that calls new IndriAndQuery(List<BooleanClause>) (public, in org.apache.lucene.search); outside tests only IndriAndQuery.java and IndriAndWeight.java mention the class, so no parser or other Lucene code builds it.
• Trigger. Two or more clauses with a scorer in the same segment, at least one of them two-phase. Executed with PhraseQuery, SortedNumericDocValuesField.newSlowRangeQuery and NumericDocValuesField.newSlowRangeQuery. By reading, not run, any clause whose scorer returns a TwoPhaseIterator takes the same path, for example MultiPhraseQuery, the SortedNumericDocValuesField and NumericDocValuesField newSlowSetQuery, and the SortedDocValuesField and SortedSetDocValuesField newSlowRangeQuery. The SortedDocValuesField and SortedSetDocValuesField newSlowSetQuery do not get this far: they return a rewrite-only TermInSetQuery, and since IndriAndWeight never rewrites its clauses (IndriAndWeight.java:41-44) they throw UnsupportedOperationException, a separate defect (LFV-299). All runs use SHOULD clauses only; IndriAndWeight ignores the clauses' Occur, a separate defect (LFV-871).
• Default vs opt-in. A plain IndexWriterConfig and IndexSearcher, no skip index; assertions on or off; opt-in only in choosing IndriAndQuery. The runs used the test framework's randomized codec (seed DEADBEEF: Asserting(Lucene104)); by reading, the path depends on the scorers, not on the codec.
• Size. Every document in the two-phase clause's approximation becomes a hit (in each segment where another clause also has a scorer): for a doc-values range on a field without a skip index (the default), every document with any value in the field; for a phrase, every document that has all its words (from the mechanism). With a skip index the candidates are instead every document of each skip block whose value range overlaps the query, documents with no value included (SkipBlockRangeIterator.java:67-87; by reading, not run). count() agrees with the wrong hits, so totals and paging are wrong too.
• Silent and unstable. No exception; all hits score 0.0 (with the clause types tested); whether a document is returned depends on which documents share its segment, so results can change after a flush or merge.
• Bounded. An enclosing BooleanQuery FILTER still restricts every hit, deleted documents never appear, and every extra hit lies in some clause's approximation (an executed bounds probe, 4 of 4 checks pass with assertions on and off). The index is never changed.
• Why tests missed it. Upstream TestIndriAndQuery uses only plain TermQuery clauses: both tests call the two-argument tq() (TestIndriAndQuery.java:113-114, :130-131); a boosted tq(f, t, b) helper exists (:151-154) but is never called.

The test fixes the directory, the documents, the query and the searcher (a plain IndexSearcher); the test framework still picks its environment per seed (seed DEADBEEF ran with codec=Asserting(Lucene104), locale=nl-CW); seed C0FFEE (run with assertions off) gives the same message as DEADBEEF. The field has no skip index, so the range clause takes its two-phase path: its candidates are the documents with an n value (docs 0 and 1), and its matches() would keep only doc 0. The test needs org.apache.lucene.tests.*, so it compiles against 9.1.0 and later.

The same bug with a phrase clause ("b a" returned for the phrase "a b") goes through the same lines; the finder's four-document test, described in the full explanation above, shows it on every version from 8.9.0 to main.

Suggested regression test: this same test, as one method in TestIndriAndQuery or as its own class.

# main 5adff78bb168, gradle, seed DEADBEEF, -Ptests.asserts=false (assertions on: the same failure)
TestIndriAndQueryTwoPhaseClause > testHitsMatchAtLeastOneClause FAILED
    java.lang.AssertionError: expected:<[0, 2]> but was:<[0, 1, 2]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:25136116A809E31E]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.search.TestIndriAndQueryTwoPhaseClause.testHitsMatchAtLeastOneClause(TestIndriAndQueryTwoPhaseClause.java:61)
:lucene:core:test (FAILURE): 1 test, 1  failure

# 10.6.0 and 9.12.3 release jars, -da: the same 'expected:<[0, 2]> but was:<[0, 1, 2]>'
# with the recommended fix on main: 1 test, 0 failures (assertions on and off)

• Contracts. Scorer.java:33-34: "Return a DocIdSetIterator over matching documents". TwoPhaseIterator.java:23-27: after the approximation moves, matches() "needs to be checked in order to know whether the returned doc ID actually matches". IndriAndQuery.java:22: "A Query that matches documents matching combinations of subqueries".
• Tests. TestIndriAndQuery.testSimpleQuery1 asserts "2 docs should match" for [george, washington], testSimpleQuery2 "all docs should match" (TestIndriAndQuery.java:111-144).
• The author's description. "We iterate through all documents that that [sic] have a posting for at least one of the search terms" (cammiemw, lucene-solr#2097 r555482275, 2021-01-12). On #10577 (2020-09-22) Cameron VandenBerg, the feature's author, wrote: "For the IndriDisjuctionScorer, I was using Lucene's DisjunctionScorer as an example for implementing the docID function."
• The core counterpart confirms matches (DisjunctionScorer.java:42-78, :118-157).
• Precedent. #7477 (LUCENE-6418, "SpanNot sometimes forgets to call matches() on excl approximation") was fixed as a bug in 2015.

Strongest case for "misuse". The Indri code was designed for term clauses and nested IndriAndQuery clauses only: the PR's first javadoc read "combinations of {@link TermQuery}s or other IndriAndQuerys"; mikemccand asked on the introducing PR "it looks like we are only supporting TermQuery and other IndriAndQuery clauses ... If so, should we check/enforce this?" (lucene-solr#2097 r532221029, 2020-11-29); jpountz later proposed on #11194 that IndriAnd and IndriOr "only accept instances of IndriQuery as opposed to Query". Read that way, a phrase or range clause is outside the design.

Why it fails. The day after that review comment, the PR commit 8e784de47c27 ("Addressed first set of git pull request comments", 2020-11-30) changed the javadoc to "combinations of subqueries", and no check was added. The reviewer pinged once more on 2020-12-14 and the thread has no reply. The public constructor accepts any BooleanClause (IndriAndQuery.java:25-27), no javadoc, CHANGES or MIGRATE text warns about two-phase clauses, and the restriction proposed on #11194 was never made. Even under an Indri "candidate set" reading, returning n=50 for the range [1, 10] breaks the Scorer#iterator and TwoPhaseIterator contracts above, and the same document is a hit in one segment and not in another. Whether maintainers prefer the scorer fix or a restrictive one (rejecting such clauses, or a move to the sandbox) is their call. Verdict: a defect. Not security: the application builds the query, it is a disjunction rather than a restriction, and enclosing filters still apply.

The recommended fix was applied to a scratch worktree of 5adff78bb168 and run through gradle; the other options were not run.

In the IndriDisjunctionScorer constructor, note whether any DisiWrapper has a twoPhaseView; if so, build a TwoPhaseIterator over the DisjunctionDISIApproximation whose matches() walks approximation.topList() and returns true at the first wrapper with no two-phase view or whose view matches. iterator() returns TwoPhaseIterator.asDocIdSetIterator(twoPhase) and twoPhaseIterator() returns twoPhase; with no two-phase clause nothing changes. +36/-2 lines in IndriDisjunctionScorer.java.

private final DisjunctionDISIApproximation approximation;
private final TwoPhaseIterator twoPhase; // null when no clause is two-phase

// constructor, after building the approximation:
if (hasTwoPhase) {
  final DisjunctionDISIApproximation approx = this.approximation;
  this.twoPhase =
      new TwoPhaseIterator(approx) {
        @Override
        public boolean matches() throws IOException {
          for (DisiWrapper w = approx.topList(); w != null; w = w.next) {
            if (w.twoPhaseView == null || w.twoPhaseView.matches()) {
              return true;
            }
          }
          return false;
        }

        @Override
        public float matchCost() {
          float cost = 0;
          for (DisiWrapper w : wrappers) {
            cost += w.matchCost;
          }
          return cost;
        }
      };
} else {
  this.twoPhase = null;
}

@Override
public DocIdSetIterator iterator() {
  return twoPhase == null ? approximation : TwoPhaseIterator.asDocIdSetIterator(twoPhase);
}

@Override
public TwoPhaseIterator twoPhaseIterator() {
  return twoPhase;
}

Trade-offs

• The same check the core DisjunctionScorer.TwoPhase makes; queries with term clauses only are unchanged. When some clause is two-phase, the new matches() runs on every document of the merged approximation and stops at the first wrapper on it with no two-phase view or a matching view. Unlike DisjunctionScorer.TwoPhase, which checks the exact clauses first (DisjunctionScorer.java:123-143), it may call a two-phase clause's matches() on a document an exact clause already matches; same result, but a reviewer may want that order.
• matchCost is a plain sum of the clauses' costs, a rough upper bound; DisjunctionScorer uses a cost-weighted average (DisjunctionScorer.java:48-61), which a reviewer may prefer.
• Scoring caveat: IndriAndScorer.scoreDoc treats an IndriScorer child as present when its docID() equals the document (IndriAndScorer.java:49-58), so a nested IndriAndQuery child standing on an unconfirmed candidate would still be scored as present. With today's all-zero scores that changes nothing; the next option closes it.
• The open, stale PR #463 would need the same change if it is ever revived: it changes only the superclass of IndriDisjunctionScorer (+2/-2) and adds an IndriOrScorer that extends it.

What the checks validated

• Gradle, scratch worktree of 5adff78bb168 with the repro test added, seed DEADBEEF: stock, the repro fails with -Ptests.asserts=true and false; with the fix it passes both ways.
• With the fix: TestIndri* plus the repro, 3 tests, 0 failures; the whole core org.apache.lucene.search.* test package (with its subpackages), 216 classes, 2,351 tests, 0 failures, 0 errors, 49 skipped.
• :lucene:core:checkGoogleJavaFormat passes with the fix.
• Earlier runs with the same change: the bounded-exhaustive check lost exactly its 119 hits that match no clause (6,708 checks) and gained no miss; the independent reproducer's five cases and the finder's test pass.
• Not run: the full Lucene test suite, and any platform other than Linux aarch64 with OpenJDK 25.0.3.

On top of the first option, have IndriAndScorer.scoreDoc treat a child with a two-phase view as present only when its match on this document was confirmed. Not run.

Trade-offs

Makes scores of nested Indri queries exact once Indri scoring is non-zero. More code: the confirmation must be remembered, because matches() may be called at most once per document (TwoPhaseIterator.java:96-101). No visible effect while all Indri scores are 0.

Validate in the IndriAndQuery constructor that every clause is a (boosted) TermQuery or an IndriQuery, as asked in the 2020 review and proposed on #11194. Not run.

Trade-offs

Stops the wrong hits by refusing phrase and range clauses: it breaks callers that pass them today and takes phrase matching away from Indri queries, while the scorer fix is small. Not recommended.

Method: semi-formal. The hypothesis came from code reading while modelling IndriDisjunctionScorer for a Z3 model of IndriAndQuery matching (2026-10-28, main a9779ef3c193). A Z3 query for "a hit that matches none of its clauses" is satisfiable when two or more clause scorers are present in a segment and one is two-phase, and unsatisfiable with a single scorer, with no two-phase clause, and with the fix.

A bounded-exhaustive translation validation of the real core jar then compared the jar with the model: 258 SHOULD-only clause lists over 3 terms and 3 phrases, 13 documents, 2 segment layouts, 6,708 checks, 0 model mismatches; 119 hits matched none of their clauses (92 in one layout, 27 in the other). With the fix applied to the jar's classes, exactly those 119 hits disappeared and no miss appeared. A public-API JUnit (the finder's four-document test) confirmed it on the stock jars; this doc's minimal test is a smaller version of it.