IndriAndQuery hides its clauses from every QueryVisitor

Tools that need to know what a query contains (term extraction, highlighting, IndexSearcher's cumulative clause limit, the query RAM estimate) walk the query tree with a QueryVisitor, and every query that holds other queries is supposed to pass the visitor on to them. IndriAndQuery holds a list of clauses, but IndriQuery.visit only calls visitor.visitLeaf(this). So for an IndriAndQuery of body:president and body:washington, QueryVisitor.termCollector returns [] instead of both terms, the UnifiedHighlighter in term-extraction mode highlights nothing, a 6-clause IndriAndQuery runs at maxClauseCount=4 while a DisjunctionMaxQuery over the same clauses is refused, and RamUsageEstimator.sizeOf gives 1,024 bytes for 1 clause and for 1,000 (on main the query cache sizes cached keys this way). Search hits are right (matching and scoring never call visit), and no error is raised. It has been there since the Indri classes were added (LUCENE-9537, first released in 8.9.0). A one-method fix that visits every clause through a SHOULD sub-visitor makes all repros pass, and the search-package and highlighter test suites pass with it.

• QueryVisitor: Lucene's query-introspection API. Query#visit is documented to "Recurse through the query tree, visiting any child queries". A query with children pulls a sub-visitor with getSubVisitor(occur, this) and visits each child with it; a query that matches on terms calls consumeTerms; visitLeaf is for "leaf queries that do not match on terms".
• Who walks queries this way: QueryVisitor.termCollector ("collects all terms that may match a query"), the UnifiedHighlighter's term extraction, IndexSearcher's cumulative clause check (run by IndexSearcher.rewrite), RamUsageEstimator.sizeOf(Query), and on main the LRUQueryCache key accounting that uses it.
• IndriAndQuery: a core query (since 8.9.0, LUCENE-9537) built from a List<BooleanClause>; its base class IndriQuery keeps the list, and IndriAndWeight builds a Weight for every clause and, per segment, a scorer for each clause that has one.

IndriQuery.visit (IndriQuery.java:71-74 at a9779ef3c193) is one line, visitor.visitLeaf(this). The visitor is told the whole query is a leaf that does not match on terms, and the clauses are never handed to it. Each consumer then sees an empty box:

1. termCollector collects only from consumeTerms, and visitLeaf is a no-op: it returns [].
2. The UnifiedHighlighter's extractTerms is query.visit(QueryVisitor.termCollector(...)); it is used whenever the highlighter is not in weight-matches mode (withWeightMatches(false), or any of multi-term, strict-phrase or passage-relevancy handling turned off). No terms, so no highlights.
3. IndexSearcher.rewrite runs a clause-counting visitor that adds one per visitLeaf, consumeTerms or consumeTermsMatching call, so a whole IndriAndQuery counts as one clause. Neither Indri class overrides rewrite, and the IndriAndQuery constructor has no size check (BooleanQuery.Builder.add refuses more than maxClauseCount clauses at build time).
4. RamUsageEstimator.sizeOf(Query) starts at the default size for the root and adds nothing for a root that calls visitLeaf, so it is 1,024 bytes whatever the clauses hold.

Matching and scoring do not use visit (only the clause check before a search does): IndriAndWeight iterates the clause list itself, so the hits are right (the security probe found the same hit sets as the SHOULD BooleanQuery for 1, 2, 10 and 50 clauses; it did not compare scores).

Not claimed: in the UnifiedHighlighter's default weight-matches mode the IndriAndQuery is not highlighted either, but for another reason (IndriAndWeight does not implement Weight#matches, whose default reports a match with no term positions); the visit fix does not change that.

• Who is exposed. Only application code builds an IndriAndQuery: in the whole lucene/ tree no parser or builder produces one (only IndriAndQuery.java itself, IndriAndWeight.java and TestIndriAndQuery.java mention it). Every such query is affected, every time, for every visitor-based consumer.
• Term extraction and highlighting. Custom code that calls termCollector (for example to log, cache or route on query terms) gets nothing. The UnifiedHighlighter uses it whenever it is not in weight-matches mode. Two clauses and one document are enough.
• Clause limit. The cumulative check counts the whole IndriAndQuery as one clause, at any nesting depth, so it only matters once counting the clauses would trip the limit: more than maxClauseCount + 1 calls (default 1,025), since the counter checks before it increments. A QueryTimeout still stops a large query.
• RAM estimate. RamUsageEstimator.sizeOf ignores the clauses in every release since 8.9.0; on main (unreleased, #15124) LRUQueryCache sizes cached query keys with it, which needs an application-installed cache (caching is off by default) and a non-scoring use, and grows with the clauses, about 184 bytes per term clause (a cached 16-clause IndriAndQuery: 192 bytes against 3,136 with the fix; 500 clauses: 192 against 92,192; independent reproducer).
• How likely. IndriAndQuery is a rarely used API, and the visible symptoms are in tools that inspect the query, not in search results.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[f:a, f:b]> but was:<[]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.TestIndriAndQueryVisit.testTermCollectorSeesClauseTerms(TestIndriAndQueryVisit.java:39)

• Javadoc. Query#visit: "Recurse through the query tree, visiting any child queries." QueryVisitor#visitLeaf: "Called by leaf queries that do not match on terms". termCollector: "collects all terms that may match a query". TooManyNestedClauses: "Thrown when a client attempts to execute a Query that has more than getMaxClauseCount() total clauses cumulatively in all of its children."
• CHANGES. LUCENE-3041 (the visitor API): queries should "either pass the visitor down to any child queries, or call a visitX() or consumeX() method on it". LUCENE-8811 (9.0.0): the clause checks "are now implemented using a QueryVisitor and apply to all queries, rather than only booleans".
• The API designer's rule (review of the visitor API, apache/lucene-solr#581, 2019-02-26, @romseygeek): "Yes, all compound queries should pull a sub visitor and call visit on each of their sub queries with it. And visitLeaf is for queries that don't match on terms."
• Tests. TestQueryVisitor.testExtractTermsEquivalent asserts that termCollector reaches terms inside nested BooleanQuery and BoostQuery composites; TestMaxClauseLimit.testLargeDisjunctionMaxQuery asserts that a non-Boolean composite with 1,050 children throws TooManyNestedClauses. TestIndriAndQuery checks hit counts only.
• History. visitLeaf(this) is in the first commit of the introducing PR (apache/lucene-solr#2097, 2020-11-24) and unchanged since on the main line (on branch_8x, which shipped 8.9.0, Query#visit then still defaulted to the same visitLeaf(this)); no review comment, issue comment (#10577, #11194, #463) or commit message mentions it.
• Precedent. LUCENE-8849 ("DocValuesRewriteMethod.visit wasn't visiting its embedded query", 8.6.0) and LUCENE-9050 ("MultiTermIntervalsSource.visit() was not calling back to its visitor", 8.3.1) were fixed as bugs.

The strongest case for "intended": the visitor API lets a composite be opaque, and several are on purpose (ToParentBlockJoinQuery, ToChildBlockJoinQuery, DrillDownQuery, MoreLikeThisQuery and CompositeVerifyQuery call visitLeaf(this); block-join highlighting was dropped by design). ToParentBlockJoinQuery gets past the cumulative limit the same way. RamUsageEstimator.sizeOf is documented as an estimate. And maintainers see Indri as marginal (#11194 proposes moving it to the sandbox).

Why it is weaker than the contract: most precedents have a reason that does not apply here. The block-join children match other documents. DrillDownQuery and MoreLikeThisQuery rewrite to a BooleanQuery, which the clause check (run after rewriting) sees; a visitor applied to the query as given, such as the UnifiedHighlighter's term extraction, does not, so for those consumers they are as opaque as IndriAndQuery. The closest case, CompositeVerifyQuery (spatial-extras), holds one same-document child and calls visitLeaf(this), but it is @lucene.experimental, its child is an approximation that "matches more than needed", and a shape predicate decides the final match, so its child's terms only approximate what it matches. IndriAndQuery is in core, not experimental, has no rewrite, and its clauses are the match: a plain disjunction over the same document. The estimate argument only explains why 1,024 follows from the leaf, which is why the RAM effect is a consequence, not the headline. "Marginal" bears on priority, not on whether it is a bug: the classes are public, not experimental, and shipped since 8.9.0. Verdict: most likely not intended (a defect), or at least a contradiction with the documented visit contract; maintainers could still choose to keep Indri opaque and document it.

One fix was validated on the module test suites; an Occur-based variant was validated on a narrower set.

IndriQuery.visit pulls visitor.getSubVisitor(Occur.SHOULD, this) and visits each clause's query with it. SHOULD describes what the clauses do today: IndriAndQuery matches the union of its clauses whatever their Occur (a separate finding), so a MUST_NOT clause, which still matches, keeps its terms visible. Fixed in the base class, so a future IndriOrQuery (open PR #463) inherits it.

@Override
public void visit(QueryVisitor visitor) {
  // The Indri scorers match the union of all clauses, whatever their Occur, so every clause is
  // visited as a SHOULD clause.
  QueryVisitor v = visitor.getSubVisitor(BooleanClause.Occur.SHOULD, this);
  for (BooleanClause c : clauses) {
    c.query().visit(v);
  }
}

Trade-offs

• Behaviour change: an IndriAndQuery whose clauses take the cumulative count over the limit now throws TooManyNestedClauses, like other composites. The counter checks before it increments, so at the default 1,024 an IndriAndQuery of 1,025 term clauses still runs and one of 1,026 throws (read; the executed check used 6 clauses at a limit of 4). Queries under the limit keep their hits; oversized ones that ran before now fail. Mention it in CHANGES.
• RamUsageEstimator.sizeOf and, on main, the query cache now account for the clauses (a cached 16-clause IndriAndQuery: 3,136 bytes, equal to the BooleanQuery; 500 clauses: 92,192).
• Visitors now also receive callbacks from clauses that need rewriting (for example a PrefixQuery). Searching such a query already fails at createWeight, because IndriAndQuery does not rewrite its clauses (a related finding), but visitor-only consumers (term extraction, highlighting, sizeOf, custom visitors) never call createWeight, so for them these callbacks are new.
• Does not fix the blank highlights in the default weight-matches mode, which needs Weight#matches in IndriAndWeight.
• Not benchmarked (a visit walks the clause list once).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: :lucene:core, every test class of org.apache.lucene.search plus util.TestRamUsageEstimator, with the 3 repro classes: 218 classes, 2,353 tests, 0 failures, with assertions on and off (TestMaxClauseLimit, TestQueryVisitor, TestLRUQueryCache, TestBooleanQuery, TestIndriAndQuery among them).
• Whole :lucene:highlighter with the highlighter repro: 35 classes, 723 tests, 0 failures, assertions on and off.
• :lucene:core:check -x test and :lucene:highlighter:check -x test with the fix and both tests: BUILD SUCCESSFUL.
• The repro runner: the finder's and the reproducer's own fixes (the same code) pass their repros.

Mirrors BooleanQuery.visit: c.query().visit(visitor.getSubVisitor(c.occur(), this)) for every clause.

for (BooleanClause c : clauses) {
  c.query().visit(visitor.getSubVisitor(c.occur(), this));
}

Trade-offs

Right once IndriAndQuery honours (or rejects) MUST, FILTER and MUST_NOT. Today it hides a MUST_NOT clause's terms from termCollector (the default sub-visitor for MUST_NOT is empty) although that clause still produces matches. The clause count includes MUST_NOT clauses either way.

What the checks validated

The 3 repro classes, TestIndriAndQuery, TestQueryVisitor and TestMaxClauseLimit (22 tests) and the highlighter repro pass, assertions on. Not run on the whole package or with assertions off.

Say in the javadoc that term extraction, highlighting and the clause limit do not see Indri clauses.

Trade-offs

A legitimate maintainer choice, but it contradicts Query#visit's contract and leaves the cumulative clause limit not counting the clauses. Not run.

The same loop in the subclass instead of the base class.

Trade-offs

Works for today's one subclass; a future IndriOrQuery would repeat the bug. Not run.

Count getClauses() before searching or set a QueryTimeout; walk getClauses() yourself for term extraction; or wrap the query in your own Query whose visit descends. Such a wrapper must also delegate createWeight, equals, hashCode and toString, and its rewrite must keep the wrapper rather than return the bare IndriAndQuery. Not run.

Trade-offs

Every application has to know about it.

Method: semi-formal (an automated scan, then bounded-exhaustive translation validation of the real core jar against a Z3 model).

• First signal (2026-10-26). An earlier unit scanned the bytecode of every concrete Query class in the stock main jars (JDK ClassFile API) for contract checks; its rows VISIT-MISS and VISIT-NOSUB flagged that IndriAndQuery's clauses are never handed to a visitor. That unit registered the related rewrite bug and recorded this visit root as separate.
• Hunt (2026-10-28). An outside-reviewer probe pack re-raised it (termCollector returns []). The finder built a bounded-exhaustive harness: every clause list over 3 terms and 4 Occur values (1,884 queries), run through the real jar and compared with the same clauses in a BooleanQuery and with a Z3 model of the expected visitor results. termCollector differed in 1,845 lists and the clause-count visitor in 1,860, with 0 mismatches against the model's prediction for the leaf-only visit.
• The finder then wrote a public-API JUnit test (term collection, clause limit, RAM estimate) and a highlighter test, and found the RAM-estimate and query-cache consumers by searching for callers of Query.visit.