CompositeVerifyQuery.visit hides the index query it runs, so term extraction and the nested clause limit miss it

CompositeVerifyQuery (spatial-extras, @lucene.experimental, public constructor) runs an index query as the approximation of every match and verifies each candidate against a shape predicate in TwoPhaseIterator#matches(). Its rewrite rewrites the index query and keeps it in the tree, and its createWeight runs it, so every hit must match it. But visit is visitor.visitLeaf(this); and never visits the index query.

Two consumers go wrong. QueryVisitor.termCollector and other visit-based term extraction (Lucene's UnifiedHighlighter extracts its terms this way) see none of the index query's terms: every release since 8.1.0. IndexSearcher's cumulative clause check, built on Query#visit since 9.0.0, counts the whole wrapper as one clause, so an index query with more clauses than maxClauseCount runs instead of throwing TooManyNestedClauses. No run shows a wrong hit or count, and matching does not use visit. In-tree, CompositeSpatialStrategy loses at most one binary cell token (points-only prefix tree, IsEqualTo or IsWithin on a point) and never changes the clause count. The fix is one line, as in ConstantScoreQuery; it changes admission, since some wrapped queries that run today then throw TooManyNestedClauses.

1. The index query is a real child. CompositeVerifyQuery.java:41 private final Query indexQuery; // approximation (matches more than needed); :44-47 the public constructor accepts any Query.
2. It is rewritten and kept. :49-56: rewrite calls indexQuery.rewrite(indexSearcher); if that returns a different query it returns a new CompositeVerifyQuery around it, otherwise super.rewrite(indexSearcher), the query itself. Either way the index query stays in the tree.
3. It is executed as the approximation. :92-118: createWeight calls indexQuery.createWeight(searcher, ScoreMode.COMPLETE_NO_SCORES, boost) directly (:95-97), takes its scorer per segment (:102; no scorer, no hits) and wraps that iterator in the predicate's TwoPhaseIterator (:107-108). A document the index query does not match is never a hit (CompositeVerifyQuery(TermQuery(body:t2), P), with a predicate all 4 documents satisfy, returns 1 of 4 documents).
4. visit hides it. :87-90: visitor.visitLeaf(this);.
5. Term extraction sees nothing. QueryVisitor.java:91-98: termCollector overrides only consumeTerms; visitLeaf is a no-op by default (:58). UnifiedHighlighter.extractTerms is query.visit(QueryVisitor.termCollector(queryTerms)) (UnifiedHighlighter.java:481-485; read, not run with this class).
6. The clause guard counts one. IndexSearcher.java:871-880: rewrite rewrites to a fixed point and then runs query.visit(getNumClausesCheckVisitor()) (:878); the counter (:896-931) adds one per visitLeaf, consumeTerms or consumeTermsMatching call (it tests the limit before counting, :909-912, so at a limit of 8 the 10th leaf throws: the separate off-by-one LUCENE-9299, #10339, which this bug and its fix do not touch). The wrapper is one visitLeaf call whatever it holds, and the class never sends its index query through IndexSearcher#rewrite, so the searcher's clause check never sees it. (AbstractKnnVectorQuery also does not visit its filter, but rewrites it through indexSearcher.rewrite(...), AbstractKnnVectorQuery.java:93, so its filter is still checked.)
7. What still limits a query. Each BooleanQuery.Builder#add rejects more than maxClauseCount clauses in that one builder (BooleanQuery.java:74-86, "We do the final deep check ... during IndexSearcher.rewrite"). DisjunctionMaxQuery has no build-time check, and nested builders each stay under the limit. A query timeout still applies.

• termCollector over CompositeVerifyQuery(TermQuery(body:a), predicate): []; over the bare TermQuery and over ConstantScoreQuery(TermQuery): [body:a].
• maxClauseCount = 8, index query = DisjunctionMaxQuery of 20 TermQuery clauses: searcher.rewrite(dmq) throws; BooleanQuery(MUST CompositeVerifyQuery(TermQuery(body:t0), predicate), FILTER dmq) throws; searcher.rewrite(new CompositeVerifyQuery(dmq, predicate)) does not. With a BooleanQuery of 4 MUST sub-queries of 5 SHOULD terms each, the wrapped query passes and count() returns the correct 4.
• In-tree: CompositeSpatialStrategy over a points-only prefix tree (6 levels) and a serialized shape field, IsEqualTo or IsWithin on Pt(10, 20), builds CompositeVerifyQuery(rpt:s5x1g8, shapeDocVal(sdv) Equals Pt(x=10.0,y=20.0)) around a TermQuery on a cell token (RecursivePrefixTreeStrategy.java:216-229): 0 terms instead of 1; 1 leaf either way; the 1 hit is correct. Its other index queries (IntersectsPrefixTreeQuery and the like) visit as one leaf themselves, so recursing changes neither their terms nor their clause count.

• Term extraction. QueryVisitor.termCollector, UnifiedHighlighter's term extraction (read) or an application's own visitor (field-based term lists, query logging, cache keys) miss the index query's terms. Every release since 8.1.0. (The classic Highlighter skips every org.apache.lucene.spatial.* query anyway, WeightedSpanTermExtractor.java:266-276, LUCENE-7293.)
• Clause guard. Since 9.0.0, an index query with more leaf clauses than maxClauseCount (1024 by default), in a DisjunctionMaxQuery or spread over nested builders that each stay under the limit, runs instead of throwing TooManyNestedClauses. Per-builder checks and query timeouts still apply.
• In-tree reach is small. CompositeSpatialStrategy reaches only the term half, only with a points-only prefix tree and a non-optimized predicate (IsEqualTo, IsWithin; also Intersects after setOptimizePredicates(false): 0 terms instead of 1, executed in the fact check). The lost term is one binary cell token on the spatial field, so the in-tree highlighting effect is small.
• Default vs opt-in. Default searcher, codec and maxClauseCount suffice; using the class is opt-in. No scale is needed.
• What does not happen. No run shows a wrong hit or count; the score is the wrapper's constant; no exception; no index change. The documents a query matches do not depend on visit. Nobody has reported it since 8.1.0 (2019).

Query#visit needs no reader. The class sits in org.apache.lucene.spatial.composite only to be next to the class under test; it uses no package-private access. The predicate is the smallest valid ShapeValuesPredicate; visit never touches it. The seed only satisfies LuceneTestCase.

The clause-limit effect is a second symptom of the same line. The finder's longer test shows it (20-clause DisjunctionMaxQuery, maxClauseCount = 8; it fails 2 of 2 on 645d4fe14a20 with assertions on and off and on the 10.6.0 jars); it is evidence for the issue text, not part of this repro.

Regression test: this same test; it passes with fix A (and with B or C).

# $REPRO = the directory holding TestCompositeVerifyQueryVisit.java (this bug's repro-minimal/src/org/apache/lucene/spatial/composite/)
cp $REPRO/TestCompositeVerifyQueryVisit.java $LUCENE/lucene/spatial-extras/src/test/org/apache/lucene/spatial/composite/
cd $LUCENE && ./gradlew :lucene:spatial-extras:test --tests TestCompositeVerifyQueryVisit -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
> Task :lucene:spatial-extras:test FAILED

TestCompositeVerifyQueryVisit > testTermCollectorSeesIndexQuery FAILED
    java.lang.AssertionError: expected:<[body:a]> but was:<[]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:987833AD403AD8D3]:0)
        at org.apache.lucene.spatial.composite.TestCompositeVerifyQueryVisit.testTermCollectorSeesIndexQuery(TestCompositeVerifyQueryVisit.java:43)

# the finder's two-symptom test, 645d4fe14a20 (assertions on and off) and 10.6.0 jars (off)
testTermCollectorSeesIndexQueryTerms
  => java.lang.AssertionError: visit must recurse into the executed index query expected:<[body:a]> but was:<[]>
testClauseLimitCountsIndexQueryClauses
  => junit.framework.AssertionFailedError: the 20 executed index-query clauses must count toward maxClauseCount=8

• Contracts. Query.java:89 "Recurse through the query tree, visiting any child queries." QueryVisitor.java:54 visitLeaf is "Called by leaf queries that do not match on terms"; :87 termCollector "collects all terms that may match a query". IndexSearcher.java:1194-1195 TooManyNestedClauses is "Thrown when a client attempts to execute a Query that has more than getMaxClauseCount() total clauses cumulatively in all of its children." CHANGES.txt:4681-4684 (9.0.0, LUCENE-8811): the checks "apply to all queries, rather than only booleans"; MIGRATE.md:1533: "on all types of queries".
• The design rule. The QueryVisitor author, in its review (lucene-solr#581, 2019-02-26): "Yes, all compound queries should pull a sub visitor and call visit on each of their sub queries with it. And visitLeaf is for queries that don't match on terms."
• Peers recurse. ConstantScoreQuery.java:77-79 (FILTER), BoostQuery.java:105-107 (MUST), IndexOrDocValuesQuery.java:141-145 (MUST, both children), FunctionScoreQuery (MUST).
• History. LUCENE-5579 (#6641, 489f56def84, 2015-04-10) created the class with extractTerms(terms) { indexQuery.extractTerms(terms); }. The LUCENE-6425 sweep (#7484, f4c05465673, 2015-04-16) moved extractTerms to Weight and dropped that delegation without a replacement, although its own no-op ConstantScoreWeight.extractTerms says "override if your constant-score query does wrap terms"; both landed before 5.2.0. LUCENE-3041 (fbd05167f45, 2019) then wrote visitLeaf(this); none of the 123 comments on lucene-solr#581 mentions this class.
• Precedent. LUCENE-8849 (#9892, opened by the class's author; 8.6.0 Bug Fixes, CHANGES.txt:5800): "DocValuesRewriteMethod.visit wasn't visiting its embedded query". LUCENE-9050 (#10092, CHANGES.txt:6163): "MultiTermIntervalsSource.visit() was not calling back to its visitor".
• No test or doc says otherwise. No test in lucene/spatial-extras/src/test calls visit or termCollector; no javadoc, CHANGES or MIGRATE entry calls the class opaque.

The strongest case for "intended": the class is one atomic, experimental spatial predicate whose approximation is an implementation detail; the 8.1 CHANGES entry for LUCENE-3041 (CHANGES.txt:6469-6475) lets a query "either pass the visitor down to any child queries, or call a visitX() or consumeX() method on it"; Lucene has deliberately opaque composites (block-join queries, DrillSidewaysQuery); before 8.1 the class's extractTerms reported nothing, so visitLeaf(this) kept the old behaviour; the visitor-based clause count was said in LUCENE-8811 (#9855) to be not exact; and LUCENE-7293 (#8348) made the classic Highlighter skip spatial queries.

Why it fails: the index query is a conjunct every hit must match, not an alternative implementation; the old behaviour came from two mechanical sweeps, while the class was written to delegate and its author later filed the same omission elsewhere as a bug; block-join children match other documents, while this index query matches the same ones, and this class is public, takes any Query and stays itself after rewrite; LUCENE-8811's stated goal covers "booleans wrapped in other queries"; LUCENE-7293 is about one highlighter, predates QueryVisitor and says nothing about visit or the clause limit; @lucene.experimental marks API stability, not an exemption. Verdict: a defect, on a case that is persuasive rather than conclusive: no javadoc, test or upstream discussion says what this class must report to visitors. The fact-check reviewer from another model family (OpenAI GPT-5.6, reading only) rated the intended case "moderate" and the bug as clearing the no-noise bar narrowly; a maintainer could call it low-priority housekeeping of an experimental API.

Option A was run through gradle on 645d4fe14a20; B and C were run with the standalone runner (the patched class compiled in front of the 645d4fe14a20 jars). A is preferred; D only documents the gap.

One line plus an import (+2/-1). FILTER matches how the index query runs: required but not scored (COMPLETE_NO_SCORES), as in ConstantScoreQuery.visit.

@Override
public void visit(QueryVisitor visitor) {
  indexQuery.visit(visitor.getSubVisitor(BooleanClause.Occur.FILTER, this));
}

Trade-offs

• Admission changes (worth the CHANGES line). Some queries that run today start throwing TooManyNestedClauses: with the fix at a limit of 8, the wrapped 20-clause DisjunctionMaxQuery and the 4 MUST x 5 SHOULD shape throw on rewrite and count (executed in the fact check). That is the point of the fix, but applications see it.
• Visitors see the child. The visitor receives the index query's leaves (consumeTerms or visitLeaf) instead of one visitLeaf for the wrapper, so a visitor that recognised CompositeVerifyQuery through visitLeaf now sees it only as the getSubVisitor parent, and the index query's own acceptField gating starts to apply.
• From reading only, not run: UnifiedHighlighter.hasUnrecognizedQuery (UnifiedHighlighter.java:1185-1205) sets its unknown-leaf flag only from visitLeaf, and today the wrapper's ungated visitLeaf(this) sets it whatever the field. With the fix and a TermQuery index query (the repro, the in-tree points-only path) the child reports consumeTerms, so the flag is not set; with a prefix-tree index query (a field-gated visitLeaf) it is set only for the spatial field.
• The shape predicate is no longer reported at all; it reads doc values, not terms. Option C keeps that signal.

What the checks validated

Scratch worktree of 645d4fe14a20, gradle, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off;
• TestCompositeStrategy, TestQueryEqualsHashCode and TestGeo3dRpt, the only tests that use the class or CompositeSpatialStrategy: 3 classes, 6 tests, 0 failures, 1 skipped (nightly);
• the whole :lucene:spatial-extras:test: 26 classes, 117 tests, 0 failures, 5 skipped;
• checkGoogleJavaFormat, ecjLintMain, forbiddenApisMain and renderJavadoc for spatial-extras: pass;
• the finder's two-symptom test passes 2 of 2 with the fix (standalone runner);
• earlier, in the hunt: the model-based harness has 0 of 18 visit mismatches with it (3 without), and TestCompositeStrategy and TestQueryEqualsHashCode pass with both fixes for this class (this one and LFV-947's), 5 iterations.

indexQuery.visit(visitor.getSubVisitor(BooleanClause.Occur.MUST, this));, as IndexOrDocValuesQuery and BoostQuery do.

Trade-offs

Same effect on term extraction and the clause count. MUST tells occur-aware visitors that the child scores, which it does not here.

What the checks validated

The minimal test and the finder's two-symptom test pass (standalone runner, 645d4fe14a20 jars, assertions on); not run through gradle.

Add visitor.visitLeaf(this); after the line of option A, to report the shape predicate, which does not match on terms.

Trade-offs

Keeps today's signal for visitors that detect the class, or any non-term condition, through visitLeaf; costs one extra counted clause per wrapper, and visitLeaf(this) is not gated by acceptField, so field-filtering visitors still see an unknown leaf on every field.

What the checks validated

The minimal test and the finder's two-symptom test pass (standalone runner, 645d4fe14a20 jars, assertions on); not run through gradle.

Say in the javadoc that visit does not descend into the index query.

Trade-offs

At odds with the general Query#visit guidance for a child every hit must match, and leaves the clause guard bypassable through a public constructor. Workaround meanwhile: visit, or IndexSearcher#rewrite, your own index query before wrapping it.

Method: semi-formal (the hunt's tier for model-based testing against an explicit model of the contract).

• Lead. The hunt's planner had an executed lead from an ideas list on committer-stated contracts, written with an independent reviewer from another model family (OpenAI GPT-5.6), and a probe of it, which showed termCollector finding 20 terms for a bare index query and 0 for the wrapped one, and the bare query rejected by the clause limit while the wrapped one ran. The hunt unit re-ran the probe first.
• Model-based differential harness. For 9 kinds of index query I, it compared QueryVisitor.termCollector and the clause counter's leaf count of CompositeVerifyQuery(I, P) with those of I: 18 checks, 3 mismatches (the TermQuery and the 2-clause BooleanQuery index queries lose their terms, and the BooleanQuery counts 1 leaf instead of 2), 0 with the fix; the same on 10.5.2.
• JUnit. Reduced to a public-API test with two methods (terms; clause limit), which fails 2 of 2 with assertions on and off and 20 of 20 over 10 random seeds; later minimized to the one-method test above.