Vector similarity queries never visit their filter, so its clauses escape the cumulative clause limit

FloatVectorSimilarityQuery, ByteVectorSimilarityQuery and, on the unreleased 11.x line (main and branch_11x), Float16VectorSimilarityQuery (all @lucene.experimental) take an optional filter and execute it in createWeight, but the visit() they inherit from AbstractVectorSimilarityQuery only calls visitor.visitLeaf(this). IndexSearcher's cumulative clause check walks the rewritten query with a visitor, so it never counts the filter's clauses; the filter is checked only on its own, when createWeight rewrites it. A query of 600 term clauses plus a similarity query with a 600-term filter runs at the default maxClauseCount of 1,024, while the same 1,200 clauses with the filter as a sibling FILTER clause are rejected; ten similarity queries with 1,000-term filters run too. QueryVisitor.termCollector and RamUsageEstimator.sizeOf (on the 11.x line also the query cache's key estimate) miss the filter for the same reason. Search hits are correct. The gap is as old as the API: every release from 9.10.0 to 10.6.0 and main. A 3-line fix visits the filter through a FILTER sub-visitor.

// :145-152 createWeight: the filter is executed, after a rewrite (and so a clause check) of its own
final Weight filterWeight =
    filter == null
        ? null
        : searcher.createWeight(searcher.rewrite(filter), ScoreMode.COMPLETE_NO_SCORES, 1);

// :259-264 visit: the whole query reports one leaf; the filter is never visited
public void visit(QueryVisitor visitor) {
  if (visitor.acceptField(field)) {
    visitor.visitLeaf(this);
  }
}

1. The check. IndexSearcher.search rewrites the query first (IndexSearcher.java:750-754). rewrite (:871-880) rewrites until stable and then calls query.visit(getNumClausesCheckVisitor()) (:878). That visitor (:891-932) returns itself from getSubVisitor and counts one per visitLeaf, consumeTerms or consumeTermsMatching call. Each call first throws TooManyNestedClauses if the count is already above maxClauseCount, then adds one (:907-929), so at the default 1,024 the 1,026th counted unit throws. It counts visitor callbacks, not terms as such; for the repro's TermQuery disjunctions that is one per term.
2. The filter stays in the tree. Neither AbstractVectorSimilarityQuery nor its subclasses override rewrite, so the rewritten query still holds the filter (protected final Query filter, :57). It is skipped only because visit does not descend.
3. The filter's own check. searcher.rewrite(filter) in createWeight starts a new counting visitor, so a filter over the limit on its own still throws, but its clauses are never added to the rest of the query.
4. Counted, executed probe. For the repro's query a replica of the check's visitor counts 601 (600 terms + 1 leaf); with the 600 filter terms as a sibling FILTER clause it counts 1,201; the filter alone counts 600. With the fix the repro's query counts 1,201 too. An independent fact check found the same counts on the rewritten tree and through IndexSearcher.rewrite's real acceptance threshold.
5. Other visitors. QueryVisitor.termCollector (QueryVisitor.java:86-98) collects only from consumeTerms and ignores visitLeaf (:53-58): 0 of the filter's 600 terms. RamUsageEstimator.sizeOf walks visit and charges a lone leaf the default 1,024 bytes: 1,024 instead of 701,824. On the unreleased 11.x line only (main and branch_11x), LRUQueryCache sizes cached query keys this way, with sizeOf(query, 32) (LRUQueryCache.java:465-471, #15124): 32 bytes for the similarity query, 105,632 for its filter alone (executed); released 10.x charges such queries a fixed default. Caching is off by default.
6. For comparison. ConstantScoreQuery.visit passes its query a FILTER sub-visitor (ConstantScoreQuery.java:76-79), and BooleanQuery visits FILTER clauses, so the same filter in either is counted and collected.

• Who. Users of FloatVectorSimilarityQuery, ByteVectorSimilarityQuery or the unreleased 11.x line's Float16VectorSimilarityQuery with a filter. No Lucene query parser builds them; the application calls the constructor. Downstream (read on default branches, not run): Solr's {!vectorSimilarity} parser passes the request's pre-filter or filter queries as this filter, and the OpenSearch k-NN plugin uses FloatVectorSimilarityQuery for radial search; Elasticsearch has its own similarity query.
• Default vs opt-in. Opt-in: the @lucene.experimental similarity query with a filter. Default: codec, IndexSearcher, maxClauseCount 1,024.
• What it takes. The total must pass the limit only through filter clauses, each filter staying within the limit on its own (600 + 600 in the repro). One filter above the limit is still rejected. Large id lists usually go into a TermInSetQuery, which counts as one clause, so real queries rarely get there (likelihood: rare).
• Effect. A query of N filtered similarity queries can carry about N x maxClauseCount clauses (executed: 10 and 50 queries with 1,000-term filters accepted). Visitor-based tools under-report the query: no filter terms, a 1,024-byte size estimate, and on the 11.x line a query-cache key charge that ignores the filter (32 bytes). No error, no wrong hits, no index effect; the work stays proportional to the query the application built.

Public API plus LuceneTestCase, in package org.apache.lucene.search only to sit next to Lucene's own tests. new IndexSearcher rather than newSearcher keeps it deterministic, and an empty MultiReader mirrors TestMaxClauseLimit. Each part (600) is under the limit and the sum (1,200) far above it, so the counter's boundary rule plays no part.

It compiles against 10.5.2 and later, where FloatVectorSimilarityQuery.Adaptive exists. For 9.10.0 to 10.5.1 use the old constructor, new FloatVectorSimilarityQuery("v", new float[] {1, 0}, 0f, 0f, filter); the result is the same.

Regression test: this same test; it passes with the recommended fix and fits as one more method of TestFloatVectorSimilarityQuery or TestMaxClauseLimit. Formatting: google-java-format wraps its one long comment line, nothing else.

cp TestVectorSimilarityQueryFilterClauseCount.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE
./gradlew :lucene:core:test --tests TestVectorSimilarityQueryFilterClauseCount -Ptests.seed=DEADBEEF
# assertions off (production-like): add -Ptests.asserts=false

# gradle, stock main 645d4fe14a20, assertions on, seed DEADBEEF (stack frames and environment lines dropped)
TestVectorSimilarityQueryFilterClauseCount > testFilterClausesCountTowardsMaxClauseCount FAILED
    junit.framework.AssertionFailedError: Expected exception TooManyNestedClauses but no exception was thrown
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:2DC7ECFDF571A694]:0)
:lucene:core:test (FAILURE): 1 test, 1  failure

# the same with assertions off: 1 test, 1 failure; with the fix: 1 test, 0 failures (on and off)
# the repro runner: fails on main (on, off), on the 10.6.0 and 10.5.2 jars (off)

# supporting probe (not the repro), stock main vs the fix:
                                                         stock      with the fix
clause-check count, BQ(MUST a600, MUST simQ(filter=b600))   601        1201
clause-check count, BQ(MUST a600, FILTER b600, MUST simQ)   1201       1201
clause-check count, the filter b600 alone                   600        600
termCollector over simQ(filter=b600)                        0 terms    600 terms
RamUsageEstimator.sizeOf(simQ(filter=b600))                 1024       701824

• The contract. Query.java:88-93: "Recurse through the query tree, visiting any child queries." QueryVisitor.java:53-58: visitLeaf is "Called by leaf queries that do not match on terms". IndexSearcher.java:1193-1199: TooManyNestedClauses is "Thrown when a client attempts to execute a Query that has more than getMaxClauseCount() total clauses cumulatively in all of its children"; the check's javadoc (:891-895) speaks of "the total number of clauses that a query and its children cumulatively have".
• The design. CHANGES LUCENE-3041 (lucene/CHANGES.txt:6469-6473): queries should "either pass the visitor down to any child queries, or call a visitX() or consumeX() method on it". LUCENE-8811 (:4681-4684): the clause checks "are now implemented using a QueryVisitor and apply to all queries"; MIGRATE.md:1533: "max clause count checks on all types of queries". On #9855 (LUCENE-8811) Adrien Grand wrote that it "removes ways to by-pass the maximum clause count by splitting boolean queries into multiple sub boolean queries".
• Precedent. LUCENE-8849 "DocValuesRewriteMethod.visit wasn't visiting its embedded query" (CHANGES.txt:5800; its fix also chose a FILTER sub-visitor, but never passes it on to the wrapped query, DocValuesRewriteMethod.java:73-77, which is the registry's LFV-038) and LUCENE-9050 (:6163-6164) were treated as bugs. On the open #14756, Luca Cavanna wrote "for the purpose of visiting, we should visit all variants".
• No recorded intent. The leaf-only visit was copied from AbstractKnnVectorQuery in #12679; no comment or review there mentions visit, QueryVisitor or maxClauseCount. No test covers it; TestMaxClauseLimit.testLargeDisjunctionMaxQuery asserts the nested walk for another composite.
• How strong the case is. Persuasive, not conclusive: LUCENE-3041 lets a query "call a visitX() or consumeX() method" instead of descending, so a leaf callback is allowed as such, and Query#visit does not define whether a filter held by a query is one of its children. What no contract allows for is an executed child that no visitor sees; maintainers may still consider vector queries opaque by design.

The strongest case (the intended-behaviour verifier, weighed by the adjudication): every Lucene vector query is an opaque visitor leaf; ToParentBlockJoinQuery has a leaf visit and, when no child scores are needed, an inner rewrite of its child with its own budget (ToParentBlockJoinQuery.java:90-92, 98-108; with child scores no separate check runs, :109-121), and that branch is accepted the same way; the filter is still checked on its own; and maintainers called the limit approximate (Alan Woodward on LUCENE-8811: "it wouldn't be exact").

Why it fails: the kNN queries use up their filter inside rewrite (AbstractKnnVectorQuery.java:74-97), so no visit could expose it to the check; this query keeps the filter in the checked tree. Block-join children match other documents, which is why they stay out of term extraction; this filter matches the same documents, like a FILTER clause. "Approximate" was said about counting rules, not about skipping executed sub-queries, and nothing documents the exclusion. Maintainers could still choose to keep vector queries opaque and document it; that is why the adjudication's confidence is medium.

Option A was built and run through gradle. B (rewrite the filter in place) needs the same visit change and more code; C (document the gap) leaves the contract broken. Workarounds that need no fix: count the filter's clauses yourself before searching, keep large id filters in one TermInSetQuery, or set a QueryTimeout. Not part of this fix: the kNN queries' visit, which cannot help their clause check because they consume the filter in rewrite.

After the leaf call, if (filter != null) filter.visit(visitor.getSubVisitor(BooleanClause.Occur.FILTER, this)); in the shared base class, so Float, Byte and Float16 all get it. FILTER because the filter restricts matching without scoring, like a FILTER clause; it is visited whatever acceptField(field) says about the vector field, since its own leaves check their own fields. LUCENE-8849 also chose a FILTER sub-visitor (its fix never passes it on, see LFV-038).

if (visitor.acceptField(field)) {
  visitor.visitLeaf(this);
}
if (filter != null) {
  filter.visit(visitor.getSubVisitor(BooleanClause.Occur.FILTER, this));
}

Trade-offs

• Behaviour change. A query whose filters take the cumulative count over the limit now throws TooManyNestedClauses, as other composites do. The code change is limited to visit, so matching is expected to stay the same and queries under the limit keep their hits; visitor consumers and query-cache accounting change (next point).
• Visitors see the filter. Term collectors and term-extracting highlighters now get the filter's terms (600 instead of 0 in the probe), as they do for a FILTER clause; RamUsageEstimator.sizeOf gives 701,824 bytes instead of 1,024, and on the 11.x line the query cache's sizeOf(query, 32) gives 105,632 instead of 32 (executed; no cache run, so admission and eviction were not observed).
• The similarity queries become stricter than the kNN queries, whose filter keeps a separate budget because rewrite consumes it.
• The filter is still also rewritten and checked on its own in createWeight (a cheap second walk, left unchanged). Not benchmarked.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF): stock, the minimal test fails with assertions on and off; with the fix it passes on and off.
• The closest suites (8 classes: the Float, Byte, Float16 and Explicit similarity-query tests, TestVectorSimilarityCollector, TestMaxClauseLimit, TestQueryVisitor), assertions on and off: 8 classes, 97 tests, 0 failures (3 skipped).
• Every test class of org.apache.lucene.search in :lucene:core, assertions on: 215 classes, 2,340 tests, 0 failures (49 skipped).
• :lucene:core ecjLint and forbiddenApis (main and test) with the fix and the test: pass.
• google-java-format passes on the fix and asks only to wrap the test's one long comment line; with that line wrapped it passes too.
• Earlier: the finder's 2 tests and the 34 tests of the Float and Byte similarity-query suites pass with it; the independent reproducer's own identical fix passes its 7 tests; on 9.10.0 the fix passes the old-API variant.

Override rewrite to return a new similarity query with the rewritten filter, and let createWeight use it without its own searcher.rewrite.

Trade-offs

Removes the second clause walk and makes the post-rewrite tree honest, but still needs the visit change, and touches equals/hashCode (query-cache keys) and every subclass. More change for no extra correctness. Not run.

Say in the class javadoc that the filter's clauses are checked only on their own and that visitors do not see the filter.

Trade-offs

A legitimate choice for consistency with the kNN queries, but it leaves the contradiction with Query#visit and the cumulative limit, and visitor tools still miss the filter. Not run.

Method: semi-formal (the hunt's tier for bounded-exhaustive checks of the real jar against a reference model). The hunt unit's lens was queries that execute an embedded query but hide it from QueryVisitor.

• Differential check. HiddenSubqueryTV built every combination of 16 wrapper sites x primary leaves {1, 2, 4} x hidden leaves {1..10} x copies {1, 2, 3} (1,440 checks at maxClauseCount 8) and compared IndexSearcher's verdict with a model that counts the clauses each query executes, with assertions on and off. The site BQ(MUST P, MUST FloatVectorSimilarityQuery(filter = H)) was under-counted in 35 of its 90 checks. The BooleanQuery, ConstantScoreQuery and DisjunctionMaxQuery controls had 0 mismatches in 270, and the model also caught the known IndexOrDocValuesQuery over-count, so it is not vacuous.
• Bytecode scan. A JDK ClassFile scan over 142 concrete Query classes in 17 module jars flagged AbstractVectorSimilarityQuery.filter as a field that is executed but never visited (an earlier unit's scan had the same row, triaged there for matching only).
• Reduction. The finding was reduced to a public-API JUnit (float and byte), later minimized to the one-test repro above. Not claimed: the kNN queries and RescoreTopNQuery show the same count gap in the harness, but consume their sub-query in rewrite, so no visit change could help there.