IndexSearcher's clause limit never counts the query inside a function query's values source

IndexSearcher is documented to reject a query with more than maxClauseCount clauses (1,024 by default) "cumulatively in all of its children". It counts them once, after rewrite, by walking the query with a QueryVisitor. Some function queries hold a second query inside their values source: the boost query of FunctionScoreQuery.boostByQuery (the replacement for the removed BoostingQuery), any DoubleValuesSource.fromQuery, and QueryValueSource. Their visit() stops at the values source, which has no visit method, so those clauses are never added to the total. The embedded query still runs: it is rewritten when the weight is created, through searcher.rewrite, which checks it alone with a fresh count. So boostByQuery over two 600-term disjunctions runs 1,200 term clauses at the default limit, while the same clauses in a BooleanQuery are rejected, and repeating or nesting such queries multiplies the total without bound. Hits and scores are not affected; every release since 9.0.0 (LUCENE-8811) is affected. A counting-only visitor channel fixes it without changing what term extraction, highlighting or Monitor see.

1. One check, after rewrite. IndexSearcher.search(Query, int) (IndexSearcher.java:623) goes through searchAfter (:587, :601) to search(Query, CollectorManager) (:750), which rewrites the query (:753) through rewrite(Query) (:871-880, reached via :884); it ends with query.visit(getNumClausesCheckVisitor()) (:878). The visitor (:896-932) returns itself for every occur, "even for MUST_NOT" (:902-905), and counts every visitLeaf, consumeTerms and consumeTermsMatching call. Each call tests numClauses > maxClauseCount before it increments, so one walk accepts 1,025 clauses at the default limit and throws at the 1,026th.
2. The walk stops at the values source. FunctionScoreQuery.visit is in.visit(visitor.getSubVisitor(MUST, this)) (FunctionScoreQuery.java:133-136); FunctionMatchQuery.visit (:80-83), FunctionRangeQuery.visit (:132-135) and FunctionQuery.visit (:92-95) are visitor.visitLeaf(this). DoubleValuesSource and ValueSource have no visit method.
3. Where the boost query lives. boostByQuery (FunctionScoreQuery.java:101-106) builds the source MultiplicativeBoostValuesSource(QueryBoostValuesSource(DoubleValuesSource.fromQuery(boostMatch), boost)).
4. It is checked later, alone. For a scoring search FunctionScoreQuery.createWeight calls source.rewrite(searcher) (:121), which reaches QueryDoubleValuesSource.rewrite (DoubleValuesSource.java:776-780) and searcher.rewrite(query) (:779): step 1 again, on the boost query alone, with a new count that starts at 0. FunctionMatchQuery.createWeight does the same (:88), and QueryValueSource.createWeight calls searcher.rewrite(q) (QueryValueSource.java:83) for FunctionRangeQuery (:149) and FunctionQuery (:71). A non-scoring search skips the boost query (FunctionScoreQuery.java:118-120).

boostByQuery(600, 600): the outer check counts 600, the boost query's own check 600; both pass and the hit scores 0.329 instead of 0.164, so the boost query ran. BooleanQuery(MUST 600, SHOULD 600): one check, 1,200, rejected. 32 and 100 boostByQuery clauses with 1,000-clause boost queries run 32,032 and 100,100 term clauses. With maxClauseCount 4, nesting boostByQuery 1, 2 and 3 levels deep builds 20, 84 and 340 leaf queries, and on main and 10.5.2 pulls a scorer for each (16, 64 and 256 on 9.12.3); each level gets a fresh budget, so the total grows exponentially with depth. An embedded query that is over the limit by itself is still rejected.

• Who. Users of FunctionScoreQuery.boostByQuery (the documented replacement for BoostingQuery, MIGRATE.md), of FunctionScoreQuery or FunctionMatchQuery over DoubleValuesSource.fromQuery, and of the @lucene.experimental FunctionQuery and FunctionRangeQuery over QueryValueSource, with the default IndexSearcher and limit.
• What goes wrong. A query the documented cumulative check should reject is accepted. Nothing else: hits and scores are not affected, each embedded query is still checked on its own, and each BooleanQuery is still capped when it is built.
• Matching role. FunctionMatchQuery and FunctionRangeQuery always run the embedded query, and it decides which documents match. FunctionScoreQuery runs its boost query only on scoring searches.
• Reach. No Lucene query parser builds these queries. Downstream function-query syntaxes can (for example a query() function that builds a QueryValueSource).
• Likelihood. Rare: it needs a large embedded query, and the only visible effect is a missing exception. No earlier report was found (see Prior reports).

ByteBuffersDirectory, default IndexWriterConfig, one document with f:a0 and f:b0 so that both disjunctions match and the boost query really scores. Each part alone (600) is under the limit and the total (1,200) is far above it, so the test does not sit at the limit's boundary. It sits in org.apache.lucene.queries.function, like Lucene's own function-query tests; it fails with seeds DEADBEEF and 1234ABCD.

Release jars: the test fails on 9.12.3, 10.5.2 and 10.6.0 (assertions off). It needs org.apache.lucene.tests.util.LuceneTestCase, which exists from 9.1.0; on 9.0.0 a variant with the old import fails the same way.

Regression test: this same test; it passes with the recommended fix and fits as one more method of the existing TestFunctionScoreQuery.

cp $REPRO/TestBoostByQueryClauseLimit.java $LUCENE/lucene/queries/src/test/org/apache/lucene/queries/function/
cd $LUCENE && ./gradlew :lucene:queries:test --tests TestBoostByQueryClauseLimit -Ptests.seed=DEADBEEF -Ptests.asserts=false
# the same with -Ptests.asserts=true

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed)
TestBoostByQueryClauseLimit > testBoostQueryCountsTowardsMaxClauseCount FAILED
    junit.framework.AssertionFailedError: Expected exception TooManyNestedClauses but no exception was thrown
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:9AEB3E5629C00982]:0)
        at org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1432)
        at org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
        at org.apache.lucene.queries.function.TestBoostByQueryClauseLimit.testBoostQueryCountsTowardsMaxClauseCount(TestBoostByQueryClauseLimit.java:56)
1 test completed, 1 failed

# the same failure with -Ptests.asserts=false

# with the recommended fix (both settings)
1 test, 0 failures (TooManyNestedClauses is thrown)

• Contract. IndexSearcher.java:1193-1199: "Thrown when a client attempts to execute a Query that has more than getMaxClauseCount() total clauses cumulatively in all of its children". CHANGES.txt:4681-4684 (9.0.0, LUCENE-8811): "The checks are now implemented using a QueryVisitor and apply to all queries, rather than only booleans." MIGRATE.md:1531-1534: "IndexSearcher now performs max clause count checks on all types of queries". BooleanQuery.java:75-76: "We do the final deep check for max clauses count limit during IndexSearcher.rewrite". Query.java:88-93: visit should "Recurse through the query tree, visiting any child queries." The boost query is part of the query's equals, hashCode and toString.
• Intent. On LUCENE-8811 (#9855) jpountz wrote "I like that it removes ways to by-pass the maximum clause count by splitting boolean queries into multiple sub boolean queries" and that the limit is "now actually the maximum number of clauses across the entire query"; romseygeek: "we also want to handle nested booleans, booleans wrapped in other queries, etc, in which case a QueryVisitor makes most sense".
• Role does not matter. The check visitor descends into every occur, MUST_NOT included (IndexSearcher.java:903); executed controls BooleanQuery(MUST 768, SHOULD 768) and BooleanQuery(MUST 768, MUST_NOT 768) are both rejected.
• History. BoostingQuery, which boostByQuery replaced, rewrote its context query as part of the tree; the query became hidden when it moved behind the values source.
• Precedent. The same kind of gap was fixed as a bug elsewhere: LUCENE-8849 "DocValuesRewriteMethod.visit wasn't visiting its embedded query" (8.6.0) and LUCENE-9050 (8.3.1). We found no javadoc, test, CHANGES entry or thread that excludes values-source queries; TestMaxClauseLimit covers only Boolean, DisjunctionMax and phrase shapes.

The strongest case (the intended-behaviour verifier): the four visit() bodies are a deliberate matching-only view, ported from Weight#extractTerms by LUCENE-3041, which already skipped the values source; LUCENE-8099 made the highlighters extract only getWrappedQuery(); on #12938 romseygeek wrote "FunctionQuery doesn't actually match things, it just scores, so it shouldn't be used for highlighting"; LUCENE-4261 (#5331, 2012) was closed as not a bug because value sources are opaque to the Scorer API; setMaxClauseCount says "per Query"; each embedded query is still checked alone; and visiting the source as ordinary clauses made Monitor miss two stored queries.

Why it fails: every item concerns term extraction, highlighting or Scorer navigation, and each predates or is independent of the cumulative check; none says values-source queries are outside the clause limit. "Per Query" was written during LUCENE-8811 to mean the entire query. FunctionMatchQuery and FunctionRangeQuery decide matching from the embedded query, so "it just scores" does not cover them. And the matching view need not change: the counting-only fix keeps termCollector and Monitor as they are. Still, the behaviour is inconsistent with the cumulative-limit contract rather than an explicit violation: neither Query.visit nor TooManyNestedClauses says outright that a query reachable only through a values source is a child, and maintainers may regard values sources as a deliberately opaque boundary or prefer a javadoc clarification, so the confidence is medium.

One option was built and validated; the others come from the hunt's verifiers. The requirement (adjudication, intended-behaviour verifier, tie-breaker): count the embedded query in the clause check without changing what matching-oriented visitors (termCollector, Monitor's query analyzer, the highlighters) see.

QueryVisitor gets getValuesSourceVisitor(Query parent), returning EMPTY_VISITOR by default; the clause-check visitor in IndexSearcher.getNumClausesCheckVisitor() overrides it to return itself. DoubleValuesSource and ValueSource get a no-op visit(QueryVisitor) hook; DoubleValuesSource.fromQuery's source and QueryValueSource forward it to their query, and the two sources boostByQuery builds and the two ValueSource/DoubleValuesSource bridges forward it to the source they wrap. FunctionScoreQuery, FunctionMatchQuery, FunctionRangeQuery and FunctionQuery add source.visit(visitor.getValuesSourceVisitor(this)) to visit (9 files, +74 lines, no deletions). The method name is a placeholder: ask maintainers which channel they prefer.

// QueryVisitor: a channel that only accounting visitors follow
public QueryVisitor getValuesSourceVisitor(Query parent) {
  return EMPTY_VISITOR;
}
// IndexSearcher.getNumClausesCheckVisitor(): count these clauses too
@Override
public QueryVisitor getValuesSourceVisitor(Query parent) {
  return this;
}
// DoubleValuesSource and ValueSource: a no-op hook; query-holding sources forward it
public void visit(QueryVisitor visitor) {}
// FunctionScoreQuery.visit (likewise FunctionMatchQuery, FunctionRangeQuery, FunctionQuery)
in.visit(visitor.getSubVisitor(BooleanClause.Occur.MUST, this));
source.visit(visitor.getValuesSourceVisitor(this));

Trade-offs

• API addition on three public abstract classes (QueryVisitor, DoubleValuesSource, ValueSource); every default is a no-op, so existing subclasses compile and behave as before.
• Composite sources stay hidden until they forward the hook: ValueSource functions that wrap other sources (sum, product, if, ...), expression bindings in lucene/expressions, custom sources. The patch covers every source that holds a query directly and the wrappers boostByQuery uses; the rest is a follow-up.
• Non-scoring searches now count the boost query too, although FunctionScoreQuery does not run it there: an over-count in the same direction as IndexOrDocValuesQuery today (#14756).
• Nothing is relaxed: each embedded query is still also checked on its own when it is rewritten.
• Behaviour change: queries accepted today whose total, counted with their embedded queries, is over the limit start to throw TooManyNestedClauses. That is the point of the fix, but callers who rely on today's acceptance need a CHANGES/MIGRATE note; with the API addition this makes option A a design and compatibility question for maintainers, not an obvious one-line bug fix.
• Optional: RamUsageEstimator's query visitor (query-cache sizing) could follow the same channel; not done, not part of this report.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-31; the worktree was removed afterwards):

• stock: the minimal test fails with assertions on and off; with the fix it passes with assertions on and off;
• closest suites: org.apache.lucene.queries.function.* (14 classes, 111 tests, 0 failures) and core TestMaxClauseLimit, TestQueryVisitor, TestDoubleValuesSource, TestIndexSearcher, TestBooleanQuery (62 tests, 0 failures, 1 skipped); all of core org.apache.lucene.search.* (214 classes, 2,339 tests, 0 failures, 49 skipped);
• whole modules: queries (512 tests, 0 failures, 5 skipped), highlighter (722 tests), monitor (181 tests, 1 skipped), expressions (170 tests), all 0 failures;
• matching views unchanged: the intended-behaviour verifier's Monitor and termCollector probe passes 4 of 4 with the fix, as on stock (Monitor still matches the stored boostByQuery and FunctionQuery(QueryValueSource) queries; termCollector still returns only [f:a]);
• with the fix shadowed in the standalone runner (assertions off): the finder's 5 tests and the tie-breaker's 7 (including nesting) pass; an independent reproducer's 10 tests have 1 failure, its own termCollector assertion, which this fix deliberately leaves unchanged;
• checkGoogleJavaFormat, ecjLintMain, forbiddenApisMain and renderJavadoc pass for core and queries.

Forward the hook through getSubVisitor(SHOULD or FILTER, this), as the finder's first fix did.

Trade-offs

It makes the repro pass but changes what matching tools see: Monitor stops matching a stored boostByQuery(f:a, f:<rare>) and a stored FunctionQuery(QueryValueSource(...)) for a document with only f:a, and termCollector returns [f:a, f:b] instead of [f:a] (3 of 4 probe tests fail). That undoes the deliberate matching-only view (LUCENE-8099, #12938).

The tie-breaker's experiment: no new QueryVisitor method, because the clause check already descends into MUST_NOT and matching visitors do not. It passes the repros, the Monitor probe, 88 function-query and 70 highlighter tests.

Trade-offs

It labels a scoring query as prohibited: a visitor that also collects prohibited terms (the highlighter's QueryTermExtractor.getTerms(query, true)) would start returning the boost query's terms as if they came from a MUST_NOT clause. It only shows that a counting-only path exists.

Say in setMaxClauseCount and TooManyNestedClauses that queries held by values sources are checked on their own, not in the cumulative total.

Trade-offs

Cheapest, no API change, but it leaves the bypass, including the exponential growth with nesting, and goes against LUCENE-8811's aim of no bypass by splitting clauses.

Method: semi-formal (the hunt's tier for bounded-exhaustive checks against a reference model). The unit's lens was queries that execute an embedded query but hide it from QueryVisitor.

• Differential check. On the stock main jars, 16 wrapper sites x primary leaves {1, 2, 4} x hidden leaves {1..10} x copies {1, 2, 3} = 1,440 checks at maxClauseCount 8 compared the limit's decision with a count of the clauses that really execute, with assertions on and off. boostByQuery, FunctionScoreQuery(fromQuery), FunctionMatchQuery(fromQuery), FunctionQuery(QueryValueSource) and FunctionRangeQuery(QueryValueSource) under-counted in 44, 44, 35, 35 and 35 of their 90 checks; the BooleanQuery, ConstantScoreQuery and DisjunctionMaxQuery controls had 0 mismatches in 270, and the oracle caught the known IndexOrDocValuesQuery over-count (52), so it is not vacuous.
• Bytecode scan. The JDK ClassFile API listed query- and values-source-typed fields that visit() never reads (64 such fields in 142 query classes). The four function-query fields are among them, each read when the weight is built: FunctionScoreQuery.source and FunctionMatchQuery.source by createWeight, FunctionQuery.func and FunctionRangeQuery.valueSource by their weight constructors.
• JUnit. The mismatch was reduced to a public-API test against a BooleanQuery control, then minimized to the one-test repro above.