With distributed statistics, SynonymQuery, CombinedFieldQuery, MultiPhraseQuery and SpanOrQuery leave out every term the local shard does not have

IndexSearcher#termStats and #fieldStats "can be overridden for example, to return a term's [field's] statistics across a distributed collection". For a query over a fixed set of terms, an application that searches one shard with such overrides expects each document to get the score the same query gives it in one index holding the whole collection (the javadoc does not promise this for every query), and TermQuery and PhraseQuery deliver that. SynonymQuery, MultiPhraseQuery, CombinedFieldQuery and span queries (through SpanWeight.buildSimWeight, so SpanOrQuery and enclosing span queries that score through it, such as spanNear over spanOr) instead build TermStates on the local reader and call the hook only if (ts.docFreq() > 0). A query term that occurs only on other shards is not offered to the override, so its collection statistics are silently left out of the similarity's input. In the repro, SynonymQuery(quick, fast) on a shard without "fast" scores doc A 0.5733204 instead of 0.16984521, and a merged top-k puts A first although it ties for second in the whole collection. SynonymQuery and CombinedFieldQuery over-score (under BM25 only when the missing term is the more frequent one), MultiPhraseQuery and span-or under-score when a term is missing locally but present elsewhere and a document still matches through another alternative. Matching never changes. The checks came with LUCENE-8921 (8.3.0), an API cleanup that replaced the callers' "the hook returned null" test with "the local docFreq is 0"; in 8.0-8.2 the hook was asked about every term and the shard scores were right (executed).

• Distributed statistics: two-phase scoring of a sharded collection. The application first gathers each query term's docFreq and totalTermFreq and each field's statistics from all shards, then searches every shard with an IndexSearcher subclass whose termStats(Term, int docFreq, long totalTermFreq) and fieldStats(String) return the collection-wide values. In-tree, the test mock ShardSearchingTestBase does this ("Matches docs in the local shard but scores based on aggregated stats").
• Combining queries: SynonymQuery and CombinedFieldQuery score their terms as one pseudo-term whose docFreq is the largest docFreq of the terms; MultiPhraseQuery and span queries pass every term's statistics to the similarity, which for BM25 sums their idf.

1. The weight builds TermStates for each term on the local reader (SynonymQuery.java:215, MultiPhraseQuery.java:224, CombinedFieldQuery.java:279, span queries when the weight is created).
2. It calls searcher.termStats(...) only if (ts.docFreq() > 0): SynonymQuery.java:217, MultiPhraseQuery.java:227, CombinedFieldQuery.java:281, queries/spans/SpanWeight.java:109.
3. A term that occurs only on other shards has local docFreq 0, so the override is never asked about it, and the ordinary weight path has no other per-term statistics callback (overriding IndexSearcher#createWeight, or expert constructors with caller-built TermStates, are heavier workarounds). Its collection statistics are missing from the max (SynonymQuery, CombinedFieldQuery) or from the idf sum (MultiPhraseQuery, span-or).
4. TermQuery (TermQuery.java:65) and PhraseQuery (:479) have the same check but cannot go wrong: a term missing from the shard cannot make a local document match.

Shard 0: A "quick fox" (title "red"). Shard 1: B "fast dog", C "fast cat", D "fast fast bird" (titles "fox", "fox", "bird"). Whole collection: N = 4, docFreq quick 1, fast 3, body:fox 1, title:fox 2.

Under BM25 SynonymQuery and CombinedFieldQuery change only when the missing term has the larger collection docFreq (BM25 ignores totalTermFreq; with equal docFreqs the independent reproducer saw equal scores). MultiPhraseQuery and span-or change when a term is missing locally but present elsewhere and a local document still matches through another alternative. Shard 1, which lacks quick, is wrong for span-or too: B and C score 0.16984521 instead of 0.74316555, D 0.20381427 instead of 0.89179873. For SynonymQuery and CombinedFieldQuery shard 1 is right, because the term it lacks has the smaller collection docFreq.

• Who. Applications that subclass IndexSearcher and override the @lucene.experimental hooks termStats/fieldStats with collection-wide values. In-tree the only override of both hooks with collection-wide values is the test mock ShardSearchingTestBase; the core and sandbox TestCombinedFieldQuery also subclass IndexSearcher but override only fieldStats, with synthetic other-shard values. Search servers' distributed-statistics (global IDF) modes answer through this hook, and the 2019 change moved Solr's distributed-IDF overrides to the new signature; that downstream reach comes from reading code, nothing downstream was run.
• Opt-in. Without an override the default hook returns local statistics, and a locally missing term contributes nothing either way, so single-index search is correct.
• Trigger. Ordinary data once the override exists: any query-time synonym, phrase alternative, span-or clause or CombinedFieldQuery field term that is missing from the shard being searched but present on another. No scale is needed; small, routed or time-partitioned shards and rare terms make it more frequent.
• Effect. Wrong scores on every shard that lacks one of the terms, and with them possibly a wrong merged ranking: in the repro a document that ties for second in the whole collection is ranked first (the merged order changes for SynonymQuery and span-or, not for MultiPhraseQuery or CombinedFieldQuery). Matching, filters and deleted documents are unaffected, though which matches make a merged top-k can change. The runs saw finite scores and no error with assertions on or off (BM25 in the repro, 16 core similarities in the harness).
• How often in random data. The finder's harness (random 8-40 document collections split into two shards, 20 query families x 16 similarities, 600 cases) recorded 10,739 mismatches in 175,837 checks of the five affected families, and none for TermQuery, PhraseQuery or the other span queries.
• Easy to miss. The visible symptom is ranking drift between shards; nothing fails.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: score of doc 0 on shard 0 vs. in the whole collection expected:<0.21363801> but was:<0.44583148>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.apache.lucene.search.TestSynonymQueryDistributedStats.testSynonymOnlyOnOtherShard(TestSynonymQueryDistributedStats.java:63)

• The hooks' purpose (IndexSearcher.java:1122-1123, :1139-1140): "This can be overridden for example, to return a term's statistics across a distributed collection" (field's, for fieldStats).
• The queries' own promise: SynonymQuery "tries to score the terms as if you had indexed them as one term" (SynonymQuery.java:47-49); CombinedFieldQuery scores terms "as if they had been indexed in a single field whose values would be the union of the values of the provided fields" (CombinedFieldQuery.java:47-49).
• Upstream tests: TestShardSearching asserts shard results equal whole-collection results with ShardSearchingTestBase's override, but only for TermQuery and PrefixQuery; its TODO lists "other queries besides PrefixQuery & TermQuery" (TestShardSearching.java:35-41). TestCombinedFieldQuery.testOverrideFieldStats (:488-563) checks CombinedFieldQuery under an overridden fieldStats (not termStats). TermQuery is the real scoring precedent: PrefixQuery rewrites to a constant score by default (PrefixQuery.java:27-34), so its scores do not depend on term statistics.
• 8.0-8.2 (LUCENE-8020, e0bde579815): callers asked the hook about every term and skipped a null result, so the overridable hook decided whether a term exists. On 8.0.0 and 8.2.0 the shard scores equal the whole-collection scores and the call log shows the hook asked about body:fast (executed).
• LUCENE-8921 (8.3.0; issue #9964, PR apache/lucene-solr#797, by @bruno-roustant, committed by @dsmiley): an API cleanup, because the hook "requires a TermStates param although it only cares about the docFreq and totalTermFreq". In review @romseygeek wrote "I think all callers are already doing this work, it's just that they're checking if the returned TermStatistics object is null after the call", and @bruno-roustant: "Previously all callers checked if the returned TermStatistics was null before doing something. So I replaced the null check by a check if docFreq is 0. I tried to remove the 0 check and ran TestShardSearching ... It failed with the exception because docFreq was 0." Compatibility was discussed for the method signature only (@jpountz: it "might not only be used internally but also by Lucene users"); nobody discussed overrides that answer for locally absent terms.
• The mock changed with it: the same commit turned ShardSearchingTestBase's "local null -> continue; // term not found, sum the other nodes" into assert distributedDocFreq > 0 (ShardSearchingTestBase.java:289 today), so no in-tree test could notice.
• Earlier design discussion: when LUCENE-6789 (#7847) created SynonymQuery, its author wrote that BlendedTermQuery has "problems like not working with distributed scoring (doesn't consult indexsearcher for stats)". LUCENE-10582 (#11618), CombinedFieldQuery mishandling overridden statistics, was fixed under Bug Fixes in 9.3 (lucene/CHANGES.txt:4119).

Strongest case for "intended" (an OpenAI GPT-5.6 reviewer, reading only, and the GPT-5.6 fact check of this doc both rated it strong; a devil's-advocate verifier built the same case): the hook is @lucene.experimental; its javadoc says docFreq "must be greater or equal to 1" and the result is "never null" (IndexSearcher.java:1125-1127); the 8.3.0 CHANGES entry says "And don't call if docFreq <= 0" (lucene/CHANGES.txt:6164-6166), so the rule was written down on purpose; in the PR #797 review the check was moved before the call deliberately (@romseygeek: "maybe we should move this check up a level and make this throw an Exception if somebody passes in an empty docFreq?"); LUCENE-8020 deliberately stopped passing non-existent terms to similarities (:6710-6713); a committer wrote in PR #910 that "Lucene doesn't really have a native concept of 'distributed collection statistics'"; Lucene does not promise exact cross-shard equality for every query (per-shard prefix, fuzzy and wildcard expansion already differ); heavier workarounds exist (overriding createWeight, expert constructors with caller-built TermStates); and any fix changes a documented calling convention that existing overrides may rely on. So this may be a documented limitation of an experimental extension point.

Why we still lean towards a bug, and where that is weaker: those texts say how the hook is called, not that multi-term queries lose the distributed statistics the same javadoc offers. Nothing documents, tests or discusses that loss; 8.0-8.2 supported it (the hook's own null result decided absence, so an override could answer for a locally absent term); the reviewers treated the new check as equivalent to the old null check for the existing callers and the default implementation, and did not consider overrides that answer for locally absent terms; and upstream fixed a sibling overridden-statistics problem in CombinedFieldQuery as a bug. This is strongest for SynonymQuery and CombinedFieldQuery, whose design history considered distributed statistics; for MultiPhraseQuery and spans (whose idf summing is itself disputed), "an enhancement that restores cross-shard parity" is arguably the more defensible framing. Upstream may still label it an enhancement.

Option A was run through gradle in a scratch worktree at a9779ef3c193 (commands in the fix folder's commands.txt); options B and C are untested design alternatives. Either way, the same check in sandbox TermAutomatonQuery, PhraseWildcardQuery and the sandbox CombinedFieldQuery is a follow-up, and a TestShardSearching case with a SynonymQuery would make the in-tree mock cover the bug.

The change is +36/-16 lines in 6 files. SynonymQuery, MultiPhraseQuery, CombinedFieldQuery and SpanWeight.buildSimWeight call searcher.termStats(term, ts.docFreq(), ts.totalTermFreq()) for every term and skip a null result. The default termStats returns null when docFreq == 0; its javadoc now says that multi-term queries also call it with docFreq 0, that the result is never null when docFreq is at least 1 (callers that only pass a positive docFreq, such as PhraseQuery and BM25Similarity, do not check for null), and that for docFreq 0 null means the term does not occur in the collection. ShardSearchingTestBase skips a null node result and returns null when no node has the term, instead of assert distributedDocFreq > 0. This restores the 8.0-8.2 behaviour (the hook decides absence) without bringing back TermStates in the signature.

// SynonymQuery (CombinedFieldQuery, MultiPhraseQuery and SpanWeight alike)
TermStats termStats = searcher.termStats(term, ts.docFreq(), ts.totalTermFreq());
if (termStats != null) {
  docFreq = Math.max(termStats.docFreq(), docFreq);
  totalTermFreq += termStats.totalTermFreq();
}

// IndexSearcher#termStats, the default; its javadoc:
//   @return A {@link TermStats}, never null if {@code docFreq} is at least 1. If {@code docFreq} is
//       0, {@code null} means that the term does not occur in the collection; the default
//       implementation returns {@code null} if, and only if, {@code docFreq} is 0.
if (docFreq == 0) {
  return null;
}
return new TermStats(term.bytes(), docFreq, totalTermFreq);

Trade-offs

• Contract change. The hook is documented today as "docFreq ... must be greater or equal to 1" and "never null". An override written for that contract which builds new TermStats(term.bytes(), df, ttf) from a sum that can be 0 throws IllegalArgumentException: docFreq must be positive for a query term that is on no shard, which any user can type (executed with the metamorphic harness's own override under the finder's sketch: in the five affected families and also in plain SpanTermQuery, SpanNearQuery, SpanFirstQuery and SpanNotQuery, because every scoring span weight goes through SpanWeight.buildSimWeight). So it needs a CHANGES/MIGRATE note ("return null when the term exists nowhere") and fits main (11.0) better than a 10.x backport.
• Cost. One extra hook call per locally absent term at weight creation; the default is a branch.
• Pending work. Open draft PR #16847 ("Support more than 2^31 documents per index (long global doc ids)", 438 changed files) also widens termStats' docFreq from int to long; whichever lands second needs a small rebase.

What the checks validated

Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the new tests 7/7 (core) and 3/3 (queries) with assertions on and off; :lucene:core:test --tests 'org.apache.lucene.search.*' 2,338 tests in 215 classes (including the 7 new ones, TestShardSearching, TestSynonymQuery, TestMultiPhraseQuery, TestCombinedFieldQuery) and :lucene:queries:test 515 tests (including the 3 new ones), assertions on and off, 0 failures; :lucene:sandbox:test --tests 'org.apache.lucene.sandbox.search.*' 114 tests, assertions off (and on, in the fact check), 0 failures. Skipped: core 49 (assertions on) / 51 (off), queries 5 / 6, sandbox 5. Tidy, ecjLint, forbiddenApis and core renderJavadoc pass. The sandbox's own copies of the check are not changed by the patch, and no targeted sandbox repro was run. After the fact checks the @return javadoc was tightened (the first wording allowed null for a positive docFreq); the code is byte-identical, and the tightened patch passed tidy, the new tests (assertions on and off), TestShardSearching, TestSynonymQuery, TestMultiPhraseQuery and TestCombinedFieldQuery (41 tests, 2 skipped), and core ecjLint, forbiddenApis, renderJavadoc and the Google Java Format check. Earlier the finder's 5-file sketch of the same change (IndexSearcher and the four callers) took the harness's mismatches in the five affected families from 10,739 to 0; mismatches in unrelated families remained, as on stock main.

Keep termStats(Term, int, long) exactly as documented and add, for example, public TermStats absentTermStats(Term term) returning null by default, which the four callers call when the local docFreq is 0.

Trade-offs

No existing override can break, so it can go into 10.x. But distributed implementations must opt in by overriding a second method, the two hooks must agree, and until they do the bug stays.

State on termStats that multi-term queries skip terms missing from the searcher's reader.

Trade-offs

No code risk, but the distributed use the javadoc offers stays broken for exactly the queries recommended for distributed scoring (synonyms, combined fields).

Method: randomized metamorphic testing (a hunt unit, 2026-10-26, main a9779ef3c193). No model checker, SMT solver or prover was involved.

• The relation: statistics injection == whole collection. The harness builds a random collection of 8-40 documents with two text fields, deletions and one or more segments, splits it into two shards, and searches one shard with an IndexSearcher whose termStats/fieldStats return the whole collection's statistics (two override styles: replace with the collection value, and the mock's local-plus-other-shards sum). Each live shard document must get exactly the score and explanation value it gets in the whole collection.
• 600 random cases (3 seeds x 200), 20 query families x 16 core similarities: 10,739 of 175,837 checks in MultiPhraseQuery, SynonymQuery, CombinedFieldQuery, SpanOrQuery and span-near over span-or mismatched, every one with a query term whose local docFreq is 0 and collection docFreq above 0; 0 mismatches for TermQuery, PhraseQuery and the other span queries. All mismatches were score values; no document matched on one side only.
• The finder (an AI tool) then read the docFreq() > 0 checks in the four weights, traced them to LUCENE-8921 with the git history, wrote a public-API JUnit and bracketed the releases; a sketch of the fix took the five affected families to 0 mismatches.
• Then a cross-model review, independent verifiers, an adjudication, a release sweep, a prior-report search and a security screen (see Independent checks).