SynonymQuery with a repeated term counts it once per copy, so its statistics can exceed the field's totals

SynonymQuery "tries to score the terms as if you had indexed them as one term": its weight passes the similarity one pseudo-term whose docFreq is the largest docFreq of the terms and whose totalTermFreq is the sum of theirs. SynonymQuery.Builder keeps a term that is added twice, and the sum then counts that term's occurrences once per copy while the docFreq counts it once. When the copies add up to more than the field holds, the pseudo-term breaks the bound TermStats documents as "always true" (totalTermFreq <= sumTotalTermFreq of the field). With -ea, every SimilarityBase similarity (LM Dirichlet, LM Jelinek-Mercer, Indri, DFR, IB, DFI, Axiomatic) trips the assertion in the shared SimilarityBase.fillBasicStats; with -da, the formulas that use totalTermFreq (the LM family, DFI and some DFR and IB configurations, not Axiomatic) score from impossible statistics (LM Dirichlet's collection probability goes above 1). Stock components produce such queries: the synonym rule résumé, resume, cv with ASCII folding makes QueryBuilder and the classic query parser build Synonym(body:cv body:resume body:resume) for the query resume. BM25 and Classic never read totalTermFreq, and which documents match never changes. Present since SynonymQuery was added in 6.0.0; not fixed on main.

• totalTermFreq, sumTotalTermFreq: how often a term occurs in a field over all documents, and how many tokens the field holds in total. A real term can never occur more often than the field has tokens, and TermStats lists totalTermFreq <= sumTotalTermFreq of the field among the conditions that are "always true" (TermStats.java:32-38).
• Pseudo-term: SynonymQuery "tries to score the terms as if you had indexed them as one term: it will match any of the terms but only invoke the similarity a single time, scoring the sum of all term frequencies for the document" (SynonymQuery.java:47-49). It passes one TermStats for all its terms.
• SimilarityBase family: LM Dirichlet, LM Jelinek-Mercer, Indri Dirichlet, DFR, IB, DFI and Axiomatic. All of them assert the bound in the shared fillBasicStats. Their formulas differ: the LM family, DFI, the DFR parts G, I(F), I(ne), after-effect B and normalization H3, and IB's LambdaTTF read totalTermFreq, for example the LM collection probability P(t|C) = (totalTermFreq + 1) / (sumTotalTermFreq + 1) (LMSimilarity.java:150-152). Axiomatic, DFR I(n) with L and H1/H2/Z, and IB with LambdaDF and H1/H2/Z do not. The default BM25 and Classic do not read it at all.

1. The builder keeps repeats. Builder.addTerm checks only the field, the boost range and the clause count (SynonymQuery.java:79-100); build() sorts and copies (:103-106). A repeated term stays, next to its first copy. No javadoc asks for unique terms.
2. The weight sums per entry. SynonymWeight loops over every entry (:213-222): docFreq = Math.max(termStats.docFreq(), docFreq) (:219) but totalTermFreq += termStats.totalTermFreq() (:220). k copies of a term add its totalTermFreq k times; the max counts it once.
3. The similarity gets it. new TermStats("synonym pseudo-term", docFreq, totalTermFreq) goes to similarity.scorer (:225-227). TermStats does not check it against the field; that is a TODO (TermStats.java:62-63: "otherwise many similarity functions will implode").
4. Where it breaks. SimilarityBase.fillBasicStats asserts totalTermFreq <= sumTotalTermFreq (SimilarityBase.java:80); without assertions it copies the value into its statistics (:88), LM's collection probability goes above 1, and the other formulas that read totalTermFreq use the inflated value. Executed with -da: SynonymQuery(a, a) scores the same with and without the fix under the 6 Axiomatic variants, DFR I(n)L1/I(n)L2 and IB LL-D2, and differently under LM Dirichlet, LM Jelinek-Mercer, Indri, DFI, DFR GB2, DFR I(F)L2 and IB LL-L2.
5. When. The distinct terms' occurrences are disjoint, so they stay within the field on their own; the bound breaks when the extra copies, (k - 1) × ttf for a term present k times, outnumber the field's tokens that are not occurrences of any of the query's distinct terms. With no other term in the query, the term must be more than 1/k of the field; with other synonyms a smaller share is enough (docs a b, b, b: Synonym(f:a f:a f:b) gives 5 > 4 with a 1 of 4 tokens). Short or low-cardinality fields, small indexes.
6. Not matching. Without scores the query becomes a BooleanQuery of SHOULD TermQuerys (SynonymQuery.java:186-194) and never builds the pseudo-term, so hits, count() and filters are unchanged.

• Synonym rules whose entries coincide after analysis. SolrSynonymParser with expand=true adds every pair i != j by position in the rule, after analysing each entry (SolrSynonymParser.java:116-127). résumé, resume, cv with ASCII folding, or Dog, dog, puppy with lower-casing, maps a term to itself; SynonymGraphFilter emits it twice, and QueryBuilder.newSynonymQuery copies every stacked token into the builder (QueryBuilder.java:603-609). The classic and simple query parsers extend QueryBuilder. WordnetSynonymParser does the same.
• KeywordRepeatFilter plus a stemmer, without RemoveDuplicatesTokenFilter, for a word already in stem form: Synonym(f:run f:run). Weaker, because KeywordRepeatFilterFactory advises adding that filter.
• Code that adds the same term to SynonymQuery.Builder twice.

The fix rows keep the query and its doubled per-document frequency (freq 2 for resume, 4 for a a b); only the statistics change. LM Dirichlet with mu 2000.

The claimed -da effect is the statistic: a collection probability above 1, from which the scores are computed. The toy scores are not a harm on their own: TermQuery gives the same 0.0 values (LM Dirichlet clamps negative scores to 0, LMDirichletSimilarity.java:81) and the same order, and the fix row's order in the second example comes from the doubled per-document frequency the fix keeps, which is not claimed.

• Who. Applications that set a SimilarityBase or custom similarity with IndexSearcher#setSimilarity. With assertions on, every SimilarityBase similarity fails; with assertions off, only the formulas that use totalTermFreq change: the LM family including Indri, DFI and some DFR and IB configurations, not Axiomatic. The default BM25Similarity (IndexSearcher.java:123), ClassicSimilarity and BooleanSimilarity never read totalTermFreq, so the statistics part does not change their scores. Codec, writer, merge policy and searcher stay default.
• Trigger. A SynonymQuery that holds the same term more than once: from code, or from QueryBuilder and the stock query parsers when the analysis chain stacks a token on itself (synonym rules with case or accent variants under expand=true, or KeywordRepeatFilter without RemoveDuplicatesTokenFilter).
• Scale. The bound breaks only when the extra copies outnumber the field's tokens outside the query's distinct terms: a repeated term that is more than 1/k of the field for k copies, or a smaller share when the other synonyms' occurrences make up the difference. Ordinary text fields of large indexes do not get there; short fields (titles, tags, categories), small indexes and test indexes can. Nobody has reported it since 6.0.0, so the registry rates it rare.
• With assertions on (tests, assertion-enabled deployments): AssertionError while the Weight is built, so the query fails. A probe of 2,340 cases saw 1,488 such errors, all above the bound and none at or below it.
• With assertions off (production): scores from impossible statistics, silently, under the formulas that use totalTermFreq: LM Dirichlet's P(t|C) is 1.5 and 1.4 in the examples, above 1 (0.75 and 0.8 for the equivalent TermQuery). The two-document examples show no ranking harm against TermQuery; how much rankings move in real indexes was not measured.
• Not affected. Which documents match, count() and filters. The probe (65 similarity configurations, 3 corpora, k in {1, 2, 3, 8, 64, 1023} copies with and without a second term, assertions off) found no NaN or infinite score, no hit dropped because of the repeat, and the same top-10 score values pruned and exhaustive. The only dropped hits were three IndriDirichletSimilarity k=1 baselines without a repeat, and negative scores appeared only under IndriDirichletSimilarity, which scores negative without repeats too (both a separate registry bug).
• Below the bound the totalTermFreq is still counted per copy, but no documented contract breaks, so that is not claimed; the doubled per-document frequency (which changes BM25 too) is not claimed either.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: pseudo-term totalTermFreq 6 > field sumTotalTermFreq 4
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.search.TestSynonymQueryRepeatedTermStats.testRepeatedTermStatsStayWithinFieldTotals(TestSynonymQueryRepeatedTermStats.java:59)

• The bound (TermStats.java:32-38): "The following conditions are always true: ... totalTermFreq <= sumTotalTermFreq of the field"; @param totalTermFreq (:54-59): "never exceeds FieldStats#sumTotalTermFreq()"; the TODO (:62-63): "actually add missing cross-checks to guarantee TermStats is in bounds of FieldStats, otherwise many similarity functions will implode."
• The assertion (SimilarityBase.java:80): added by LUCENE-3723 (2012, 4.0, then with a -1 escape for missing statistics) and rewritten by LUCENE-8007 (2017, 8.0.0), which also wrote the "always true" javadoc.
• One term (SynonymQuery.java:47-48). Distinct terms cannot break the bound, because their occurrences are disjoint; only a repeat can.
• Legal input. Builder.addTerm checks only the field, the boost range and the clause count, and documents no uniqueness requirement (:70-100). Upstream's own TestSynonymQuery.testRewrite builds a SynonymQuery from the same term twice and asserts that both entries are kept (LUCENE-9823).
• Siblings keep the bound. CombinedFieldQuery.Builder keys fields in a map, so a repeated field counts once (CombinedFieldQuery.java:85, :117). On main BooleanQuery merges repeated SHOULD clauses (when minimumNumberShouldMatch <= 1) and repeated MUST clauses into one clause with a combined boost (BooleanQuery.java:438-455, :706-714), never into inflated statistics. The test framework draws a similarity's random term statistics so that the term "fits within the bounds of the corpus" (BaseSimilarityTestCase.java:203).
• Precedents. LUCENE-8020 (#9068, fixed in 8.0): "we shouldn't pass terms that dont exist or bogus statistics", a general rule (the issue is about span queries and every similarity formula) whose fix also made SynonymQuery skip docFreq-0 terms. LUCENE-3723 (#4797, open): the same out-of-bounds symptom in FieldMaskingSpanQuery. LUCENE-7468 (#8520): an analysis filter emitting duplicate stacked tokens was fixed as a bug.

Strongest case (rated "moderate" by the cross-model reviews, built in full by the intended-behaviour verifier): SynonymQuery treats its terms as a multiset. The javadoc says "scoring the sum of all term frequencies", build() never deduplicates, testRewrite keeps two identical terms, testRandomTopDocs forces term2 != term1 without saying why (TestSynonymQuery.java:476-480), and in LUCENE-8644 (#9690) a reviewer asked for an array rather than a map because duplicate terms "should be rare but still possible". Repeats may also be called a configuration issue: KeywordRepeatFilterFactory advises RemoveDuplicatesTokenFilter. The statistics API is @lucene.experimental (TermStats.java:60, SimilarityBase.java:38), and the pseudo-statistics are approximate by design (the max docFreq is not the docFreq of the union).

Why it fails: all of that is about whether repeats may exist and how much a repeat may weigh per document, which is not claimed. None of it allows corpus statistics above the field's totals, which describe the index, not the query; the max-docFreq approximation stays within the bounds, BooleanQuery expresses multiplicity as a boost, and no document, test or thread says out-of-range statistics are acceptable. The configuration argument does not cover direct Builder input or an ordinary-looking rule with an accent variant, about which nothing warns. A reviewer may still weigh that the recommended fix also changes multiset scoring below the bound.

Option A ran through gradle in a scratch worktree at a9779ef3c193 (commands in the fix folder's validate.sh); option B was run and rejected; options C and D are untested alternatives. If the separate statistics-override fix (the ts.docFreq() > 0 guard on the same line) lands first, keep both conditions.

The change is +4/-1 lines in 1 file: skip the statistics of an entry whose term equals the previous sorted entry's. The docFreq max and the totalTermFreq sum then see each distinct term once, as for one indexed term; TermStates, the scorer and the per-document frequency sum are unchanged.

        termStates[i] = ts;
        // terms are sorted, so a repeated term follows its first copy: count its statistics once,
        // otherwise totalTermFreq can exceed the field's sumTotalTermFreq
        boolean repeat = i > 0 && terms[i].term.equals(terms[i - 1].term);
        if (ts.docFreq() > 0 && repeat == false) {
          TermStats termStats = searcher.termStats(term, ts.docFreq(), ts.totalTermFreq());

Trade-offs

• Scores under the similarity formulas that use totalTermFreq (the LM family, DFI, some DFR and IB configurations) change for every SynonymQuery with a repeated term, also below the bound (its totalTermFreq is then counted once). Axiomatic, the DFR and IB configurations that ignore totalTermFreq, BM25 and Classic scores do not change.
• TermStates is still built once per entry (unchanged cost).
• What a term repeated with different boosts should mean stays open; the fix does not touch boosts.
• It relies on sorted adjacency. Builder.addTerm(BytesRef, float) keeps the caller's BytesRef without copying it (SynonymQuery.java:95), so a caller that mutates it after build() already breaks the sort order and equals; QueryBuilder copies its bytes. A seen-set, or a deep copy in addTerm, would not depend on adjacency.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF, -Ptests.asserts=true and =false (commands in validate.sh).
• The new tests: TestSynonymQueryRepeatedTermStats 2 of 2 and TestSynonymQueryFromFoldedRule 1 of 1 pass, assertions on and off (they fail 3 of 3 on stock main both ways).
• Surrounding suites, 258 classes and 3,013 tests (including the 2 new classes and their 3 tests), 0 failures with assertions on (50 skipped) and off (52 skipped): :lucene:core:test --tests 'org.apache.lucene.search.*' (215 classes, 2,333 tests, including TestSynonymQuery with testRewrite), --tests org.apache.lucene.util.TestQueryBuilder (26), :lucene:analysis:common:test --tests 'org.apache.lucene.analysis.synonym.*' (12 classes, 121) and the whole :lucene:queryparser:test (30 classes, 533).
• tidy left the patch unchanged; ecjLint (core main and test, analysis-common test), forbiddenApis (core, analysis-common) and checkGoogleJavaFormat (core, analysis-common) pass.
• The same logic passed the finder's, the reproducer's and a reviewer's JUnits and upstream TestSynonymQuery (10 of 10) through the hunt's shadow-class runner.

Merge repeated entries into one in build().

Trade-offs

Changes the query structure (getTerms(), equals, the per-document frequency, so BM25 scores too) and needs a rule for the boost; breaks upstream TestSynonymQuery.testRewrite, which asserts that two identical terms are kept.

What the checks validated

Executed with the first fix sketch: testRewrite fails, 1 of 10 TestSynonymQuery tests.

Skip a self-mapping in SolrSynonymParser/WordnetSynonymParser when two entries are equal after analysis, or deduplicate stacked tokens in QueryBuilder.newSynonymQuery.

Trade-offs

Does not cover direct Builder input or custom analysis chains (KeywordRepeatFilter), and changes the parsers' token output, a separate behaviour change.

Validate TermStats against FieldStats, or clamp the pseudo-term's totalTermFreq to the field's sumTotalTermFreq.

Trade-offs

A check turns the silent -da case into an exception; a clamp hides the symptom while the statistics stay per copy below the bound.

Method: differential testing of a javadoc claim (a hunt unit, 2026-10-26, main a9779ef3c193; the hunt's "semi-formal" tier: a hypothesis formed while modelling the claim, then witnessed by a tool). No model checker, SMT solver or prover was involved.

• Before running, the unit wrote down a reading and an input domain for the sentence: for distinct terms the two halves of the javadoc agree; for a repeated term "as if indexed as one term" and "the sum of all term frequencies" disagree, so a doubled score alone would count as ambiguous, and only an invariant Lucene itself asserts, or a score neither reading allows, would count.
• A differential probe on the stock main jar ran SynonymQuery(a, a), (a, a^0.5) and (a, a, b) against TermQuery(a) and SynonymQuery(a, b) over 11 similarities with -ea and -da. The -ea runs threw the SimilarityBase.fillBasicStats AssertionError; a reach probe then found that stock QueryBuilder and synonym parsers produce such queries.
• An earlier unit (distributed-statistics scoring) had seen the same assertion for Synonym(f:aa f:aa^0.5) and noted it without pursuing it.
• Then a cross-model review, independent verifiers, an adjudication, a release sweep, a prior-report search and a security screen (see Independent checks).