StableTflSimilarity scores synonyms and combined fields by the length of a placeholder string

StableTflSimilarity, new in 10.6.0 (released 2026-10-29), replaces IDF with a rarity that grows with the length of the query term, and reads that length from TermStats#term; it is the only similarity in core that does. SynonymQuery and CombinedFieldQuery merge statistics over several terms or fields and pass a fixed placeholder there, "synonym pseudo-term" (19 code points) and "pseudo_term" (11). So every synonym group that is scored as a SynonymQuery (an empty group and a single unboosted term are rewritten to other queries first) gets the rarity of an almost maximally rare 19-character term, and every combined-field term that of an 11-character one, whatever was searched. SynonymQuery{ab, cd} on a 40-token document containing ab once scores 1.4050031 instead of TermQuery(ab)'s 0.9569799 (BM25 gives equal scores); a one-field CombinedFieldQuery scores 1.4047582 instead of the copy-field TermQuery's 1.3647432; and through QueryBuilder with a synonym analyzer, a 200-token "dog" document outranks an "elephant" document. The error is negligible for long terms in short documents and large for short terms in long documents. Only scores and ranking (and so the top hits) change; the set of matching documents does not. The placeholders date from 2017 and 2018 and were harmless until #16243 added a similarity that reads the term bytes; 10.6.0, main, branch_10_6 and branch_10x are affected, no earlier release is.

• StableTfl rarity tr: p = 1 - (1 - m * 2^(-c * |t|))^|d|, tr = log(1 + (1 - p + 0.05) / (p + 0.05)), score = boost * tr * tf, tf = freq / (freq + k1) (StableTflSimilarity.java:36-43, :170-175). |t| is the term length in code points, |d| the document length; m = 0.00781, c = 0.917, k1 = 1.2. Longer terms are rarer, and tr is capped at log(22) = 3.091.
• Placeholder: the fixed string that SynonymQuery and CombinedFieldQuery put into the one TermStats they hand the similarity, because their statistics are merged over several terms or fields.

Rarity at the repro's document lengths (freq 1, tf = 0.4545; hand-computed from the formula, matching the executed scores):

1. The reader. StableTflSimilarity#scorer takes getTermLength(termStats[i].term()) (StableTflSimilarity.java:195), the code-point count (:181-187), and caches tr(termLength, docLength) for all 256 norm values (:190-201); score takes the cached value for the document's norm and applies the tf saturation, weight - weight / (1 + freq / k1) (:244-245). Its javadoc: rarity "based on term length and document length, rather than relying on corpus-level statistics" (:30-31).
2. SynonymQuery. SynonymWeight takes the maximum docFreq and the summed totalTermFreq of the synonyms and calls the similarity once with new TermStats(new BytesRef("synonym pseudo-term"), docFreq, totalTermFreq) (SynonymQuery.java:209-227): |t| = 19 for every group that reaches this weight, even {a, b}; an empty group and a single unboosted term are rewritten to other queries before (:160-168).
3. CombinedFieldQuery. CombinedFieldWeight merges the per-field statistics and passes new TermStats(new BytesRef("pseudo_term"), ...) (CombinedFieldQuery.java:274-291): |t| = 11, although the query has exactly one term (:86-96, :135).
4. Reach. For unquoted text, QueryBuilder builds a SynonymQuery when the analyzer stacks tokens at one position (QueryBuilder.java:395, :409, :555, :603), and the classic and simple query parsers extend it; quoted text with stacked tokens becomes a MultiPhraseQuery (:347-351), which passes real terms. TermQuery and PhraseQuery pass real term bytes (IndexSearcher.java:1132; a recording-Similarity probe confirmed it), and no other core similarity reads the bytes.
5. Direction and size. tr falls towards 0.047 for short terms in long documents while a 19-character term stays near 3.09, so synonym groups of ordinary words are over-weighted, more so in longer documents: Synonym{dog, dogs} / TermQuery(dog) is 1.07x at about 10 tokens, 2.13x at 200, 7.56x and 66x at decoded document lengths of about 984 and 9,240 (executed). In CombinedFieldQuery, terms shorter than 11 characters gain and longer ones lose; a 14-character term on a 4-token document changes by 0.016%.

• Who is exposed. Applications that set StableTflSimilarity (default constructor is enough) and run a SynonymQuery, built directly or by QueryBuilder and the classic and simple query parsers when the analyzer puts several tokens at one position in unquoted text (a synonym filter, or keyword-repeat stemming where the stem differs from the word), or a CombinedFieldQuery.
• Default or opt-in. Opt-in similarity (@lucene.experimental); everything else is ordinary: default codec, plain IndexSearcher, the standard query builders.
• Releases. 10.6.0 (released 2026-10-29), where StableTfl first shipped; main, branch_10_6 and branch_10x are affected too.
• Scale. Any index size, ordinary text. Negligible for long terms in short documents (0.016%), large for short terms in longer documents (+47% at 40 tokens, 2.13x at 200).
• Symptom. Silent: wrong scores and ranking, no exception with assertions on or off; the explanation prints the placeholder's length as tl. The set of matching documents is unchanged; which documents make the top N can change.
• Impact class: silent wrong results (priority Medium, likelihood plausible).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<1.3855287> but was:<1.4050188>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.junit.Assert.assertEquals(Assert.java:701)
       org.apache.lucene.search.TestStableTflSynonymQuery.testSynonymGroupScoresLikeTermQuery(TestStableTflSynonymQuery.java:44)

• Contracts. StableTfl: rarity from |t| (StableTflSimilarity.java:29-43), and termLengths is "the decoded UTF-8 term length (in code points) of the i-th query term" (:217-220). TermStats: "Contains statistics for a specific term", term = "Term bytes" (TermStats.java:22-23, :46). Similarity: the statistics passed in "already contain all of the raw statistics involved, so a Similarity can freely use any combination of statistics" (Similarity.java:72-77). SynonymQuery "tries to score the terms as if you had indexed them as one term" (SynonymQuery.java:47-49). CombinedFieldQuery's requirements "hold for all similarities that don't customize computeNorm" (CombinedFieldQuery.java:60-64); StableTfl does not customize it.
• Tests. TestCombinedFieldQuery#testCopyFieldWithSingleField asserts that a one-field CombinedFieldQuery scores like a TermQuery on a copy field; its similarity list (:392-400) dates from LUCENE-9725 (2021). Adding StableTfl to it fails 6 of 336 iterations on stock and passes with the fix. TestStableTflSimilarity uses only real words (including the constant "term" of the inherited BaseSimilarityTestCase tests); no test mentions the placeholders.
• History. "synonym pseudo-term" replaced a null term in LUCENE-8020 (2017), whose stated intent is "we shouldn't pass terms that dont exist or bogus statistics" (said about bogus docFreq statistics, not about the term bytes); "pseudo_term" came from the sandbox BM25FQuery (LUCENE-8216, 2018), which forced BM25. No similarity read the bytes until #16243, whose reviews never mention synonyms, combined fields or placeholders.
• Precedent. LUCENE-10039 (2021): a one-field CombinedFieldQuery that scored unlike the copy field was fixed as a bug (different cause).

The strongest case for "intended" (built by the devil's advocate, an OpenAI GPT-5.6 reviewer and the adjudication): the placeholders predate StableTfl and only satisfied a non-null check, so the term bytes of merged statistics never had a meaning; a synonym group of different lengths has no natural length, and a constant is at least deterministic; StableTfl, Similarity, TermStats and CombinedFieldQuery are @lucene.experimental; and TestCombinedFieldQuery does not list StableTfl.

Why it fails: CombinedFieldQuery has one query term, so there is nothing to choose; {ab, cd} has length 2 and {a, b} length 1, yet both get 19; the Similarity and TermStats javadocs allow reading the bytes and call them the term's; the first term in sorted order is just as deterministic; experimental status allows API changes, not wrong scores; and the test's list predates StableTfl. The devil's advocate rated the case weak to moderate, the adjudication confirmed with high confidence. Conceded: the length for a mixed-length synonym group is a design choice, and documenting a limitation is a possible maintainer answer.

Option A was built and validated with gradle in a private clone of ea57360912ed; A2, B, C and D were not run. An optional test change adds StableTflSimilarity to TestCombinedFieldQuery#randomCompatibleSimilarity.

CombinedFieldQuery passes its query term term; SynonymQuery passes terms[0].term, the first term in sorted order (the builder sorts, SynonymQuery.java:104). +5/-4 in two files, no API change; on branch_10_6 the class is still TermStatistics. Recommended for CombinedFieldQuery, which has exactly one real term; for SynonymQuery, "first sorted term" is the validated minimal choice that removes the placeholder, not a recommendation of that rule (the adjudication's caveat): the rule is a maintainer decision.

// CombinedFieldQuery.CombinedFieldWeight
TermStats pseudoTermStats = new TermStats(term, docFreq, Math.max(1, totalTermFreq));

// SynonymQuery.SynonymWeight
TermStats pseudoStats = new TermStats(terms[0].term, docFreq, totalTermFreq);

Trade-offs

• Mixed-length synonyms. "First sorted" is arbitrary with respect to length: {dog, dogs} gets "dog" (3), {colour, color} gets "color" (5). It is deterministic and corpus-independent, which keeps StableTfl's promise that rarity does not come from corpus statistics; do not pick, for example, the highest-docFreq synonym. "First" is BytesRef order: it ignores length and boosts, and it can be a synonym that is absent from the index while another one matches.
• One term's bytes with merged statistics. The TermStats pairs one synonym's bytes with the group's merged docFreq and totalTermFreq; no other core similarity reads the bytes, so BM25 and the rest are unchanged. Under StableTfl the score, the explanation and possibly the ranking change; matching does not. An outside similarity that reads the bytes would see the change (experimental APIs).
• Sandbox copy. The deprecated sandbox CombinedFieldQuery (lucene/sandbox/.../CombinedFieldQuery.java:325) has the same placeholder, but it holds several terms (Builder.addTerm at :131-137, BytesRef[] terms at :155, sorted at :169), like SynonymQuery; a terms[0]-style change would apply, with the same mixed-length question (not tested).

What the checks validated

• Stock: the repro fails 3 of 5 (assertions on and off), the finder's test 3 of 3; with StableTfl added to TestCombinedFieldQuery's similarity list, 24 iterations fail 6 of 336 (testCopyFieldWithSingleField, testCopyFieldWithMissingFields, testOverrideFieldStats), plus a suite-level file-handle-leak error that follows from those failures.
• With the fix: the repro passes 5 of 5 (assertions on and off), the finder's 3 of 3, TestCombinedFieldQuery with StableTfl 336 of 336; TestCombinedFieldQuery, TestSynonymQuery, similarities.* and TestQueryBuilder (29 classes, 168 tests), the whole core module (749 classes, 8,873 tests, 322 skipped) and :lucene:queryparser:test (533 tests, 1 skipped) have 0 failures; :lucene:core:check -x test passes; tidy leaves the patch unchanged.
• Standalone harness, fixed classes shadowed: the repro passes 5 of 5.

Pass the shortest synonym (never rates the group rarer than its least rare member), the longest, or the highest-boost one with ties broken by sort order (follows the application's synonym weights).

Trade-offs

As cheap as A, but couples SynonymQuery to a notion of length only StableTfl uses. Any of them gives {dog, dogs} length 3 or 4, so the repro's "elephant" document ranks first either way.

Let a Similarity receive every synonym's statistics through a separate method.

Trade-offs

Most principled, but an API change touching every Similarity; the built-in similarities, BM25 included, add up several TermStats passed to scorer (as for phrases), so passing every synonym's statistics there would change BM25's synonym scores.

Javadoc: StableTfl's term length is undefined for SynonymQuery and CombinedFieldQuery.

Trade-offs

Cheap, but leaves the unambiguous cases (one-field CombinedFieldQuery, equal-length synonyms) wrong.

Special-case the two strings in the similarity.

Trade-offs

Fragile, and there is still no real length to use.

Method: non-formal. A hunt unit (new similarities, 2026-10-26). The finder, an AI tool, was building an SMT model of StableTflSimilarity's norm monotonicity. Reading the scorer's inputs, it noticed that StableTfl is the only Similarity in core that reads TermStats#term; a grep of every new TermStats( call site then showed the placeholders in SynonymQuery and CombinedFieldQuery. It wrote a public-API JUnit on the stock jar. The model's output played no part.

Then: cross-model reviews (OpenAI GPT-5.6, reading only; one of them is the root-cause adversary, whose executed checks added a recording-Similarity probe), an independent reproducer with BM25 controls, an intended-behaviour devil's advocate, an adjudication, a history run on 7 builds, a prior-report search, a security probe, a credibility audit and this doc's gradle validation.