SweetSpotSimilarity's length plateau makes top-k search skip the best hits

SweetSpotSimilarity (lucene/misc) lets an application declare a plateau of ideal field lengths with setLengthNormFactors(min, max, steepness): lengths from min to max get a length factor of 1.0, shorter and longer fields less. It keeps the inherited computeNorm, which stores the field length as the norm, so below the plateau a longer field has a greater norm and a higher score. Similarity#computeNorm forbids exactly that ("greater unsigned norms must produce scores that are lower or equal"), because the postings writer keeps, for each block of 256 postings, only the (freq, norm) pairs that can produce the best score under that rule. A short field then hides a better-scoring longer field of the same or lower term frequency: the block claims a too-low maximum, and the default IndexSearcher#search(q, n) skips it once it has counted more than 1,000 hits. In the repro search(q, 10) returns docs 0-9 (1.0) and misses doc 2150 (1.414); Lucene's CheckHits.checkTopScores fails. Scores of returned hits are correct and nothing throws. Every plateau with min >= 2 (and a positive steepness) is affected, the default min = max = 1 is not; every release from 8.0.0 (block impacts, LUCENE-4198) to 10.5.2 and main.

• Norm: one byte per document and field, written at index time by Similarity#computeNorm; by default the field length (SmallFloat.intToByte4, equal to the length up to 40 terms; Similarity.java:151-161).
• Plateau: lengthNorm(x) = 1/sqrt(steepness * (|x-min| + |x-max| - (max-min)) + 1), 1.0 for min <= x <= max and lower on both sides (SweetSpotSimilarity.java:97-120). The default min = max = 1, steepness 0.5, is 1/sqrt(x), like ClassicSimilarity.
• Block impacts: for each block of 256 postings the writer stores the (freq, norm) pairs that can produce the block's best score; top-k search computes the block's maximum score from those pairs and skips the block when it cannot beat the current k-th score.

With setLengthNormFactors(10, 20, 0.5f) (freq 1, idf 1):

Below 10 terms the score rises with the norm; above 20 it falls, as the rule requires.

1. Score. SweetSpotSimilarity overrides lengthNorm (SweetSpotSimilarity.java:110-120) and tf (:127-130, sqrt(freq) with the default tf factors, as in ClassicSimilarity), but not computeNorm, so the norm is the length (Similarity.java:151-161). TFIDFSimilarity#scorer (final) fills normTable[i] = lengthNorm(byte4ToInt(i)) for the norm bytes 1-255 (TFIDFSimilarity.java:426-448) and scores tf(freq) * idf * boost * normTable[norm] (:467-472): freq 1 scores 1.0 at norm 10 and 0.333 at norm 2.
2. Rule. Similarity#computeNorm: "for a given term-document frequency, greater unsigned norms must produce scores that are lower or equal" (Similarity.java:140-145); SimScorer#score: "Score must not increase when the unsigned norm increases" (:211-213).
3. Index. Lucene104PostingsWriter adds each posting's (freq, norm) to a CompetitiveImpactAccumulator (Lucene104PostingsWriter.java:280-296). getCompetitiveFreqNormPairs() walks the norms upward and keeps a pair only if its freq beats every freq at a smaller norm (CompetitiveImpactAccumulator.java:101-111). In block 2048-2303 norm 2 has freq 2 (short docs) and norm 10 has freq 2 (doc 2150): the norm-10 pair is dropped, the block's maximum becomes sqrt(2)/3 = 0.4714, and Scorer#getMaxScore(2150) returns 0.47140452 while doc 2150 scores 1.4142135 (executed). The level-1 skip data of every 32 blocks merges these pairs through the same accumulator (Lucene104PostingsWriter.java:482-490), so its bound can be too low the same way.
4. Search. IndexSearcher#search(q, n) counts 1,000 hits exactly (IndexSearcher.java:100, :597-599), then sets the minimum competitive score to Math.nextUp(1.0) (TopScoreDocCollector.java:154-165), and ImpactsDISI#advanceTarget skips every block whose maximum score is below it (ImpactsDISI.java:67-98). The block of doc 2150 claims 0.471 and is skipped; the top 10 is docs 0-9.
5. Reach. A document is hidden when a shorter, sub-plateau field in its block has an equal or greater term frequency and a lower score. Plateau fields qualify only with a term frequency below min (without overlapping tokens, freq <= length); fields above the plateau qualify too: for 10..20 a 21-term field with t once (0.707) was hidden by 2-term fields with t once (0.333) in an executed variant, and found with the fix. One short field in the same block is enough (the independent reproducer's layout: doc 2000 "t t" hides doc 2001). A bounded-exhaustive check of every plateau 1 <= min <= max <= 64 with the steepness values 0.1, 0.5, 1 and 2 found 8,064 of 8,320 settings breaking the rule: exactly those with min >= 2 (steepness 0 would flatten the curve).

• Who is exposed. Users of org.apache.lucene.misc.SweetSpotSimilarity who call setLengthNormFactors with min >= 2, the plateau the class exists for. Solr ships SweetSpotSimilarityFactory (javadoc example 3/5/0.5) but counts hits exactly unless a request sets minExactCount; Elasticsearch and OpenSearch do not ship the similarity (source reading only).
• Default or opt-in. Opt-in similarity with a non-default plateau; everything else is default: IndexWriterConfig, codec, plain IndexSearcher, IndexSearcher#search(q, n).
• Scale. More than 1,000 matches for the default API (3,000 documents in the repro), and the hidden document's block must be reached after the collector has raised its bar (in the one-segment repro: after the first 1,001 hits); callers with a lower total-hits threshold prune sooner. Only postings with stored impacts are skipped (a postings format without impacts serves SlowImpactsEnum, which never skips).
• Queries. TermQuery top-k executed wrong on main and every release run from 8.0.0 to 10.5.2; 2-clause disjunctions also on main and several releases. Phrase, synonym and other impact-based queries were not run.
• Symptom. Silent: wrong best hits, correct scores, a correctly labelled hit count, a clean index, no exception with assertions on or off. A hit is lost only when a shorter, lower-scoring field with an equal or higher term frequency shares its 256-posting block and the hidden document beats the current k-th score. In a synthetic probe one short document per 200 postings removed the whole true top 10; real-corpus effect not measured.

• Contracts. Similarity#computeNorm (Similarity.java:140-145): "greater unsigned norms must produce scores that are lower or equal ... for any legal freq", and "1 is the norm that produces the highest scores". SimScorer#score (:211-216): "Score must not increase when the unsigned norm increases". IndexSearcher (IndexSearcher.java:59-67): only the hit count may be a lower bound; "The TopDocs#scoreDocs array is always accurate however." Scorer#getMaxScore must bound the scores it covers; it returns 0.4714 for a document scoring 1.4142.
• No documented limit. setLengthNormFactors (SweetSpotSimilarity.java:85-95) names no restriction; SweetSpotSimilarity's own javadoc has no @lucene.experimental tag (its base class Similarity is experimental as a whole, Similarity.java:89); no CHANGES or MIGRATE entry warns about plateaus and top-k search.
• Tests. BaseSimilarityTestCase asserts score(freq, norm-1) >= score(freq, norm) for every norm ("scores non-decreasing as docs get shorter", BaseSimilarityTestCase.java:437-470), and TestClassicSimilarity extends it. TestSweetSpotSimilarity extends only LuceneTestCase and never checks lengths below the plateau or a top-k search. A BaseSimilarityTestCase subclass on random plateaus fails on stock ("score(1.0,1)=0.19876102 < score(1.0,2)=0.20500971") and passes with the fix.
• History. LUCENE-8119 (#9167) wrote the rule for impacts: "scores must not increase when the unsigned norm increases. The latter is something I need for #5270"; LUCENE-4198 (#5270): "This patch adds the assumption than (unsigned) greater norms produce equal or lower scores". Neither touched lucene/misc. The project fixed or dropped other non-monotone formulas (LUCENE-8010's sweep of core similarities; LUCENE-8015 for DFR; LUCENE-7997, #9045, changed BM25 to guarantee monotonicity, and LUCENE-9071 later sped it up and kept that guarantee; DFR's BE, P and D models were left out for this reason).

The strongest case for "intended" (built by the devil's advocate, the GPT-5.6 intended-lens reviewer and the adjudication): the plateau and its drop-off below min are the class's documented purpose since 2006 (LUCENE-577); Similarity is a low-level API whose class javadoc says "Lucene makes no assumption about what is in this norm" (Similarity.java:47-48); the Similarity class (:89), ClassicSimilarity#lengthNorm and Similarity#computeNorm are @lucene.experimental; and a user can count all hits instead. So a maintainer could call it "garbage in".

Why it fails: the plateau's scores are intended, wrong top hits are not, and IndexSearcher promises accurate scoreDocs. The user implements nothing; a shipped, non-sandbox class breaks a "must" that core relies on. The later, specific rule (8.0, written for impacts) governs the older general sentence, and experimental status allows API changes, not wrong results. Every verifier and the adjudication rejected the argument. What is fairly conceded: any code fix changes the stored norm and needs re-indexing, so documenting "use min = 1 or count all hits" is a legitimate maintainer choice for a misc class.

Option A was built and validated with gradle in a scratch clone of ea57360912ed; A2 was run in the standalone harness; B, C and D were not run. The finder's earlier sketch (store the distance from the plateau) failed the upstream nightly test and is not offered.

computeNorm writes a field of min - d terms (d >= 1) as max + d, which the plateau formula scores the same (both give 1/sqrt(2 s d + 1)), and lengthNorm uses x = max(numTerms, min), so over all 255 norm bytes the factor is 1.0 up to max and falls after it. Lengths >= min and empty fields are encoded as today; new indexes score exactly as before whenever the mirrored length is at most 40 (for example 29 for the plateau 10..20; identical scores for every length 1 to 2,000 on the plateaus 10..20 and 3..5), while wider plateaus change some scores (Precision below). Patch: SweetSpotSimilarity.java +39/-6; the computeNorm body copies Similarity#computeNorm.

@Override
public long computeNorm(FieldInvertState state) {
  final int numTerms = ...; // as in Similarity#computeNorm
  if (numTerms > 0 && numTerms < ln_min) { // below the plateau: store max + (min - numTerms)
    final long mirrored = (long) ln_max + (ln_min - numTerms);
    return SmallFloat.intToByte4((int) Math.min(Integer.MAX_VALUE, mirrored));
  }
  return SmallFloat.intToByte4(numTerms);
}

@Override
public float lengthNorm(int numTerms) {
  final long x = Math.max(numTerms, ln_min); // then the unchanged formula, in long arithmetic
  ...

Trade-offs

• Re-indexing. Sub-plateau fields get a new norm (the computeNorm javadoc warns that overriding it needs re-indexing). Until an index is rebuilt, its old sub-plateau fields score like plateau fields, while newly indexed ones keep their penalty; merging copies norms and does not recompute them, so mixed segments score inconsistently until the documents are re-indexed (read, not executed).
• lengthNorm(int) changes for direct callers: below min it returns 1.0; the drop-off moves into computeNorm. The class javadoc needs a sentence.
• CombinedFieldQuery requires an additive computeNorm encoded with SmallFloat#intToByte4 (CombinedFieldQuery.java:60-64; the default computeNorm satisfies it); the mirrored encoding is not additive. Today the combination is correct, because its bound score(+Infinity, 1) is +Infinity and no block is skipped; with the fix, a document with two 2-term fields is scored as one 56-term field (28 + 28) instead of 4 terms: 0.5345225 becomes 0.23249526 (executed).
• Precision (wide plateaus). A mirrored length above 40 keeps 4 significant bits and is rounded down, possibly into the plateau. For the plateau 50..100, 43 of the lengths 1 to 110 score higher than today, none lower; a 47-term field with t once goes from 0.4472136 to the plateau score 1.0 (executed). The norm order stays correct. Not tried: rounding the mirrored length up instead.
• Not covered by the new tests: IndexOptions.DOCS fields and overlapping tokens (the override copies those branches from Similarity#computeNorm).
• Backports. The design needs 9.12.0 or later (overridable Similarity#computeNorm; tested on main only). Through 9.11.1 TFIDFSimilarity#computeNorm is final and setLengthNormFactors takes a fourth argument, so 8.x and early 9.x need another design.
• The long arithmetic also removes a NaN length factor for fields of more than about 1.07 billion terms (an int overflow noticed during the hunt, a separate edge). The BaseSimilarityTestCase subclass needs it because it probes norm byte 255; drop both for the smallest change.

What the checks validated

• Gradle, scratch clone of ea57360912ed, OpenJDK 25.0.3: stock: the repro fails 5 of 6 (assertions on and off), the BaseSimilarityTestCase subclass fails, upstream TestSweetSpotSimilarity with -Ptests.nightly=true passes (its @Nightly testSweetSpotComputeNorm ran).
• With the fix: the repro passes 6 of 6 (assertions on and off); the subclass passes 2 of 2 and 40 of 40 over 20 iterations; upstream TestSweetSpotSimilarity passes with -Ptests.nightly=true (two seeds, assertions on and off), including the 0.0-tolerance comparison with ClassicSimilarity that the finder's sketch failed; the whole misc module (25 classes, 323 tests, 0 failures, 9 skipped) and :lucene:misc:check -x test pass.
• Standalone harness, fixed class shadowed: the finder's JUnit passes 2 of 2; the devil's advocate's main() agrees on all four configurations; the independent reproducer's JUnit passes 3 of 4: its testNormContract asserts as a precondition that a 2-term field's norm byte is 2, which the fix changes on purpose (28).

Old segments keep their old scores (and their wrong pruning) until re-indexed; new segments prune correctly.

Trade-offs

The score table still inverts for norm bytes below min, which new segments never write: the repro's two score-table checks fail (norm 2 scores 0.33333334 > norm 1 scores 0.31622776), the search tests pass, and score(Float.MAX_VALUE, 1) is then not the documented maximum.

What the checks validated

Standalone harness: the repro fails 1 of 6 (the score-table check); the subclass fails.

Only the lengthNorm clamp, no computeNorm change.

Trade-offs

No re-indexing and CombinedFieldQuery stays compatible, but short fields are no longer penalized at all, which drops half of what the class documents ("Below the min, and above the max the lengthNorm drops off").

Javadoc on the class and on setLengthNormFactors: a plateau with min >= 2 breaks the computeNorm order, so top-k search with pruning can miss hits; use min = 1 or count all hits (totalHitsThreshold = Integer.MAX_VALUE; in Solr, no minExactCount).

Trade-offs

Cheap and honest, but a shipped class keeps silently breaking a core contract.

For example a per-similarity flag that disables impact-based skipping.

Trade-offs

Touches the postings format or every pruning scorer to support one misc class.

Method: semi-formal. A hunt unit (misc module, 2026-10-26). The finder, an AI tool, read SweetSpotSimilarity against the Similarity#computeNorm rule (the plan's secondary method, "SweetSpotSimilarity monotonicity"), then checked the rule bounded-exhaustively on the stock jar: every plateau 1 <= min <= max <= 64, 4 steepness values, every consecutive pair of norm bytes. 8,064 of 8,320 configurations violate it, exactly those with min >= 2; none with min = 1. It then wrote an end-to-end public-API JUnit with IndexSearcher#search(q, 10) and CheckHits.checkTopScores.

Then: an independent reproducer with its own layout, cross-model reviews (OpenAI GPT-5.6, reading only), a devil's advocate with a plain-IndexSearcher main(), an adjudication, a version sweep with a commit-level bisect, a prior-report search, a document-only security probe, and this doc's fix with gradle validation.