PerFieldSimilarityWrapper does not forward computeQueryTermWeight, so a BM25 or StableTfl k3 setting has no effect behind a per-field similarity

#15818 (unreleased, 11.0.0) added query-side term frequency saturation: BM25Similarity built with k3 >= 0 weighs a term repeated qtf times in a query ((k3 + 1) * qtf) / (k3 + qtf) instead of qtf; #16243 gave StableTflSimilarity the same k3. The weighting is applied in BooleanQuery.rewrite, which merges equal unboosted clauses and takes the merged clause's boost from indexSearcher.getSimilarity().computeQueryTermWeight(qtf).

That hook has no field parameter and defaults to linear. PerFieldSimilarityWrapper forwards only computeNorm and scorer to get(field) and inherits the default. So behind a wrapper, even one that returns the same k3 similarity for every field, k3 does nothing: scores and ranking are exactly those of k3 disabled, with no error or warning. The matching set is unchanged. No release has the bug; main and branch_11x do.

1. Duplicates are merged. BooleanQuery.java:438-447 deduplicates SHOULD clauses when minimumNumberShouldMatch <= 1, :449-455 MUST clauses; both call deduplicateClauses (:714-758).
2. The searcher's similarity is asked. :734 Similarity similarity = indexSearcher.getSimilarity();. When no occurrence has an explicit boost, the merged clause's boost is similarity.computeQueryTermWeight(Math.toIntExact(Math.round(boostSum))) (:743-746); with explicit boosts it stays the linear sum, by design. This is the hook's only production caller.
3. The hook. Similarity.java:163-177: computeQueryTermWeight(int queryTermFrequency), no field parameter, @lucene.experimental, default return (float) queryTermFrequency; (:175-177), "Subclasses may override this to apply saturation (e.g. BM25's k3 parameter)".
4. The overrides. BM25Similarity.java:130-136 and StableTflSimilarity.java:158-164 return ((k3 + 1f) * qtf) / (k3 + qtf) when k3 >= 0, else qtf. k3 is off by default (BM25Similarity.java:79-81 passes -1f; StableTflSimilarity.java:65-66 DEFAULT_K3 = -1f).
5. The wrapper. PerFieldSimilarityWrapper.java:31-48 overrides only computeNorm (:36-39) and scorer (:41-44), both final delegations to get(field). It does not override computeQueryTermWeight, so step 2 lands on the linear default and never reaches the per-field BM25Similarity.

One document, so idf = ln(1 + 0.5 / 1.5) = 0.2877 and the tf part is 1 / (1 + 1.2) = 0.4545: base score 0.13076. Bare k3 similarity: × 16/9 = 0.23247036. Behind the wrapper: × 2 = 0.26152915, a ratio of 1.125.

• Duplicate MUST clauses: +(f:w)^2.0 +f:x instead of +(f:w)^1.8 +f:x (k3 = 8).
• Classic QueryParser text w w w: (f:w)^3.0 instead of (f:w)^2.4545455 (k3 = 8).
• StableTflSimilarity with k3: (f:w)^2.0 instead of (f:w)^1.7777778.
• 50 repeats of one term plus two other terms (k3 = 8): 10.250272 behind the wrapper versus 2.1188598 bare, a factor of 4.84; the repeated term weighs 50 instead of 9 × 50 / 58 = 7.76.
• Ranking: documents A apple, B banana and three fillers apple filler (k3 = 7). For apple repeated n times plus banana, the bare similarity and a wrapper that forwards the hook rank B first for n = 2 to 6; the stock wrapper ranks A first for n = 5 and 6.

• Who. Users of main or branch_11x who build BM25Similarity or StableTflSimilarity with k3 >= 0 (off by default) and set it through a PerFieldSimilarityWrapper, the documented route for per-field similarities and parameters (Similarity.java:62-66; CHANGES.txt:16128-16130, 4.0.0-ALPHA: "implement PerFieldSimilarityWrapper with your per-field logic").
• Trigger. A BooleanQuery with a repeated unboosted SHOULD clause (with minimumNumberShouldMatch <= 1) or MUST clause on one field that reaches the duplicate-merging step (earlier rewrite rules can replace the query first; for example a MUST clause that is also MUST_NOT becomes MatchNoDocsQuery): query-parser text that repeats a word, or programmatic queries built from verbose text, the queries k3 is meant for. Any index size; one document is enough.
• Effect. Silent wrong results: scores, and so ranking, equal the k3-disabled ones. Nothing throws. The matching set does not change, and no score goes above the k3-disabled score, because k3 can only lower a weight. The repeated term is over-weighted by qtf / w(qtf): 1.125 for 2 repeats with k3 = 7, 6.44 for 50 repeats with k3 = 8.
• Not claimed. A repeated clause that spans several fields (for example a duplicated DisjunctionMaxQuery) has no single correct delegate. MultiSimilarity and the test framework's AssertingSimilarity and RandomSimilarity also inherit the linear default; they are related, but what MultiSimilarity should return is unclear and the others are test-only.
• How likely: plausible once k3 is on: the wrapper is the standard per-field route, while users who set a bare similarity never hit it.

ByteBuffersDirectory, plain IndexSearchers, every similarity set explicitly. The test itself draws nothing random; the test framework still picks a random default codec, locale and time zone, and seeds DEADBEEF and 5EED1234 give identical numbers. The class sits in org.apache.lucene.search only to be next to Lucene's own tests.

The verifiers' longer tests show the same defect for MUST clauses, StableTflSimilarity and QueryParser input; they are not part of this repro.

Regression test: this same test; it passes with fix A (and with fix C).

# $REPRO = the directory holding TestPerFieldWrapperK3.java (this bug's repro-minimal/src/org/apache/lucene/search/)
cp $REPRO/TestPerFieldWrapperK3.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestPerFieldWrapperK3 -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
> Task :lucene:core:test FAILED

TestPerFieldWrapperK3 > testWrappedBM25AppliesK3 FAILED
    java.lang.AssertionError: expected:<0.23247036> but was:<0.26152915>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:E003A562A2B2A7D3]:0)
        at org.junit.Assert.assertEquals(Assert.java:701)
        at org.apache.lucene.search.TestPerFieldWrapperK3.testWrappedBM25AppliesK3(TestPerFieldWrapperK3.java:61)

1 test completed, 1 failed

• The k3 contract. BM25Similarity.java:44-46: "When duplicate terms appear in a query, their boost is computed as ((k3 + 1) * qtf) / (k3 + qtf) instead of linear summing." StableTflSimilarity.java:103-105 says the same. No javadoc, CHANGES or MIGRATE text says k3 needs a bare similarity on the searcher.
• The wrapper's contract. PerFieldSimilarityWrapper.java:24-27: subclasses implement get(String) "to return an appropriate Similarity (for example, using field-specific parameter values) for the field". A wrapper whose get() returns S for every field should behave like S, the argument of LUCENE-4559 (#5625).
• Precedent. In 6.x the wrapper did not forward the field-less queryNorm and coord; LUCENE-7395 / SOLR-9315 fixed it under 6.2.0 Bug Fixes (CHANGES.txt:9006-9008, "Fix PerFieldSimilarityWrapper to also delegate query norm and coordination factor using a default similarity added as ctor param", Uwe Schindler and Sascha Markus); jpountz called the old default "trappy" on #8448. Those hooks were removed in 7.0; computeQueryTermWeight is the first field-less hook added to Similarity since.
• In-tree convention. IDFValueSource.asTFIDF (lucene/queries/.../IDFValueSource.java:62-72) loops while (sim instanceof PerFieldSimilarityWrapper pfsw) sim = pfsw.get(field); to reach the field's similarity.
• No decision, no test. The #15818 thread (9 comments, 13 inline review comments) and #16243 never mention wrappers or per-field configuration. The k3 tests use a bare similarity (TestBooleanRewrites.java:1137-1171). testDefaultSimilarityPreservesLinearDedup (:1192) runs through the test framework's wrapper chain but passes either way, because no RandomSimilarity member enables k3.
• Context. On #14433 (2025), rmuir wrote in review that a field-less Similarity method is "impossible for someone to implement correctly if they customize just one field" (a different, unmerged method).

The strongest case: the hook is documented as field-less with a linear default and "Subclasses may override this"; PerFieldSimilarityWrapper only delegates the methods that carry a field; the hook is not final, so a wrapper subclass can forward it itself (that workaround works in the verifiers' controls); and both APIs are @lucene.experimental on unreleased code. A maintainer could answer "document that wrappers must override it".

Why it fails: the k3 promise has no condition and nothing documents the override requirement; the project treated the same non-forwarding of field-less hooks as a bug in LUCENE-7395; a field-less override cannot express the per-field parameters the wrapper exists for, so it only works when every field shares one k3; and no discussion or test shows the gap was considered. The evidence supports raising it as a pre-release bug, but it cannot show what the maintainers intend: they may still decide that a javadoc note or a wrapper override is the intended resolution.

Options A and C were run through gradle on 645d4fe14a20. A keeps per-field k3 values and is preferred; C needs no new API but special-cases one similarity class in BooleanQuery; B follows the 6.2 precedent but allows one k3 only; D only documents the trap. The hook is @lucene.experimental and unreleased, so maintainers may also prefer another API shape.

Similarity gets computeQueryTermWeight(String field, int queryTermFrequency), which by default returns the field-less method, so BM25Similarity and StableTflSimilarity need no change. PerFieldSimilarityWrapper overrides it as a final delegation to get(field), like computeNorm and scorer. BooleanQuery.deduplicateClauses, wherever it calls the hook today, collects that clause's fields with a QueryVisitor and calls the field-aware method when there is exactly one (3 files, +44/-1). As on stock main, once an occur type has any duplicate, every distinct unboosted clause of that type gets its weight from the hook (singletons with qtf = 1 included), so this covers all of them, not only the merged ones.

// PerFieldSimilarityWrapper
@Override
public final float computeQueryTermWeight(String field, int queryTermFrequency) {
  return get(field).computeQueryTermWeight(field, queryTermFrequency);
}

// BooleanQuery.deduplicateClauses: one field -> the field-aware method
return fields.size() == 1
    ? similarity.computeQueryTermWeight(fields.iterator().next(), queryTermFrequency)
    : similarity.computeQueryTermWeight(queryTermFrequency);

Trade-offs

• Per-field k3. Different fields can use different k3 values, which is what the wrapper is for.
• API. One new @lucene.experimental method. Since the hook is unreleased, a variant replaces the field-less signature instead of adding an overload; the multi-field case then needs a defined argument. That variant was not run.
• Multi-field clauses. A merged clause over several fields keeps the field-less path, as today: linear behind the stock wrapper, or whatever a wrapper subclass's own override of computeQueryTermWeight(int) returns.
• Field detection. The field comes from QueryVisitor: TermQuery reports it, but a query that reports no field (such as MatchAllDocsQuery) or a custom query with incomplete visitor information falls back to the field-less path.
• Existing overrides. A wrapper subclass that already overrides the field-less hook (the workaround below) is bypassed for single-field clauses, because the new wrapper method is final and goes to get(field); with the usual workaround body the result is the same.
• Cost. Once an occur type has duplicates, one QueryVisitor walk per distinct unboosted clause of that type, only on the dedup path; no benchmark was run.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-31):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off;
• TestBooleanRewrites, TestBM25Similarity, TestStableTflSimilarity, TestSimilarityProvider, TestSimilarity2 and TestBooleanQuery pass (6 classes, 89 tests, 0 failures);
• every test class of core's org.apache.lucene.search packages passes (214 classes, 2,339 tests, 0 failures, 49 skipped); the copied regression test was removed before this run and the closest-suites run, so it is not in these counts (it passed on its own above);
• checkGoogleJavaFormat, ecjLintMain, forbiddenApisMain and renderJavadoc for core pass;
• Earlier, the finder's equivalent sketch passed the finder's test, TestBooleanRewrites + TestBM25Similarity (37 tests) and the security probe (wrapped scores equal the bare k3 scores, match sets unchanged).

For each clause whose weight comes from the hook and that targets a single field (found with a QueryVisitor, as in A), BooleanQuery does what IDFValueSource.asTFIDF does, while (similarity instanceof PerFieldSimilarityWrapper perField) similarity = perField.get(field);, before calling computeQueryTermWeight(qtf) (BooleanQuery.java only, +25/-1).

Trade-offs

No public API change and the smallest diff, but BooleanQuery special-cases one Similarity subclass, and a custom delegating similarity that does not extend PerFieldSimilarityWrapper is not resolved. It also unwraps past a wrapper subclass's own override of the field-less hook, so the workaround below is bypassed for single-field clauses (same result with the usual workaround body), and the QueryVisitor fallback of A applies.

What the checks validated

Applied to the same scratch worktree (BooleanQuery.java only, +25/-1), gradle, seed DEADBEEF (2026-10-31):

• the minimal test passes with assertions on and off;
• the same 6 closest suites pass (89 tests, 0 failures);
• checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for core pass. The wider package run was done for option A only.

Give PerFieldSimilarityWrapper a default similarity (constructor argument) and forward the field-less computeQueryTermWeight(int) to it, as 6.2 did for queryNorm and coord.

Trade-offs

Simple and precedented, but one k3 for all fields, and every subclass must supply the default or keep today's behaviour.

A javadoc note that wrappers must forward the hook. Until a fix lands, users can override it in their wrapper subclass: public float computeQueryTermWeight(int qtf) { return bm25.computeQueryTermWeight(qtf); } (the verifiers' forwarding controls pass).

Trade-offs

Works only when every field shares one k3, and leaves the trap for the next user.

Method: non-formal (code reading plus a differential test). The hunt unit covered the similarities, with the #15818 change as its main target.

• Reading. The duplicate-clause boost path: deduplicateClauses, its only call of computeQueryTermWeight, and every Similarity subclass that overrides or inherits it.
• Differential. The same index and query on a searcher with the bare BM25Similarity(k3 = 7) and on one with a PerFieldSimilarityWrapper returning that instance: rewrites ^1.7777778 versus ^2.0, scores 0.5601189 versus 0.63013375 (two documents), failing with assertions on and off, and 10 of 10 iterations with assertions on.
• Minimal repro. The test above is that test cut down to one document and one assertion.