NoNormalization.explain reports 1 instead of tf, so IBSimilarity explains its Distribution at tfn = 1

Normalization.NoNormalization.tfn returns tf, but NoNormalization.explain returns Explanation.match(1, "no normalization"). IBSimilarity.explain passes that explanation's value to distribution.explain as tfn, so under IB with NoNormalization the Distribution child is distribution.score(stats, 1, lambda) while the node it explains is the real score at tfn = tf. On one document "a a a" with DistributionLL and LambdaDF the node is 1.3862944 (ln 4) and the child 0.6931472 (ln 2). All four IB distribution/lambda pairs show it; NormalizationH1/H2/H3/Z do not. Under DFRSimilarity, which computes tfn itself since 8.0.0, only the "no normalization" child shows 1, next to model children computed from "3.0 = tfn". Scores and rankings are correct. The code dates from LUCENE-2959 (2011): every release from 4.0.0-ALPHA to 10.6.0 has it, and all 14 releases run show the same defect, with identical numbers within each probe. A one-line fix, Explanation.match((float) tf, "no normalization"), passes the repro and the 118 similarity tests of lucene/core. Low priority: an opt-in, experimental similarity and explain output only.

1. Normalization.tfn "Returns the normalized term frequency" and explain "Returns an explanation for the normalized term frequency" (Normalization.java:33-46). The generic explain has the value (float) tfn(stats, tf, len) (:47-56).
2. NoNormalization overrides both: tfn returns tf (:64-67), explain returns Explanation.match(1, "no normalization") (:69-72).
3. IBSimilarity.score uses the real tfn: boost * distribution.score(stats, normalization.tfn(stats, freq, docLen), lambda.lambda(stats)) (IBSimilarity.java:114-118).
4. IBSimilarity.explain adds normExpl = normalization.explain(...) as a child and then calls distribution.explain(stats, normExpl.getValue().floatValue(), lambdaExpl.getValue().floatValue()) (:125-131): it takes the explanation's value as tfn.
5. Distribution.explain "Returns the name of the model only, since both tfn and lambda are explained elsewhere" and returns score(stats, tfn, lambda) (Distribution.java:35-41). With NoNormalization that is distribution.score(stats, 1, lambda).
6. The node's own value is score(stats, freq, docLen) (IBSimilarity.java:139-140), so the top value equals the search score.
7. DFRSimilarity.explain computes tfn = normalization.tfn(...) itself (DFRSimilarity.java:139-140) for its BasicModel and AfterEffect children (:143-144) but still adds normExpl (:142), so only "1 = no normalization" is wrong there.

DistributionLL.score = -ln(lambda / (tfn + lambda)) (DistributionLL.java:33-35). One document, so LambdaDF = (1 + 1) / (1 + 1) = 1, moved down one ulp to 0.99999994 (LambdaDF.java:31-39). tfn 3 gives ln 4 = 1.3862944, the node and the search score; tfn 1 gives ln 2 = 0.6931472, the child. On the hunt's 5-document index (freq 3) the four pairs show node / child 1.7047480 / 0.9162907 (LL, DF), 1.2729657 / 0.6190392 (LL, TTF), 1.5447564 / 0.7996423 (SPL, DF) and 1.3294159 / 0.6553519 (SPL, TTF); with NormalizationH1 the child equals the node. Documents where the term occurs once look right, because there tfn = 1.

CheckHits.verifyExplanation takes a "computed as ... from:" node's own value as its combined value instead of recomputing it from the children (CheckHits.java:375, 437-438, 494-496). TestSimilarityBase runs NoNormalization but compares only the top explanation value with the score (:213-231); testLLForIB and testSPLForIB check the score only (:401-414, :468-476). TestDistributionLL and TestDistributionSPL do draw NoNormalization at random for IB (DistributionTestCase.java:97-99), but compare the score with the top explanation value and only rarely run CheckHits.verifyExplanation, which lets such a node through (BaseSimilarityTestCase.java:392-401).

• Who. Callers of IndexSearcher.explain on a field scored by IBSimilarity built with Normalization.NoNormalization: every explained hit whose frequency, as passed to the similarity, is not exactly 1 is wrong, with no other data condition: a term that occurs more than once, or a fractional sloppy-phrase frequency (a sloppy phrase with freq 0.5 gives IB nodes of 0.40546513 over 0.6931472 = DistributionLL children). Under DFRSimilarity with NoNormalization only the "no normalization" child shows 1.
• Default or opt-in. Opt-in: IB and DFR are not the default similarity, and NoNormalization is one of five normalizations; both IB constructors name it as the way to ask for no normalization. Elasticsearch and OpenSearch map the IB/DFR setting normalization: no, and Solr none, to NoNormalization (read in their sources, not run).
• Effect. A misleading explanation, not wrong results. IndexSearcher.explain is "intended to be used in developing Similarity implementations", which is the audience that reads the children: here the Distribution child is about half the real score (0.6931472 against 1.3862944).
• Scale. A one-document index is enough. Since 4.0.0-ALPHA (2012); no prior report found.
• Likelihood: plausible. Anyone using that configuration meets it on the first explained hit whose frequency is not 1, but IB without normalization is a niche choice and explanation children are rarely checked by hand.

One test class, one method, single thread. The test sets its own similarity on the writer and the searcher, so the test framework's random similarity does not apply, and ByteBuffersDirectory keeps the index in memory; the codec stays random. It sits in the similarities package but uses only public classes. Seeds DEADBEEF and 5EED1234 give the same values with assertions on and off.

It needs the org.apache.lucene.tests package (9.1.0 and later). Older releases were checked with a plain-Java probe of the same similarity configurations (a 5-document index, all four IB pairs, H1 controls; 4.0.0-ALPHA to 10.5.2, assertions off; 10.5.2 also with assertions on).

Lint: with the test and the fix in place, the core module's checkGoogleJavaFormat, forbiddenApisMain, forbiddenApisTest, ecjLintMain and ecjLintTest pass.

Regression test: this same test. It fails on stock main and passes with the fix. It checks the Distribution child only, so the alternative fix (IBSimilarity computing tfn itself) passes it too.

cp $REPRO/TestIBNoNormalizationExplain.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/similarities/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.search.similarities.TestIBNoNormalizationExplain -Ptests.seed=DEADBEEF -Ptests.asserts=true
# same failure with -Ptests.asserts=false

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestIBNoNormalizationExplain > testDistributionChildMatchesScore FAILED
    java.lang.AssertionError: 1.3862944 = score(IBSimilarity, freq=3.0), computed as boost * distribution.score(stats, normalization.tfn(stats, freq, docLen), lambda.lambda(stats)) from:
      1 = no normalization
      0.99999994 = LambdaDF, computed as (n + 1) / (N + 1) from:
        1 = n, number of documents containing term
        1 = N, total number of documents with field
      0.6931472 = DistributionLL
     expected:<1.3862943649291992> but was:<0.6931471824645996>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:96C5D42B76EDBCE2]:0)
        at org.apache.lucene.search.similarities.TestIBNoNormalizationExplain.testDistributionChildMatchesScore(TestIBNoNormalizationExplain.java:49)
1 test completed, 1 failed

# the same explanation with the fix (plain-Java probe, fixed core jar)
  1.3862944 = score(IBSimilarity, freq=3.0), computed as boost * distribution.score(stats, normalization.tfn(stats, freq, docLen), lambda.lambda(stats)) from:
    3.0 = no normalization
    0.99999994 = LambdaDF, computed as (n + 1) / (N + 1) from: ...
    1.3862944 = DistributionLL

# DFR I(n)L with NoNormalization, stock main: only the normalization child is wrong
  0.31127813 = score(DFRSimilarity, freq=3.0), computed as boost * basicModel.score(stats, tfn) * afterEffect.score(stats, tfn) from:
    1 = no normalization
    1.2451125 = BasicModelIn, computed as tfn * log2((N + 1) / (n + 0.5)) from:
      3.0 = tfn, normalized term frequency
      ...
    0.25 = AfterEffectL, computed as 1 / (tfn + 1) from:
      3.0 = tfn, normalized term frequency

• The contract. Normalization.explain "Returns an explanation for the normalized term frequency" (Normalization.java:41); the generic implementation's value is that tfn (:49); NoNormalization.tfn returns tf (:66).
• IB relies on it. Distribution.explain: tfn and lambda "are explained elsewhere" (Distribution.java:35-38). IBSimilarity.explain passes the normalization explanation's value as tfn (IBSimilarity.java:129-131), and the node text says "computed as boost * distribution.score(stats, normalization.tfn(stats, freq, docLen), lambda.lambda(stats)) from:" (:145-147). With H1, H2, H3 or Z the Distribution child equals the node at boost 1, up to float rounding (the H1 control and the hunt's 16 IB configurations with H1/H2/H3/Z).
• DFR's own tree contradicts itself: "1 = no normalization" sits next to BasicModelIn, whose own child reads "3.0 = tfn, normalized term frequency". This is the weaker of the two symptoms: under DFR no computation consumes the 1, so it can be read as a display marker; under IB the Distribution child is a value the scorer never computes.
• Explanations are a supported feature of these models. LUCENE-2959's CHANGES entry: the models "support all of lucene's features (boosts, slops, explanations, etc)" (lucene/CHANGES.txt:16108-16112).
• Precedent. Lucene files explain-only value errors under Bug Fixes: "GITHUB#13691: Fix incorrect exponent value in explain of SigmoidFunction" (9.12.0, lucene/CHANGES.txt:2705); LUCENE-7065 (6.0.0, :9496).
• History. The value 1 and the IB hand-off arrived together in cfaf91c7398 (2011); later commits on these lines are mechanical. No javadoc, test, CHANGES or MIGRATE entry or code comment says 1 is intended, nor does any upstream discussion found; the LUCENE-2959 and LUCENE-3220 threads discuss NoNormalization's name only.

The strongest case for "not a bug" (the intended-behaviour check and the cross-model "intended" review): 1 is a neutral "nothing applied" marker, the convention of the old AfterEffect.NoAfterEffect, which explained itself as 1; Explanation.match documents its value as "the contribution to the score of the document"; the classes are @lucene.experimental; explain is a debugging aid; CheckHits deliberately treats "computed as" nodes as opaque; and nobody complained in 15 years.

Why it fails: normalization is not a multiplier. NoAfterEffect's 1 matched its own score, a factor of 1; NoNormalization's output is tf, and its documented explanation is of the normalized term frequency. Even read as a marker, IB consumes the 1 as tfn, so the Distribution child explains a score that was never computed, which is nobody's "contribution to the score". The experimental tag and the debugging purpose lower the impact (Low priority); they do not make the number right. Survival with tests that check only the top value is missing coverage, not intent.

Both options were run through gradle in a scratch worktree of 645d4fe14a20. The first fixes the root cause; the second only stops IB from reading tfn off an explanation. The two were not run together; the first alone is the minimal demonstrated fix.

Normalization.java +1/-1, the same change as the hunt's fix sketch:

    @Override
    public Explanation explain(BasicStats stats, double tf, double len) {
-      return Explanation.match(1, "no normalization");
+      return Explanation.match((float) tf, "no normalization");
    }

Trade-offs

• Fixes both symptoms: the IB Distribution child and the "no normalization" child under IB and DFR.
• No score change: NoNormalization.tfn is untouched.
• Code that parses explanation text and expects "1 = no normalization" would see "3.0 = no normalization"; the new value is the documented one.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle --offline, OpenJDK 25.0.3, seed DEADBEEF. Stock: the repro fails with assertions on and off.
• With the fix: :lucene:core:test --tests 'org.apache.lucene.search.similarities.*' runs 27 suites, 119 tests (the 118 upstream similarity tests, TestSimilarityBase 27 among them, plus the repro), 0 failures, with assertions on and off. :lucene:queries:test --tests TestSpanSimilarity, the only regularly run test outside the similarities package that uses NoNormalization, passes; the @Monster Test20NewsgroupsClassification in lucene/classification also uses it, for scores only (no explain), and was not run. The core module's checkGoogleJavaFormat, forbiddenApisMain, forbiddenApisTest, ecjLintMain and ecjLintTest pass.
• A plain-Java probe on the fixed jar shows "3.0 = no normalization" and "1.3862944 = DistributionLL" under IB and "3.0 = no normalization" under DFR, with unchanged scores.

IBSimilarity.java +2/-3: double tfn = normalization.tfn(stats, freq, docLen); and distribution.explain(stats, tfn, lambdaExpl.getValue().floatValue()).

Trade-offs

Fixes the Distribution child for any Normalization, including a user subclass whose explanation value differs from its tfn. Leaves "1 = no normalization" in the IB and DFR trees (run with the explain-tree probe), yet passes the regression test, which checks the Distribution child only. Good as a follow-up to the recommended fix.

What the checks validated

The similarities package with assertions on: 119 tests, 0 failures (the repro passes).

Method: semi-formal (a differential evaluation of explanation trees on the stock jars).

• Sweep. A public-API test program dumped every explained hit with its index statistics over 79 similarity configurations (DFR, IB, DFI, Axiomatic, LM, BM25, Classic and others) and many query types, with assertions on and off: 3,178 explained hits. An independent evaluator recomputed each "computed as <formula> from:" node from its own children and checked combiners and labelled statistics: 14,554 formula nodes.
• Finding. The 104 IB score nodes of the four IB configurations with NoNormalization did not match boost * distribution.score(...) on their own children; the 16 IB configurations with H1, H2, H3 or Z did. 10 of 10 injected wrong formulas were caught, so the check is not vacuous.
• Then the finder reduced it to a JUnit test. An executing cross-model review, an independent reproducer, a second cross-model review, a devil's-advocate check, an adjudication, the registration check, an audit, the version-history runs, a prior-report search and a repro minimization followed. The same sweep found two sibling bugs with their own fix sites (LFV-920, LFV-922).