ExpressionRescorer.explain shows _score as 0 and computes the expression from it (since 7.0)

Expression.getRescorer re-sorts first-pass hits by an expression, and its explain is documented to give "more detail by showing the value of each variable". Since 7.0.0 (LUCENE-7741) ExpressionRescorer.explain hands SortRescorer's explanation, whose value is always 0.0, to the expression as the score explanation, so the expression is evaluated with _score = 0 and the _score detail is that 0.0 node. For _score * 100 + popularity, a hit with first-pass score 1.0 and popularity 7 is sorted by 107.0 but explained as 7.0 = _score * 100 + popularity; the Expression javadoc's example sqrt(_score) + ln(popularity) is explained as ln(popularity), and popularity / _score as +Infinity. The rescored order, sort values and scores are correct. 6.5.1-6.6.6 show _score correctly; 7.0.0 to 10.5.2 and main do not, and putting back only the old ExpressionRescorer.java on 7.0.0 removes the failure. Passing the first-pass explanation, labelled "first pass score", fixes the repro and passes all 171 expressions tests. Low priority: an explain-only defect in an experimental, rarely used API.

1. rescore() (SortRescorer.java:38-93) sets the scorer's score to each hit's first-pass score (:77, score.score = hit.score), so the sort value is 1.0 * 100 + 7 = 107.0, and it keeps ScoreDoc.score at 1.0 (:89-91).
2. ExpressionRescorer.explain (ExpressionRescorer.java:49-62) first calls SortRescorer.explain (:52). That re-runs the rescore for the one hit and returns Explanation.match(0.0f, "sort field values for sort=...", subs) (SortRescorer.java:125), whose details are the first-pass explanation and a 0.0 node with the sort value in its text (:118-120).
3. It passes that 0.0 node to ExpressionValueSource.explain as the score explanation (ExpressionRescorer.java:59-61).
4. ExpressionValueSource.explain evaluates the expression with the scores bound to DoubleValuesSource.constant(scoreExplanation.getValue()) = 0.0 (ExpressionValueSource.java:191-195): 0 * 100 + 7 = 7.0. For _score, DoubleValuesSource.SCORES.explain returns the score explanation itself (DoubleValuesSource.java:376-377), so the _score detail is the 0.0 node.

What you see on main, with assertions on and off:

7.0 = _score * 100 + popularity, computed from:
  0.0 = sort field values for sort=<expr(_score * 100 + popularity)>!     <- the _score detail
    1.0 = first pass score
      1.0 = *:*
    0.0 = sort field <expr(_score * 100 + popularity)>! value=107.0
  7.0 = double(popularity)

(The arrow is ours.) The right value, 107.0, appears only as text inside a node whose value is 0.0.

• sqrt(_score) + ln(popularity), the Expression javadoc's example, is explained as ln(popularity) (1.9459 for popularity 7, against a sort value of 2.9459).
• popularity / _score is explained as +Infinity, and ln(_score) + popularity as -Infinity; the rescored values stay finite and nothing throws.
• The explained values can order hits differently from the rescore: on 7.0.0 the rescore ranks three hits 25.13, 25.11, 24.43, and their explanations say 6.0, 5.0, 7.0.
• Expressions that do not read _score explain correctly, and the expression's own DoubleValuesSource.explain, given the real first-pass explanation, returns the sort value.

• Who. Applications that call explain on a rescorer from Expression.getRescorer whose expression reads _score, directly or through a binding. For every explained hit the _score detail is SortRescorer's 0.0 node instead of the first-pass explanation; when the first-pass score is not 0 (true of every hit in our runs), the _score value shown is wrong, and so is the expression value wherever it depends on _score. It reproduced with a plain IndexSearcher, the default BM25 similarity and the default codec (the release probe, 3 documents) and in randomized test fixtures of 1 and 6 documents, on main with assertions on and off; by reading, nothing on the explain path depends on the codec or the index size.
• What is wrong. Only the Explanation: the _score value shown, and the expression value computed from it.
• What is right. The rescored order, the FieldDoc sort values, the ScoreDoc scores and isMatch() (checked by a probe on main with assertions on and off).
• Opt-in, rarely used API. Expression and ExpressionRescorer are @lucene.experimental; ExpressionRescorer is package-private and reached only through the public getRescorer. No Lucene code outside tests calls getRescorer.
• Likelihood: plausible. Anyone who explains a _score-based expression rescore sees it at once, but rescorer explanations are a rare diagnostic, and nobody has reported it since 7.0.0 (2017).
• Workaround. Explain the expression directly: expression.getDoubleValuesSource(bindings).explain(leaf, docInLeaf, firstPassExplanation) returns the sort value (an independent reproducer's control test).

One test class, one method, one document, single thread; the seed does not matter (MatchAllDocsQuery scores every hit 1.0, and the searcher is a plain IndexSearcher). It asserts only the _score detail, the value the class javadoc promises, not the root. getDetails()[0] is the _score detail because _score is the expression's first variable.

Regression test: this same test. It fails on stock main and passes with each of the three fixes that were run (the 6.x restore is by reading only).

Nit: new MatchAllDocsQuery() is deprecated on main in favour of MatchAllDocsQuery.INSTANCE (also in 10.6.0). The test passes the expressions module's format, ECJ and forbidden-apis checks as is.

cp $REPRO/TestExpressionRescorerExplainScore.java $LUCENE/lucene/expressions/src/test/org/apache/lucene/expressions/
cd $LUCENE && ./gradlew :lucene:expressions:test --tests org.apache.lucene.expressions.TestExpressionRescorerExplainScore -Ptests.seed=DEADBEEF -Ptests.asserts=true
# same failure with -Ptests.asserts=false

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true
TestExpressionRescorerExplainScore > testScoreDetailIsFirstPassScore FAILED
    java.lang.AssertionError: 7.0 = _score * 100 + popularity, computed from:
      0.0 = sort field values for sort=<expr(_score * 100 + popularity)>!
        1.0 = first pass score
          1.0 = *:*
        0.0 = sort field <expr(_score * 100 + popularity)>! value=107.0
      7.0 = double(popularity)
     expected:<1.0> but was:<0.0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:A8ECC047760D62A6]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:555)
        at org.apache.lucene.expressions.TestExpressionRescorerExplainScore.testScoreDetailIsFirstPassScore(TestExpressionRescorerExplainScore.java:51)
1 test completed, 1 failed

# -Ptests.asserts=false: the same message

# with the recommended fix the explanation reads
107.0 = _score * 100 + popularity, computed from:
  1.0 = first pass score
    1.0 = *:*
  7.0 = double(popularity)

• The class javadoc (ExpressionRescorer.java:29-33): functionally the same as SortRescorer "except for the explain method which gives more detail by showing the value of each variable".
• The Rescorer contract (Rescorer.java:45): "Explains how the score for the specified document was computed." In the #14776 review (2025), vigyasharma wrote that "Rescorer#explain API is meant to explain how the _Rescorer_ would score this docId for given searcher, when rescore() is called".
• The original design (LUCENE-5545, #6608, mikemccand, 2014): "explain() includes the values of each of the bindings ... so you can see the values that led to the overall expression".
• A regression. Up to 6.6.6 the code bound the scores to firstPassExplanation.getValue() and printed variable "_score" with the first-pass score. LUCENE-7741 (fb24d451cc98) replaced that loop with the call that passes SortRescorer's node. Its issue thread (#8792) never mentions rescorers, and the commit message gives no reason for this change.
• The code defines the parameter as the score's explanation. SCORES.explain returns it, and the default DoubleValuesSource.explain evaluates with scoreExplanation.getValue() as the score (DoubleValuesSource.java:80-89).
• Why nobody noticed. The same commit changed the upstream test from expl.contains("= variable \"popularity\"") to expl.contains("= double(popularity)"); TestExpressionRescorer.testBasic checks only substrings, never a value.
• Precedent. Wrong values in an explanation are fixed as bugs, for example "GITHUB#13691: Fix incorrect exponent value in explain of SigmoidFunction" under 9.12.0 Bug Fixes.

The strongest case for "works as intended" (rated moderate by the cross-model reviews, weak by the devil's-advocate verifier): SortRescorer's 0.0 root is a deliberate placeholder, since a re-sort keeps ScoreDoc.score and puts the keys in FieldDoc.fields; ExpressionRescorer is documented as functionally the same as SortRescorer; the APIs are @lucene.experimental; nothing promises that the root equals FieldDoc.fields[0]; DoubleValuesSource.explain has no @param for scoreExplanation; and LUCENE-7741 may have passed SortRescorer's node on purpose to keep the "first pass score" text that testBasic asserts.

Why it fails: that explains a 0.0 root, not binding the documented _score variable to 0. Since 7.0 the root is not the placeholder either: it claims 7.0 = _score * 100 + popularity, computed from:, false arithmetic for the real _score, so it follows no convention. Keeping the "first pass score" text does not need the 0 value: the recommended fix keeps it and testBasic passes. The rest (experimental, explain-only) argues for Low priority, which is how it is classed, not for intent. The root never equalled the sort value, 6.6.6 included. This is evidence that the behaviour conflicts with the javadoc and the 6.x code, not proof of what LUCENE-7741 intended; the fix shape is the maintainers' choice (the recommended fix gives an expression-valued root, the 6.x restore keeps SortRescorer's 0.0 root).

Three fixes were run through gradle in a scratch worktree of 645d4fe14a20; the recommended one is also lint-clean. A fourth, restoring the 6.x shape, is by reading only.

ExpressionRescorer.java +5/-3: drop the super.explain(...) call and pass this node as the score explanation:

// rescore() computes the expression with _score = the first-pass score, so explain it that way
Explanation firstPassScore =
    Explanation.match(
        firstPassExplanation.getValue(), "first pass score", firstPassExplanation);
return expression
    .getDoubleValuesSource(bindings)
    .explain(readerContext, docIDInSegment, firstPassScore);

Trade-offs

• The root becomes the value the rescore sorted by (107.0); the _score detail is 1.0 = first pass score with the first-pass explanation under it; one detail per variable.
• Keeps the "= first pass score" text that TestExpressionRescorer.testBasic asserts. Drops SortRescorer's "sort field ... value=" node, whose value is now the root, and the one-hit rescore that built it.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle --offline, OpenJDK 25.0.3, seed DEADBEEF. Stock: the repro fails with assertions on and off.
• With the fix: the whole :lucene:expressions:test (14 suites, 171 tests, the repro and TestExpressionRescorer included) passes with 0 failures with assertions on and off; the module's checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass.

The hunt's fix, ExpressionRescorer.java +13/-3: evaluate with firstPassExplanation, then return the expression's root with SortRescorer's node appended to its details.

Trade-offs

Keeps everything SortRescorer showed, but the tree gains a 0.0-valued "computed from" detail the value does not come from, shows the first-pass explanation twice, and still runs the one-hit rescore.

What the checks validated

The whole :lucene:expressions:test with assertions on: 171 tests, 0 failures. Earlier, through the standalone runner: the repro and the 170 upstream expressions tests pass, and the version-portable probe passes 6 of 6 on main.

Change only the last argument of the explain call.

Trade-offs

The _score detail loses its "first pass score" label, so TestExpressionRescorer.testBasic fails on assertTrue(expl.contains("= first pass score")); the super.explain result becomes unused. The recommended fix makes this change with the label and also drops the then unused super.explain call.

What the checks validated

TestExpressionRescorer and the repro with assertions on: 5 tests, 1 failure (testBasic, TestExpressionRescorer.java:123); the repro passes.

Return SortRescorer's 0.0 root and append one detail per variable computed from the first-pass score, as 6.6.6 did.

Trade-offs

Matches "functionally the same as SortRescorer" literally, but keeps a root that states 0.0 for a hit sorted by 107.0; a committer who prefers the placeholder root may still pick it.

Method: code reading (the hunt's non-formal tier).

• Lead. In a hunt unit on the expressions compiler, a secondary contract review checked ExpressionValueSource.explain and getSortField against the values they compute. Tracing the explain path showed that ExpressionRescorer.explain passes SortRescorer's explanation, whose value is always 0.0, as the score explanation.
• Confirmation. The finder wrote a probe and a public-API JUnit test on the stock main jars (rescored 107.0, explained 7.0, with assertions on and off) and a main() that failed on 10.5.2, 9.12.3 and 8.11.4.
• Then executing cross-model reviews (OpenAI GPT-5.6), an independent reproducer, a devil's-advocate verifier (who ran 6.6.6 and found the regression), an adjudication, 12 version-history runs (8 releases from 6.5.1 to 10.5.2, main with assertions on and off, and two shadow checks), a prior-report search, a security screen with its own probe, a repro minimization and this doc's gradle runs followed.