DoubleValuesSourceRescorer passes null scores to a source that needs them, so re-ranking with DoubleValuesSource.SCORES or a _score expression throws

DoubleValuesSourceRescorer (new in 10.3.0) re-scores first-pass hits with a value from any DoubleValuesSource; its constructor accepts any source and its javadoc states no restriction. But rescore() always calls source.getValues(ctx, null) and never looks at needsScores(), while DoubleValuesSource#getValues says callers may safely pass null scores when needsScores() is false and DoubleValuesSource.SCORES says callers must not pass null. So these stock score-reading sources fail on the first hit rescore() processes: the Expression javadoc example sqrt(_score) + ln(popularity) (an AssertionError with -ea, a misleading "Unrecognized variable (_score)" with -da), SCORES itself (AssertionError or NullPointerException) and the source of FunctionScoreQuery.boostByValue (NullPointerException). explain() for the same hits returns normally, because it passes a placeholder explanation (value 0) instead of the score, so it reports values rescore() never computes. Sibling paths (RescoreTopNQuery, FunctionScoreQuery, SortRescorer, ExpressionRescorer) all supply scores; LateInteractionRescorer, the only production subclass in Lucene, uses a score-free source and is unaffected.

• Rescorer: takes the top hits of a first, cheap search and gives them new scores, usually from something more expensive. rescore(searcher, firstPassTopDocs, topN) returns the re-ranked hits; explain(searcher, firstPassExplanation, doc) explains one hit's new score.
• DoubleValuesSource: produces one double per document, for example a field value, a constant, an expression or a function of the query score. getValues(ctx, scores) returns those values for one segment; the scores argument carries the query scores to sources that use them.
• needsScores(): a source returns true when its values depend on the score. DoubleValuesSource.SCORES is the source whose value is the score; an expression with _score bound to SCORES, or FunctionScoreQuery.boostByValue's source (score times boost), also needs scores.
• DoubleValuesSourceRescorer: an abstract rescorer added in 10.3.0 (#14776). A subclass gives it a source and a combine(firstPassScore, valuePresent, sourceValue) that turns the first-pass score and the source's value into the new score. LateInteractionRescorer is the only production subclass in Lucene.

1. For each hit, rescore() calls DoubleValues values = source.getValues(ctx, null); (DoubleValuesSourceRescorer.java:73 at a9779ef3c193), then values.advanceExact(doc) (:74) and values.doubleValue() (:75). It never calls needsScores(). The hit's first-pass score is in hand, but it goes only to combine (:76).
2. getValues says callers may safely pass null when needsScores() is false (DoubleValuesSource.java:63-64). SCORES says "callers must not pass null" (:352-354) and runs assert scores != null; return scores; (:361-362).
3. So SCORES fails the assertion with -ea, and with -da returns null, which the rescorer dereferences at :74. An expression asks SCORES for its values, gets null and reports it as an unknown variable (ExpressionValueSource.java:73-80). The boostByValue source multiplies scores.doubleValue() by the boost and dereferences null (FunctionScoreQuery.java:299).
4. It happens on the first hit processed (hits go in doc-id order, so the lowest doc id), before any score is written, on every call with at least one hit; nothing on the path depends on the index size.
5. explain() passes Explanation.noMatch("DoubleValuesSource was not initialized with query scores") instead (DoubleValuesSourceRescorer.java:107-112). SCORES returns that placeholder as its explanation, so the rescorer reports "no value in DoubleValuesSource"; an expression evaluates _score as the placeholder's value, 0. explain() returns normally with a score rescore() never computes.

Four documents (body, popularity): doc 0 "contents contents" (1), doc 1 "contents" (20), doc 2 "contents of the book" (5), doc 3 "other" (7). body:contents, top 3: doc 0 = 0.22292183, doc 1 = 0.20381427, doc 2 = 0.115056425.

The expected values are what the fix computes; with assertions on, the first two throw an AssertionError at DoubleValuesSource.java:361.

A related silent form: a function-query ValueSource wrapped with asDoubleValuesSource() reports needsScores() true and turns null scores into an empty value (ValueSource.java:92-94, :224-227), so a wrapped function that reads the score sees 0 instead of throwing. ValueSource falls back to 0 on purpose and no contract says what such a function should see here, so this is an observation, not a second bug.

• Who is hit. Applications that subclass the public abstract DoubleValuesSourceRescorer and give it a source that reads the score: DoubleValuesSource.SCORES; an expression using _score (the Expression and SimpleBindings javadoc examples bind _score to SCORES); FunctionScoreQuery.boostByValue(...).getSource(); or any custom source with needsScores() true that reads its scores. A natural use is "re-rank the top hits by score combined with popularity or recency".
• Default vs opt-in. No special index, codec or searcher setting is needed: the tests reproduce under the test framework's randomized codec and similarity, the numbers probe with new IndexWriterConfig() and BM25. The opt-in part is the API itself: the class is new in 10.3.0 and its motivating uses (full-precision vectors, late interaction) read no score.
• Loud, not silent. For these stock sources the request fails with an exception; no result is silently wrong. The one silent form is a wrapped function-query ValueSource that reads the score: it sees 0.
• Workaround. Use a score-free source (for example ln(popularity)) and apply the score part in combine(firstPassScore, ...), which already receives the first-pass score. That covers any formula of the first-pass score and one source value, but not an existing expression or boost source reused as is.
• Not affected. LateInteractionRescorer (its source's needsScores() is false) and any source that does not need scores (field values, constants, vector similarity).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected rescore() to supply scores to SCORES or reject it with IllegalArgumentException, but got java.lang.AssertionError
       org.apache.lucene.search.TestDoubleValuesSourceRescorerScores.testRescoreWithScoresSource(TestDoubleValuesSourceRescorerScores.java:47)
     Caused by: java.lang.AssertionError
       org.apache.lucene.search.DoubleValuesSource$1.getValues(DoubleValuesSource.java:361)
       org.apache.lucene.search.DoubleValuesSourceRescorer.rescore(DoubleValuesSourceRescorer.java:73)
       org.apache.lucene.search.TestDoubleValuesSourceRescorerScores.testRescoreWithScoresSource(TestDoubleValuesSourceRescorerScores.java:43)

• The callee contract. DoubleValuesSource.java:63-64: "If scores are not needed to calculate the values (ie needsScores() returns false), callers may safely pass null for the scores parameter." SCORES, :352-354: "callers must not pass null as the DoubleValues parameter". The class javadoc (:43-46) says scores reach a source by "passing the resulting DoubleValues to getValues".
• No documented restriction. The class javadoc ("A Rescorer that uses provided DoubleValuesSource to rescore first pass hits", DoubleValuesSourceRescorer.java:25), the constructor (:30-32), the combine javadoc (:34-41), Rescorer and the 10.3.0 CHANGES entry ("GITHUB#14776: Add a Rescorer that uses values from provided DoubleValuesSource to re-score first pass hits.") say nothing about score-reading sources.
• explain() and rescore() disagree. Rescorer#explain: "Explains how the score for the specified document was computed." In the #14776 review the author wrote that Rescorer#explain "is meant to explain how the _Rescorer_ would score this docId for given searcher, when rescore() is called" (in reply to a question about explain() for a doc the searcher does not hold, not about scores).
• Siblings. RescoreTopNQuery branches on valuesSource.needsScores() and passes DoubleValuesSource.fromScorer(innerScorer) (RescoreTopNQuery.java:74-79, :230-236); FunctionScoreQuery passes fromScorer (FunctionScoreQuery.java:182, :239); SortRescorer feeds each hit's first-pass score to the sort (SortRescorer.java:58, :77), which is how ExpressionRescorer gives _score a value; TestExpressionRescorer binds _score to SCORES (:100-103).
• Tests never covered it. TestDoubleValuesSourceRescorer uses only DoubleValuesSource.fromIntField (:38), TestLateInteractionRescorer a score-free source; no test expects an exception for a score-reading source.
• null is normal elsewhere; the missing check is the defect. A literal search for getValues(ctx, null) at a9779ef3c193 finds 11 other non-test sites, 9 of them on a score-free source of a known kind (the source itself, or a constant, field, vector or query source the code creates), 2 in facet wrappers. Here a public constructor accepts any source and rescore() passes null without looking at needsScores().
• History. The null is in the PR's first commit ("doubleValuesSource rescorer; no explain support"); the second ("add explain impl") added the placeholder. The PR's 6 review comments never mention needsScores() or score-reading sources.

The strongest case for "intended": the placeholder text "DoubleValuesSource was not initialized with query scores" shows the author knew rescore() passes no scores; combine(firstPassScore, ...) is the designed place to mix in the first-pass score; Rescorer is @lucene.experimental; the only subclass and the tests use score-free sources.

Why it fails: the placeholder is a string inside explain(), not documentation a user reads; @lucene.experimental lets an API change, not break the contract of the class it calls; the rescorer itself breaks the caller obligation of getValues; and even if such sources were deliberately unsupported, the answer would be a clear error, as LUCENE-5500 gave SortingMergePolicy for a sort on the score, not an AssertionError, a NullPointerException or "Unrecognized variable (_score)". Verdict: a defect under either reading: the API needs score plumbing or explicit validation. Only the remedy, and with it what SCORES means in a rescorer, is open.

One fix was built and checked (A). B and C are the alternatives a maintainer may prefer. With A or B, explain() must follow the same rule as rescore(), because today it evaluates a score-reading source against a placeholder whose value is 0; C leaves that mismatch.

In rescore(), when source.needsScores() is true, build one DoubleValues whose value is the current hit's first-pass score and pass it to getValues instead of null; sources with needsScores() false still get null. In explain(), pass firstPassExplanation to source.explain(...) instead of the noMatch("DoubleValuesSource was not initialized with query scores") placeholder. The first-pass score is the only score a rescorer has for a hit and is what combine already receives; it is what SortRescorer and ExpressionRescorer give _score, and the per-hit analogue of RescoreTopNQuery's fromScorer(innerScorer).

// a source that needs scores sees the first-pass score of the hit being rescored
final float[] firstPassScore = new float[1];
final DoubleValues firstPassScores =
    source.needsScores()
        ? new DoubleValues() {
          @Override
          public double doubleValue() {
            return firstPassScore[0];
          }

          @Override
          public boolean advanceExact(int doc) {
            return true;
          }
        }
        : null;
...
  firstPassScore[0] = hit.score;
  DoubleValues values = source.getValues(ctx, firstPassScores); // was: (ctx, null)

Trade-offs

• Every source with needsScores() true sees a change: the stock ones that throw today now work, and one that tolerates null now gets the first-pass score (the wrapped function-query ValueSource that reads the score, which sees 0 today, or a custom source with its own null fallback). Sources with needsScores() false are unchanged in rescore().
• explain() passes the first-pass explanation to source.explain() for every source; stock score-free sources ignore it, but a custom source whose explain() embeds it would show a different tree.
• A design choice: with no live Scorer after the first pass, fix A makes SCORES in a rescorer mean the hit's ScoreDoc.score. The API does not promise that; maintainers may prefer B.
• explain() then reports what rescore() computes (expression: 0.4721, 3.4472, 1.9486).
• Cost per rescore() call: a one-element float[] (every source) and one anonymous DoubleValues when needsScores() is true, plus one array write per hit (also done, unused, for score-free sources); no benchmark was run.
• The open draft #16847 edits the same file (mechanically); one of the two may need a rebase.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: stock, each repro test fails 1 of 2 with assertions on and off; with the fix all three pass 2 of 2 in both modes.
• Closest suites with the fix: core TestDoubleValuesSourceRescorer, TestLateInteractionRescorer, TestSortRescorer, TestQueryRescorer, the three TestRescoreTopNQuery* and TestDoubleValuesSource; all of lucene/expressions; all of org.apache.lucene.queries.function: 35 upstream classes, 316 tests, 0 failures (2 skipped).
• ecjLint and checkGoogleJavaFormat for core, expressions and queries pass; fix.diff and fix-with-test.diff apply cleanly. Not run: the full core and queries suites. Not tested: explain() for the boost source, with a non-matching first-pass explanation, a custom explain() override, a null-tolerant source or several segments.

Throw in rescore() and explain() (or the constructor) when source.needsScores() is true, telling the caller to use a score-free source and mix in the score in combine(). LUCENE-5500 is a precedent. Not run.

Trade-offs

Removes a natural use (re-ranking by an existing _score expression or boost) and breaks sources that work today: the generic wrapper of ValueSource#asDoubleValuesSource() reports needsScores() true for every function query, even one like LongFieldSource that never reads the score, and such a wrapped source rescores fine now.

Javadoc on the class and constructor.

Trade-offs

Leaves the AssertionError, the NullPointerException and the misleading "Unrecognized variable (_score)" in place, and explain() still disagrees with rescore(). Not recommended.

Method: code reading during modelling (the hunt's non-formal tier), in a hunt unit on rescorer equivalence (DoubleValuesSourceRescorer, RescoreTopNQuery, QueryRescorer).

• The finder wrote a bounded-exhaustive translation-validation model of DoubleValuesSourceRescorer. The model has to say what the values source receives as scores; the code passes null for every source.
• The harness itself used only sources with needsScores() false, so it produced no signal for this bug. A probe with DoubleValuesSource.SCORES and with the expression _score * 2 then threw, and the finder wrote a public-API JUnit with a constant-source control.
• Found 2026-10-26 on main ea57360912ed; then checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6), an adjudication and a credibility audit.