DoubleValues.withDefault's block-max bound ignores the default, so FunctionScoreQuery can skip the true top hit

DoubleValues.withDefault(in, missingValue) (LUCENE-7737, Lucene 7.0) returns missingValue for every document the wrapped instance has no value for. #16431 (unreleased, 11.0.0) added block-max bounds to DoubleValues and, in the same diff, a withDefault override of getMaxScore that returns the wrapped instance's bound unchanged. For a field source on a skip-indexed field that bound is the skip-index block's largest stored value, which only counts documents that have a value. When the default is larger, FunctionScoreQuery multiplies the too-low bound into its scorer's bound, and a top-k BooleanQuery skips the block that holds the document without a value, even when it is the true top hit. With the default codec and IndexSearcher.search(q, 1), 12,288 documents return doc 0 (score 50.0018) instead of doc 9000 (score 100). It needs an application-written source that uses withDefault this way; no stock source does. Only main and branch_11x have it. Letting the bound also cover the default fixes it.

1. The default. DoubleValues.java:71-98: withDefault "Wrap[s] a DoubleValues instance, returning a default if the wrapped instance has no value" (:71). advanceExact stores the wrapped answer and returns true (:83-86); doubleValue() returns hasValue ? in.doubleValue() : missingValue (:78-80).
2. The bound. DoubleValues.java:93-96: getMaxScore(upTo) returns in.getMaxScore(upTo). getMinScore is not overridden, so the lower bound stays -Infinity (:63-69), which is sound; only the upper bound is wrong.
3. Where the finite bound comes from. fromLongField is fromField(field, v -> (double) v, Monotonicity.INCREASING) and fromIntField returns it (DoubleValuesSource.java:340-347). Its values read the field's DocValuesSkipper (:528-529); advanceShallow returns the current skip interval's last doc (:541-550) and getMaxScore the decoded skipper.maxValue(level) of the first level that covers upTo (:571-584). That covers only documents that have a value, which is right for the plain source: there a document without a value scores 0 (FunctionScoreQuery.java:243-252). Inside withDefault it produces missingValue instead.
4. The scorer bound. FunctionScoreQuery.java:261-269 returns innerMaxScore * valueMaxScore * boost (it returns +Infinity outright when the value bound is infinite, :265-267); its advanceShallow is the minimum of the wrapped query's and the values' (:254-259). With MatchAllDocsQuery the inner bound is 1.0.
5. The pruning. For SHOULD clauses under ScoreMode.TOP_SCORES, BooleanScorerSupplier.java:315-328 builds a MaxScoreBulkScorer. Per window it reads each clause's getMaxScore(windowMax - 1) (MaxScoreBulkScorer.java:439); partitionScorers() makes clauses non-essential while their summed bound is below minCompetitiveScore (:474-512, :502), and with no essential clause the window is skipped ("No matches in this window", :133-138). IndexSearcher.search(q, n) collects with a 1,000-hit threshold (IndexSearcher.java:100, :599), so TOP_SCORES is on (TopScoreDocCollector.java:58-60).

Three skip-index blocks: docs 0-4095, 4096-8191 and 8192-12287. withDefault(fromLongField("price"), 100) reports 50.0, 9.0 and 9.0; doc 9000, in the third block, produces 100.0. The function clause's bound there is 1.0 x 9.0 = 9.0, the term clause's 0.0017790452. explain gives doc 9000 100.0 and doc 0 50.001778; the exhaustive top hit is doc 9000 (100.0), search(q, 1) returns doc 0 (50.001778, totalHits "4096+ hits"), and count(q) is 12,288. With the fix the three bounds are 100.0 and search(q, 1) returns doc 9000.

Affected (executed): SHOULD + SHOULD disjunctions, MUST function query + SHOULD term, and those under FILTER or MUST_NOT clauses. Not affected (read): a top-level FunctionScoreQuery alone (its FilterScorer does not override setMinCompetitiveScore, so nothing is skipped), and the stock sources: boostByValue wraps the bounded source in withDefault but returns a new DoubleValues without bounds (FunctionScoreQuery.java:294-307); boostByQuery (:371-386) and the five spatial-extras callers wrap DoubleValues that have no bounds.

• Who. Applications on main or branch_11x (unreleased 11.0.0) that write their own DoubleValuesSource returning DoubleValues.withDefault(values, d) around a field source with a finite bound (fromLongField, fromIntField, fromFloatField, fromDoubleField, or fromField with a declared monotonicity) on a numeric field with the opt-in doc-values skip index (NumericDocValuesField.indexedField), and use it in a FunctionScoreQuery that is one scoring clause next to another, searched top-k (the default search(q, n)).
• Trigger. A document without a value in a skip block whose largest stored value is below the default d. A default above the real values (100 over prices 0-9 in the tests) makes that likely; a default of 0 never triggers it for non-negative values. For a fixed index, query and collector the outcome is deterministic: no randomness, no timing.
• Effect. Silent wrong results: documents without a value, which score the default, are missing from the top hits, including the true top hit; lower-scoring documents are returned in their place. No exception, with assertions on and off, default codec and searcher.
• Never a superset. In 5 probe runs of 55,848 checks each, no non-matching, filtered-out, excluded or deleted document was returned, and every returned score was exact.
• Scale. From a few skip-index blocks (12,288 documents in the end-to-end tests); the bound itself is wrong from 2 documents.
• How likely: rare. It needs custom source code, the opt-in skip index, a default above a block's maximum and a summing composition. No stock source does this, and the only downstream uses found (the lang-expression EmptyMemberValueSource of Elasticsearch and OpenSearch) wrap an instance without a bound (read, not tested). No release has the bug.

The default codec is pinned because the bug needs its doc-values skip index (LuceneTestCase would otherwise pick a codec from the seed); ByteBuffersDirectory, nothing random. At least one document must have a value, otherwise the field and its skipper do not exist and the bound is +Infinity.

The test shows the root cause, the bound below the produced value. The 12,288-document wrong top hit in the explanation above is the same bug; it is not part of this repro.

Regression test: this same test; it passes with the recommended fix. In a PR it could sit next to testMaxScoreBeyondTheCurrentBlock in TestDoubleValuesSource.

cp $REPRO/TestDoubleValuesWithDefaultMaxScore.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestDoubleValuesWithDefaultMaxScore -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
> Task :lucene:core:test FAILED

TestDoubleValuesWithDefaultMaxScore > testMaxScoreCoversMissingValue FAILED
    java.lang.AssertionError: doc 0 value 100.0 > getMaxScore(1) = 1.0
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:7B094C9106BA0E78]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.search.TestDoubleValuesWithDefaultMaxScore.testMaxScoreCoversMissingValue(TestDoubleValuesWithDefaultMaxScore.java:49)

# the first verifier's 12,288-document test on the same stock jars (standalone runner, seed DEADBEEF): 4 tests, 2 failed (the 2 controls pass)
testTopHitDefaultSearchApi  java.lang.AssertionError: pruned top-1 doc=0 score=50.001778 vs exhaustive doc=9000 score=100.0 expected:<9000> but was:<0>

• The contract. DoubleValues#getMaxScore: "Return the maximum score that documents between the current position and {@code upTo} can produce" (DoubleValues.java:55-58). withDefault produces missingValue for documents without a value and says they have one (:78-86). Scorer#getMaxScore has the same meaning for the scorer (Scorer.java:75-79), and Scorable#setMinCompetitiveScore lets a scorer "safely ignore" only documents that score less (Scorable.java:45-50).
• Tests. TestDoubleValuesSource.java:412-422 (testMaxScoreBeyondTheCurrentBlock, added by #16697) asserts getMinScore(upTo) <= value <= getMaxScore(upTo) for a produced value. No upstream test calls DoubleValues.withDefault.
• History. Before #16431 DoubleValues had no bounds and FunctionScoreQuery's bound was +Infinity: always safe. 10.0.0, 10.5.2 and 10.6.0 return the right top hit. The #16431 review discussed the monotonicity of user functions; no comment mentions withDefault, missing values or defaults. The author described the design as "If any layer in the nesting chain is non-monotonic or lacks skip index information, it returns Float.POSITIVE_INFINITY, making nested expressions safe by construction" (rajat315315 on #16431); withDefault is such a layer and does not. The follow-up #16697 fixed other unsound bounds of the same API as defects.

The strongest case: #16431 added the forwarding override on purpose (it is in the PR's first commit), it keeps the bound tight, and a skip-index bound only has to cover documents that have a value. A second argument: defaults are meant to be neutral (0 or 1). A third: uschindler said in the same review that a user who implements maxScore wrongly for a function like sin() has "your own problem".

Why it fails: "covers only documents with a value" is right for the plain field source, but withDefault reports a value for every document, and the javadoc has no exception for defaults; no review comment chose this trade-off, and a bound below a produced score is not a bound. Nothing documents that defaults must be neutral. The sin() remark is about user code declaring a wrong bound; here the wrong bound is Lucene's own wrapper's, and the user declared nothing. Verdict: not intended, a defect (all verifiers and the adjudication; the cross-model reviewers rated the intended argument only moderate).

Option A was built and run through gradle. B would keep more pruning but needs new API; C is safe but gives up pruning for every withDefault user. The sibling fixes in FunctionScoreQuery (LFV-203, LFV-211) do not touch this bug.

In DoubleValues.withDefault, getMaxScore (DoubleValues.java:93-96) returns max(in.getMaxScore(upTo), missingValue rounded up to float), and leaves a NaN default out (+10/-1 lines).

float maxScore = in.getMaxScore(upTo);
if (Double.isNaN(missingValue)) {
  return maxScore;
}
// documents without a value produce missingValue, so the bound must cover it too
float missing = (float) missingValue;
if (missing < missingValue) {
  missing = Math.nextUp(missing);
}
return Math.max(maxScore, missing);

Trade-offs

• Rounding. The bound is a float and the default a double; (float) rounds to nearest, so the value is rounded up when it went down. A default beyond the float range becomes +Infinity, which FunctionScoreQuery treats as no bound.
• NaN. Math.max(x, NaN) is NaN and would poison the scorer's bound; FunctionScoreQuery scores a NaN value as 0, so the wrapped bound is kept. The hunt's first diff lacked this guard; the adjudication and verifiers flagged it. That bound covers the score 0 only when it is not negative: for a block whose stored values are all negative (and likewise for a negative default) this relies on the sibling LFV-203's clamp of the bound at 0 in FunctionScoreQuery.
• Cost. The bound rises to the default for every block, also blocks where every document has a value. Sound, but it can switch off pruning for the function clause when the default is above all real values: in the probe the blocks go from 50/9/9 to 100/100/100. Releases never pruned here at all (+Infinity). No benchmark was run.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-31):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off;
• core TestDoubleValuesSource and TestMaxScoreBulkScorer (26 tests) and queries TestFunctionScoreQuery, TestFunctionQuerySort, TestValueSources and TestFunctionScoreExplanations (65 tests): 0 failures;
• the whole queries module (512 tests) and spatial-extras module (117 tests, home of the five spatial withDefault callers): 0 failures; core checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass;
• on the built 645d4fe14a20 jars the 12,288-document test fails 2 of 4 on stock (controls pass) and passes 4 of 4 with the fix, assertions on and off; 10.6.0 passes 4 of 4.

The skipper knows per interval how many documents have a value (DocValuesSkipper#docCount(level)). Compared against the doc range the bound covers, which starts at the shallow-advance target and can begin before minDocID(level) (a document without a value may lie before the interval, DocValuesSkipper.java:37-38), it shows a range without gaps, which could keep the tight bound. The interval's own doc range is not enough: in the minimal repro the level-0 interval is doc 1 alone, with 1 value, while the bound covers docs 0-1 and doc 0 produces 100 (executed by the fact check).

Trade-offs

withDefault only sees a DoubleValues, so this needs a new hook on DoubleValues (for example "may documents up to upTo lack a value?") that the field source implements. More pruning for large defaults, but new API surface for a wrapper. Not built or run.

Delete withDefault's getMaxScore override, so it reports the default +Infinity, as before #16431.

Trade-offs

Always safe, but it gives up the pruning #16431 added for every withDefault user, including those with a default of 0. Not run.

Method: non-formal (code reading; no formal tool witnessed it). The hunt unit covered function queries and block-max pruning.

• Reading. The unit's SMT model of the bound pipeline (fsq_bound_smt.py) has a table mapping each model element to code; while filling it, the finder checked every DoubleValues#getMaxScore override against the values it must bound. withDefault is not in the model.
• JUnit. A public-API test (10 documents, doc 5 without a price, default 100) failed on main with assertions on and off ('value 100.0 of doc 5 > getMaxScore(9) = 9.0') and passed with the fix. The finder first rated it minor and latent.
• End to end. The first and third verifiers wrote top-hit tests over several skip-index blocks (12,288 documents) and showed a wrong top hit through plain search(q, n), which turned it into a silent-wrong-results bug. The minimal repro above is the finder's test cut down to two documents.