FunctionScoreQuery scores negative values as 0, but its block-max bound stays negative, so top-k search skips competitive documents

FunctionScoreQuery maps a negative function value to a score of 0 (LUCENE-7996, Lucene 8.0). Since #16431 (unreleased, 11.0.0) its scorer also reports a block-max bound for top-k pruning, innerMaxScore * valueMaxScore * boost, where valueMaxScore is the decoded skip-index maximum of the block. That bound has no clamp at 0, so for a block whose values are all negative it reports a negative number, below the 0 its documents really score. MaxScoreBulkScorer sums the clause bounds of a BooleanQuery of SHOULD clauses and skips windows whose sum is below the minimum competitive score, so documents made competitive by another clause are skipped; when the sum is below 0, the starting minimum, every window is skipped. With the default codec and IndexSearcher.search(q, n), a query that matches one document returns 0 hits, and a 5,000-document index returns the wrong top hit. Only main and branch_11x have it; every release returns the hit. A one-line clamp fixes it.

1. The score clamps. FunctionScoreQuery.java:243-252: score() returns (float) (factor * boost) only when factor >= 0, else 0 ("default: missing value, negative value or NaN", :250); for a negative value explain() returns a node "0 = truncated score, max of:" with a child "0.0 = minimum score" (:191-195).
2. The bound does not. FunctionScoreQuery.java:262-269 returns innerMaxScore * valueMaxScore * boost and guards only Float.isInfinite(valueMaxScore).
3. Where the negative factor comes from. fromLongField is fromField(field, v -> (double) v, Monotonicity.INCREASING) and fromIntField returns fromLongField (DoubleValuesSource.java:340-347). With a skip index, getMaxScore of their values returns the decoded skipper.maxValue(level) (or minValue for DECREASING) of the first level that covers upTo (DoubleValuesSource.java:571-584). For a decoder that really is monotone in its declared direction that is a correct bound for the values, and negative when every value of the interval decodes negative. fromFloatField and fromDoubleField are declared INCREASING too (:330-337), but their raw-bits decoders are not monotone over negative values; that is the separate LFV-189, not counted here. Only the scorer-level bound, which must cover the clamped scores, is wrong.
4. Sparse fields. For a document without a value, advanceShallow(target) moves the skipper on to the next interval and returns that interval's last doc ID (DoubleValuesSource.java:541-550), and getMaxScore then applies that interval's bound to the whole span from target (:571-584). So in a field most documents leave unset, one negative value lowers the bound for the unvalued documents before it.
5. The pruning, SHOULD + SHOULD (the minimal repro). A BooleanQuery of only SHOULD clauses (two or more) under TOP_SCORES gets a MaxScoreBulkScorer (BooleanScorerSupplier.java:210-232, :315-328). partitionScorers() sums the clause bounds and makes clauses non-essential while the sum is below minCompetitiveScore (MaxScoreBulkScorer.java:494-511); with no essential clause left it returns false and the window is skipped ("No matches in this window", :133-138). minCompetitiveScore starts at 0. The assertion at :454-455 sits in scoreNonEssentialClauses, which a skipped window never reaches, so assertions on do not catch it.
6. The pruning, MUST text + SHOULD function query (code reading). For required plus optional clauses booleanScorer() returns null (BooleanScorerSupplier.java:237-241) and the scorer is a ReqOptSumScorer (:174-182). Its getMaxScore adds the optional clause's bound to the required one's (ReqOptSumScorer.java:273-279), and once the collector has raised the minimum competitive score above 0 it skips every block whose summed bound is below it (:89-105, :127-128). So the negative bound hides competitive documents there too, but only after the threshold has risen: the route of the MUST + SHOULD probe below.
7. Default searcher. IndexSearcher.search(q, n) uses TopScoreDocCollectorManager(n, after, 1000) (IndexSearcher.java:100, :599), which is TOP_SCORES (TopScoreDocCollector.java:58-60) and raises the threshold after 1,000 hits.

One document (the minimal repro), instrumented: the function clause's bound is 1.0 (wrapped MatchAllDocsQuery) × -1000.0 × boost 1 = -1000.0; the text clause's bound is 0.13076457; the sum is -999.86926 < 0, so partitionScorers() returns false and nothing is collected. An exhaustive search returns doc 0 with 0.13076457 and count(q) is 1; the function query alone returns the hit with score 0.0. With the fix the function bound is 0.0 and search(q, 1) returns doc 0 = 0.13076457.

5,000 documents in one segment (the finder's test): doc 0 has delta = 1, the rest -1000, and only doc 4999 matches body:b. The second skip interval (docs 4096-4999) reports -1000 ("doc 4096 scores 0.0 > getMaxScore(4999) = -1000.0"); after 1,000 hits only scores above doc 0's 1.0 are competitive, -1000 + the text bound is below that, and plain search(q, 1) returns doc 0 (score 1.0) instead of doc 4999.

Earlier executed shapes (same code at ea57360912ed): 6,000 documents with values in [-1999, -1000] give no hit and totalHits=0 for 6,000 matches, also with a DECREASING decoder x -> -x over positive stored values; BooleanQuery(MUST body:common, SHOULD the function query) over 18,823 documents with one negative value in a sparse field lost 78 of the true top 100 with the default search(q, 100) (through ReqOptSumScorer, item 6).

• Who. Users of main or branch_11x who rank with FunctionScoreQuery over fromLongField or fromIntField, or over fromField with a decoder that really is monotone in its declared INCREASING/DECREASING direction (declaring it does not make it so; fromFloatField/fromDoubleField belong to LFV-189), on a numeric field with the opt-in doc-values skip index (NumericDocValuesField.indexedField, or a field type with DocValuesSkipIndexType.RANGE), as one scoring clause next to another: "text query plus a numeric boost", as SHOULD + SHOULD or MUST text + SHOULD boost.
• Trigger. A skip interval whose values all decode negative: a signed adjustment field, all-negative values in a small segment, a decreasing decoder such as x -> -x over positive values, or in a sparse field a single negative value. With the default codec an interval normally closes once it holds 4,096 documents with a value; the last one of a segment holds fewer, and a gap-free run of one repeated value can grow past 4,096. Such an interval loses matches when its negative bound pulls a window's summed bound below the current competitive threshold (at once when the sum is below 0, as in the minimal repro). For a fixed index, query and collector the outcome is deterministic: no randomness, no timing.
• Effect. Silent wrong results: wrong top hits or no hits, and totalHits too low (once even reported as exact: 4,705 hits for 18,823 matches) while count() is right. No exception, assertions on and off, default codec and searcher, from one document up.
• Never a superset. In 5 probe runs of 432 checks each, no non-matching, deleted or filtered document was returned.
• Not reproduced in the recorded controls (their scope, not a guarantee): FunctionScoreQuery alone (it returns the hit with score 0), boostByValue, boostByQuery and expressions (their bound is +∞), fields without a skip index and non-negative values. No release has the bug: every release tested returns the hit.
• How likely: rare. Negative values score 0 by design, so few applications store them for scoring, and they must fill a whole skip interval or be the only values of a sparse field.

The default codec is pinned because the bug needs its doc-values skip index (LuceneTestCase would otherwise pick a codec from the seed); ByteBuffersDirectory, a plain IndexSearcher, nothing random. The function clause's bound alone (-1000) pushes the summed bound below 0, so one document is enough.

The 5,000-document wrong-top-hit form in the collapsed explanation above is the same bug; it is not part of this repro.

Regression test: this same test; it passes with the recommended fix.

cp $REPRO/TestFunctionScoreQueryNegativeValueMaxScore.java $LUCENE/lucene/queries/src/test/org/apache/lucene/queries/function/
cd $LUCENE && ./gradlew :lucene:queries:test --tests TestFunctionScoreQueryNegativeValueMaxScore -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
> Task :lucene:queries:test FAILED

TestFunctionScoreQueryNegativeValueMaxScore > testNegativeValueDoesNotHideMatch FAILED
    java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:57AC69C71F23E2DD]:0)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.queries.function.TestFunctionScoreQueryNegativeValueMaxScore.testNegativeValueDoesNotHideMatch(TestFunctionScoreQueryNegativeValueMaxScore.java:60)

# the finder's 5,000-document test on the same stock build (gradle, seed DEADBEEF): 2 tests, 2 failed
testTopHitWithDynamicPruning  java.lang.AssertionError: TOP_SCORES top hit (score 1.0) differs from COMPLETE expected:<4999> but was:<0>
testMaxScoreIsUpperBound      java.lang.AssertionError: doc 4096 scores 0.0 > getMaxScore(4999) = -1000.0

• The contract. MIGRATE.md:1368-1375 ("FunctionScoreQuery maps negative values to 0", LUCENE-7996); CHANGES.txt:6732-6733 (8.0.0: "FunctionQuery and FunctionScoreQuery now return a score of 0 when the function produces a negative value"); the score() comment and explain() branch (FunctionScoreQuery.java:250, :191-195); TestFunctionScoreQuery.java:236-253 indexes -2 and runs QueryUtils.check on this very query, with a plain field, so the bound path is never reached (that test class has no indexedField at all).
• Bounds bound scores. Scorer#getMaxScore returns "the maximum score that documents between the last target ... and upTo" can have (Scorer.java:75-79); CheckHits.java:766 and AssertingScorer.java:130 assert score <= maxScore; MaxScoreBulkScorer.java:454-455 asserts "All clauses are essential if minCompetitiveScore is equal to zero", which holds only if no bound is negative (in the repro it is never reached: the window is skipped before any clause is scored, so assertions on stay silent).
• History. Before #16431 the bound was +Infinity, useless for pruning but always safe; 10.0.0, 10.5.2 and 10.6.0 return the hit. In the #16431 review uschindler noted that functions like 1/x or -x are not increasing, which led to Monotonicity.DECREASING (direction, not sign); no comment mentions the clamp or a negative bound. The follow-up #16697 fixed bounds that covered the wrong range, so upstream treats an unsafe bound of this feature as a defect.

The strongest case: in #9044 (LUCENE-7996), in a comment of 2018-10-14, Adrien Grand wrote that "FunctionScoreQuery is probably the easiest way to mistakenly create a scorer that returns negative scores, in which case one should sort by the function rather than using this function to compute scores", so negative values could be called a user mistake. A second argument: a negative DoubleValues bound is correct for the values.

Why it fails: the project had already chosen to truncate negative values to 0 ten months earlier. Alan Woodward proposed in the same issue on 2017-11-30 to "document that functions that return negative values will be truncated to 0 for scoring purposes", and a8a63464e7d (Adrien Grand, 2017-12-06) added the clamp, the migration note and testTruncateNegativeScores. The 2018 remark did not change that; all three are still on main. The remark bears on how many users hit the bug, not on whether a top-k search may drop a match that an exhaustive search returns. The second argument is true and not in dispute; it only places the defect in FunctionScoreQuery, whose bound must cover its clamped scores.

Option A was built and run through gradle. B agrees with A only when the wrapped query's bound is finite (it gives NaN under a wrapped FunctionQuery), so A is preferred; C is not recommended. The sibling fix for LFV-211 changes the other factor of the same line and does not fix this bug.

In FunctionScoreQuery's scorer, getMaxScore (FunctionScoreQuery.java:268) returns the product clamped at 0, the same rule score() applies (+2/-1 lines).

// score() maps negative values to 0, so the bound must not be negative
return Math.max(0f, innerMaxScore * valueMaxScore * boost);

Trade-offs

• Clamp after the multiply. innerMaxScore and boost are never negative (negative boosts are rejected), so when the wrapped query's bound is finite this equals clamping the value bound first. It also covers a wrapped query whose bound is +∞, such as a wrapped FunctionQuery: there the stock bound is -∞ and this fix gives 0.0 (executed).
• Pruning kept. Non-negative bounds are unchanged; DoubleValues bounds stay as they are, since they are correct for the values.
• NaN. Math.max(0f, NaN) is NaN, so a NaN bound from a NaN stored double (the separate LFV-189) passes through unchanged.
• Cost. One Math.max per bound computation; no benchmark was run.
• Scope. It fixes the negative bound only; the other defects of the same feature (LFV-211, LFV-189, LFV-214, LFV-221, and LFV-273 in the score clamp) need their own changes.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-30):

• stock: the minimal test fails with assertions on and off; the finder's 5,000-document test fails 2 of 2;
• with the fix: the minimal test passes with assertions on and off; the finder's test passes 2 of 2;
• TestFunctionScoreQuery, TestFunctionScoreExplanations, TestFunctionQuerySort and TestValueSources pass (65 tests, 0 failures), and in core TestDoubleValuesSource and TestMaxScoreBulkScorer (26 tests, 0 failures);
• the whole :lucene:queries:test passes (51 classes, 512 tests, 0 failures, 5 skipped); checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for queries pass.
• Earlier: the clamp makes every verifier's and reviewer's own test pass, and with all five fixes of this hunt combined upstream TestFunctionScoreQuery (17) and TestDoubleValuesSource (11) pass.

float valueMaxScore = Math.max(0f, scores.getMaxScore(upTo));, closer to score()'s factor >= 0 test.

Trade-offs

Passes the minimal repro (executed), but it is not equivalent to the first option: the two agree only when the wrapped query's bound is finite. With a wrapped FunctionQuery (bound +∞) it computes ∞ × 0 = NaN where the first option gives 0.0 (executed on 645d4fe14a20). One more reason to prefer the first option.

Return max(0, ...) from the field source's getMaxScore.

Trade-offs

Makes a correct value bound wrong for any other consumer of DoubleValues#getMaxScore, and leaves FunctionScoreQuery relying on every source to clamp. Not run.

LFV-211 stops multiplying by the max score of a wrapped query that runs without scores, the other factor of the same return.

Trade-offs

This repro still fails with that fix alone (2 of 2), and LFV-211's repro still fails with this clamp alone; the two compose (Math.max(0f, ...) around the product with the inner factor fixed).

Method: semi-formal (the hunt's tier for a solver-witnessed hypothesis confirmed by execution). The hunt unit covered function queries and block-max pruning.

• Model. The finder wrote the scorer's score and bound in floating-point SMT (QF_FP) and asked whether a score can exceed its bound. Z3 4.12.2 and cvc5 1.4.2 both answered SAT for a block whose maximum value is negative: the bound lies below the clamped score 0. A Java replay confirmed the witness.
• Differential. A randomized harness compared TOP_SCORES with an exhaustive search over 450-600 random indexes per fix level (skip index on 80%, several value modes); its negative-block mismatch classes disappear with the clamp, and none occur without a skip index or for boostByValue/boostByQuery.
• JUnit. A directed public-API test (5,000 documents) failed on main with assertions on and off and passed on 10.5.2; the minimal repro above is that test cut down to one document and one assertion.