fromDoubleField and fromFloatField read raw IEEE bits as if they sorted like the values, so one NaN hides a block's top hit

DoubleValuesSource.fromDoubleField and fromFloatField decode a stored long as raw IEEE 754 bits, the encoding of DoubleDocValuesField and FloatDocValuesField. #16431 (unreleased, 11.0.0) declared both decoders Monotonicity.INCREASING and gave the field sources block-max bounds: with a doc-values skip index, getMaxScore is the decoded largest stored long of the block and getMinScore the decoded smallest. Read as signed longs, raw bits sort like the values only for non-negative numbers; negative values sort in reverse, and a positive NaN sorts above +Infinity and every number. So one NaN makes a block's bound NaN; FunctionScoreQuery passes it on, and in a BooleanQuery of SHOULD clauses MaxScoreBulkScorer then makes the other clause required for that block, so documents that are competitive only through the function score are never collected. With 4,098 documents, the default codec and IndexSearcher.search(q, 1), the top hit is doc 1 (0.631) instead of doc 4097 (50.0). A block of negative values gets inverted bounds. Only hand-built raw-bit fields with the opt-in skip index are affected; only main and branch_11x have the code; every release returns the right hit.

1. The premise. Monotonicity.INCREASING is documented as "Increasing function" (DoubleValuesSource.java:51-58). fromDoubleField is fromField(field, Double::longBitsToDouble, Monotonicity.INCREASING) (:331), fromFloatField is fromField(field, (v) -> (double) Float.intBitsToFloat((int) v), Monotonicity.INCREASING) (:336); fromLongField is declared the same way (:341), where it is right.
2. The bounds trust it. With a skipper, getMaxScore(upTo) takes the first skipper level that covers upTo (:556-569) and returns (float) decoder.applyAsDouble(skipper.maxValue(level)) (:571-584); getMinScore does the same with minValue (:586-599). Without a skipper they stay at +/-Infinity.
3. Raw bits are not ordered like the values. As signed longs, -1.0 = 0xbff0000000000000 is smaller than -2.0 = 0xc000000000000000, and NaN = 0x7ff8000000000000 is above +Infinity = 0x7ff0000000000000 and every number. In the repro the block of docs 4096-4097 stores min = the bits of 50.0 and max = the bits of NaN, so getMaxScore(4097) is NaN and getMinScore is 50.0; block 0-4095 reports 0.5.
4. FunctionScoreQuery passes NaN on. Its scorer's getMaxScore returns +Infinity only for an infinite value bound and otherwise innerMaxScore * valueMaxScore * boost = NaN (FunctionScoreQuery.java:262-269); its score() maps the NaN document itself to 0 (:243-252).
5. The searcher makes the other clause required. IndexSearcher.search(q, n) is TOP_SCORES and raises the minimum competitive score after 1,000 hits (IndexSearcher.java:100, :599; TopScoreDocCollector.java:58-60). A BooleanQuery of two or more SHOULD clauses with minimumNumberShouldMatch <= 1 gets a MaxScoreBulkScorer (BooleanScorerSupplier.java:308-328; a larger minimumNumberShouldMatch falls back to WANDScorer, :316-321). partitionScorers() keeps a clause essential unless the running sum of bounds is below the threshold (MaxScoreBulkScorer.java:502); with NaN that < test is false, so the function clause is the only essential clause, the test at :540 is false too, and the text clause becomes required (:524-547, applied at :460-461).

• Window [0, 4096): threshold 0.0; both clauses essential (text bound 0.13076457, function bound 0.5).
• Window [4096, 4098): threshold 0.63076466 (one step above doc 1's score 0.6307646); text bound 0.0, non-essential; function bound NaN, essential; firstRequiredScorer = 0, so both clauses are required. No document in 4096-4097 matches body:b, so doc 4097 is never collected.
• Control (0.5 instead of NaN): the window holding doc 4097 has the function bound 50.0, the text clause is not required, and doc 4097 is returned.

The finder's 5,000 documents (NaN on doc 4500, 50.0 on doc 4999) return doc 1 (score 4.19) instead of doc 4999; fromFloatField fails the same way; MUST text + SHOULD function query and a disjunction nested in a conjunction also lose hits. FunctionScoreQuery alone, DisjunctionMaxQuery and, with assertions off, minimumNumberShouldMatch=2 lost no hits in the security probe; a NaN with the sign bit set (bits 0xfff8...) among positive values lost none either (448 checks), because it sorts below every non-negative value's bits. With assertions on, minimumNumberShouldMatch queries (which use WANDScorer) can throw an AssertionError from WANDScorer.scaleMaxScore instead.

An all-negative block reports its smallest value as the maximum and its largest as the minimum; a block with two or more distinct negative values reports a too-high minimum even next to positive values. getMinScore has no in-tree production caller, and FunctionScoreQuery scores negative values 0, so at query level the negative half overlaps LFV-203.

• Who. Users of main or branch_11x who index a double or float as raw IEEE bits with the opt-in skip index (NumericDocValuesField.indexedField(name, Double.doubleToRawLongBits(v)) or Float.floatToRawIntBits(v), or a field type with DocValuesSkipIndexType.RANGE) and rank it with FunctionScoreQuery over fromDoubleField/fromFloatField as one scoring clause next to another, in a top-k search: the default IndexSearcher.search(q, n) once 1,000 hits are collected, or any TOP_SCORES collector.
• Trigger. One positive NaN in a skip block, such as Double.NaN or Double.parseDouble("NaN") from unvalidated input (bits 0x7ff8000000000000). IEEE 754 does not fix the sign of a NaN that arithmetic produces; on the Linux aarch64 machine used here, 0.0/0.0 and Math.sqrt(-1.0) at run time were positive too. A NaN with the sign bit set lost no hits in the probe. A block closes after at least 4,096 documents with a value (a dense run of one repeated value can extend it). For a fixed index, query and collector the outcome is deterministic (70 of 70 bug-test runs failed over 10 iterations in a verifier's suite).
• Effect. Silent wrong results: the block's best documents disappear from the top hits, and totalHits is reported as a lower bound ("4096+") while count() is right. No exception with assertions off; default codec and searcher.
• Never a superset. In 6 probe runs of 3,024 checks each, no non-matching, deleted or filtered document was returned.
• Scale. The NaN document's block (4,096 documents by default); for a disjunction nested in a conjunction up to its 32,768-document level-1 group (329 of the 736 true top hits lost at 50,000 documents). Deleting the NaN document does not end it until its segment is rewritten (the merge itself was not executed).
• Not affected. The stock DoubleDocValuesField/FloatDocValuesField (no skip index, so the bound stays +Infinity); every release.
• How likely: rare. It needs the hand-built layout, FunctionScoreQuery next to another scoring clause, and a NaN in the field. Elasticsearch and OpenSearch do not call fromDoubleField/fromFloatField (read-only code search of their default branches, 2026-10-26).

The default codec is pinned with TestUtil.getDefaultCodec() (new Lucene104Codec()) because the bug needs its doc-values skip index (LuceneTestCase would otherwise pick a codec from the seed; its "test environment" log line names that randomized class default, Asserting(Lucene104), which the writer's setCodec overrides); ByteBuffersDirectory, a plain IndexSearcher, one thread, nothing random. Two methods, because the title claims two symptoms of one root cause; the NaN method alone carries the user-visible bug.

4,098 documents is the smallest index found for the NaN method: the NaN and the best document must sit in the second skip block, after the 1,000-hit threshold has raised the competitive score; the 12 smaller layouts tried (3 to 4,097 documents) return the right top hit, and the 7 tried with 4,098 to 8,193 documents all return the wrong one. The negative method asserts only the bound's contract (>= -1.0), not an exact value, so both fixes pass it.

Regression test: this same class; it passes with either fix.

cp $REPRO/TestRawBitsDoubleFieldBounds.java $LUCENE/lucene/queries/src/test/org/apache/lucene/queries/function/
cd $LUCENE && ./gradlew :lucene:queries:test --tests TestRawBitsDoubleFieldBounds -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
> Task :lucene:queries:test FAILED

TestRawBitsDoubleFieldBounds > testNaNValueHidesTopHit FAILED
    java.lang.AssertionError: expected:<4097> but was:<1>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:DF889CC89C4366E0]:0)
        at org.apache.lucene.queries.function.TestRawBitsDoubleFieldBounds.testNaNValueHidesTopHit(TestRawBitsDoubleFieldBounds.java:85)

TestRawBitsDoubleFieldBounds > testMaxScoreOfNegativeValues FAILED
    java.lang.AssertionError: expected getMaxScore >= -1.0 (the block's largest value), was -2.0
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1F545E63F103144B]:0)
        at org.apache.lucene.queries.function.TestRawBitsDoubleFieldBounds.testMaxScoreOfNegativeValues(TestRawBitsDoubleFieldBounds.java:64)

2 tests completed, 2 failed

# the numbers behind it (probe on the same build): exhaustive top 3 = doc 4097 (50.0), doc 1 (0.6307646), doc 0 (0.5), 4098 hits;
# search(q, 1) = doc 1 (0.6307646), totalHits 4096+; count(q) = 4098; block 4096-4097: getMaxScore = NaN

• The premise. DoubleValuesSource.java:52: INCREASING is "Increasing function". Double::longBitsToDouble over signed longs decreases over the negative half and maps the top of the range to NaN. This is Lucene's own claim about its built-in decoders, not a user lambda.
• A bound must bound. DoubleValues#getMaxScore/getMinScore "Return the maximum/minimum score that documents between the current position and upTo can produce" (DoubleValues.java:55-69); Scorer#getMaxScore likewise (Scorer.java:76-79); TOP_SCORES allows "skipping over non-competitive hits" (ScoreMode.java:28-32). The upstream test TestDoubleValuesSource#testMaxScoreBeyondTheCurrentBlock asserts getMinScore(upTo) <= value <= getMaxScore(upTo) (:412-421), but only for fromLongField (:400-403).
• NaN is a handled value. TestFunctionScoreQuery#testNaN (:255-274) indexes the bits of NaN, reads them through fromDoubleField and expects score 0; score() maps NaN to 0 on purpose (FunctionScoreQuery.java:250).
• History. Before #16431 the bound was +Infinity: useless for pruning, always safe; 10.0.0, 10.5.2 and 10.6.0 return the right hit. In the #16431 review uschindler pointed out that functions such as 1/x or -x are not increasing and asked whether only "a pure DocValuesValueSource as function" was the safe case, which led to the Monotonicity enum; the discussion was about functions applied to the field values (custom functions and expressions versus a plain doc-values source) and never mentioned NaN, negative values or the raw IEEE bit encoding. #16697 then fixed other unsafe bounds of the same feature. In #16629, which fixed the same raw-bits ordering hazard in the doc-values range query (#16573), romseygeek asked to "check that that we aren't making this assumption in other places as well".

The strongest case: a skip index on raw-bit doubles is not a supported layout, since no Lucene field class builds one, and testNaN's own explanation calls NaN "an illegal score". A weaker one: score bounds need not be value bounds.

Why it fails: NumericDocValuesField.indexedField is public API, DoubleDocValuesField documents the raw-bit encoding, and an upstream test builds exactly such a field (TestSortOptimization#testFloatSortOptimizationSkpperIndex indexes FloatDocValuesField.indexedField(field, Float.floatToRawIntBits(value))). Declaring these two factories INCREASING has an effect only on such fields; without a skipper the bound is +Infinity. The "illegal score" text is about the NaN document's own score, which FunctionScoreQuery already maps to 0. The weaker argument touches only the negative half. Two intended-behaviour reviewers (one from another model family) rated the case for "intended" weak.

Options A and B were both built and run through gradle. A is recommended as the smallest change: it restores the pre-#16431 behaviour for exactly these two sources. B is the choice if maintainers want to keep pruning on raw-bit fields. C and D are not enough on their own.

fromDoubleField and fromFloatField pass Monotonicity.NONE instead of INCREASING (DoubleValuesSource.java:331, :336; +5/-2 lines with the comments).

// raw IEEE-754 bits are not ordered like the doubles they encode (negative values are ordered
// in reverse and NaN is above +Infinity), so skip-index bounds cannot be decoded directly
return fromField(field, Double::longBitsToDouble, Monotonicity.NONE);

// same as fromDoubleField: raw float bits are not ordered like floats
return fromField(field, (v) -> (double) Float.intBitsToFloat((int) v), Monotonicity.NONE);

Trade-offs

• Pruning lost on these fields only. The bound of these two sources is +Infinity again, as before #16431 and in every release, also for blocks whose bound is correct today (non-negative values and no NaN; SMT Q2a2). The stock DoubleDocValuesField/FloatDocValuesField have no skip index, so their bound was +Infinity anyway; only hand-built skip-indexed raw-bit fields lose block-max pruning.
• Other sources unchanged. fromLongField, fromIntField and fromField with a declared monotonicity keep their bounds.
• Side effect. It also makes LFV-221 (the bound rounded to the nearest float) moot for these two sources.
• Cost. No extra work per document; those fields lose block-max pruning, so top-k queries on them can score more documents than on main today; no benchmark was run.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-31). Stock, the minimal test fails 2 of 2 with assertions on and 2 of 2 with assertions off, and the finder's 5,000-document test fails 2 of 2. With the fix:

• the minimal test: pass with assertions on (2 tests, 0 failures), pass with assertions off (2 tests, 0 failures);
• the finder's 5,000-document test: pass (2 tests, 0 failures);
• the whole :lucene:queries:test: pass (51 classes, 512 tests, 0 failures, 5 skipped);
• core TestDoubleValuesSource, TestLongValuesSource and TestMaxScoreBulkScorer: pass (32 tests, 0 failures);
• the other modules' users of fromDoubleField/fromFloatField (all of :lucene:expressions:test; facet TestMultiDoubleValuesSource, TestRangeFacetCounts, TestTaxonomyFacetValueSource; grouping TestDoubleRangeGroupSelector): pass (expressions 170 tests, 0 failures; facet 36 tests, 0 failures; grouping 7 tests, 0 failures);
• checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for core: pass.

A private flag on the field source marks the two raw-bits factories (+38/-4 lines in DoubleValuesSource.java). getMaxScore decodes the block's largest raw long when it is non-negative, else the smallest (all values negative). getMinScore decodes the smallest raw long when it is non-negative, the largest when all are negative, and returns -Infinity for mixed signs. A NaN result becomes +/-Infinity. The logic follows the sign-aware fix of the independent reproducer, which passed its 12 tests.

if (rawFloatBits) {
  // a non-negative raw maximum is the largest value; if all raw values are negative,
  // the smallest one is the largest value; NaN bounds nothing
  long maxRaw = skipper.maxValue(level);
  double max = decoder.applyAsDouble(maxRaw >= 0 ? maxRaw : skipper.minValue(level));
  return Double.isNaN(max) ? Float.POSITIVE_INFINITY : (float) max;
}

Trade-offs

• Keeps pruning for blocks without NaN; a block with a positive NaN, or for the minimum a block with mixed signs, falls back to an infinite bound.
• Positive NaN only. A NaN with the sign bit set (bits 0xfff8...) sorts among the negative longs, so getMaxScore never decodes it: for an all-negative block {-1.0, sign-bit NaN} it returns -1.0, and for a block that also holds a non-negative value it returns the largest non-negative value. The comment "NaN bounds nothing" holds for a positive NaN only (read from the diff, not run). FunctionScoreQuery scores such documents 0, and a negative bound for an all-negative block is LFV-203's clamp issue with or without NaN. No test with a sign-bit NaN was run against B.
• More code: a second private constructor and a flag that equals/hashCode include.
• Assumes the stored long is the sign-extended bit pattern, which is what DoubleDocValuesField, FloatDocValuesField and indexedField(name, Float.floatToRawIntBits(v)) store.
• Keeps the float cast, so LFV-221 still applies to fromDoubleField. No benchmark was run.

What the checks validated

None of these tests has a sign-bit NaN. Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-31). Stock, the minimal test fails 2 of 2 with assertions on and 2 of 2 with assertions off, and the finder's 5,000-document test fails 2 of 2. With the fix:

• the minimal test: pass with assertions on (2 tests, 0 failures), pass with assertions off (2 tests, 0 failures);
• the finder's 5,000-document test: pass (2 tests, 0 failures);
• the whole :lucene:queries:test: pass (51 classes, 512 tests, 0 failures, 5 skipped);
• core TestDoubleValuesSource, TestLongValuesSource and TestMaxScoreBulkScorer: pass (32 tests, 0 failures);
• the other modules' users of fromDoubleField/fromFloatField (all of :lucene:expressions:test; facet TestMultiDoubleValuesSource, TestRangeFacetCounts, TestTaxonomyFacetValueSource; grouping TestDoubleRangeGroupSelector): pass (expressions 170 tests, 0 failures; facet 36 tests, 0 failures; grouping 7 tests, 0 failures);
• checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for core: pass.
• The independent reproduction's two classes, which also cover fromFloatField, the mixed-sign minimum and a finite-bound check that the NONE fix fails by design: TestRawBitsDecoderBoundsV1 pass (7 tests, 0 failures), TestFunctionScoreNaNBoundV1 pass (5 tests, 0 failures), seed C0FFEE01.

FunctionScoreQuery's getMaxScore would treat NaN like an infinite value bound.

Trade-offs

Removes the dropped top hit (the hunt's randomized top-k-versus-exhaustive differential had no top-k mismatch left at a fix level that included this guard together with two sibling fixes), but leaves the public DoubleValues#getMaxScore/getMinScore wrong (NaN, and inverted for negative values) for every other caller. Not run as a standalone patch.

LFV-203's clamp (Math.max(0f, ...)) and LFV-211's inner-bound fix change other parts of the same return.

Trade-offs

Each alone leaves the finder's test failing 2 of 2; Math.max(0f, NaN) is NaN. With all five fixes of the hunt combined, upstream TestFunctionScoreQuery (17) and TestDoubleValuesSource (11) pass.

Method: semi-formal (the hunt's tier for a solver-witnessed hypothesis confirmed by execution). The hunt unit covered function queries and block-max pruning.

• Model. The finder encoded the decoders in floating-point/bit-vector SMT (QF_FPBV). Z3 4.12.2 and cvc5 1.4.2 both answered SAT for two raw longs (and two raw ints) in signed order whose doubles (floats) are in the opposite order, and for a block whose largest raw long decodes to NaN while it holds a positive finite value; UNSAT for "the bound is wrong although the block maximum is non-negative". Vacuity checks were SAT.
• JUnit. A directed public-API test (5,000 documents) then showed the wrong top hit on main, with assertions on and off; the minimal repro is that test cut down to 4,098 documents and a plain search(q, 1).
• Differential. The unit's randomized top-k-versus-exhaustive comparison also flags fromDoubleField with NaN values.