SortedDoubleDocValuesStats reads DoubleField values as raw IEEE bits, so negative values give a silently wrong min, sum and mean

DocValuesStats.SortedDoubleDocValuesStats in lucene/misc computes count, min, max, sum, mean, variance and stdev of "a sorted-numeric field storing double values" over the documents a query matches, and names no encoding. Core's DoubleField (since 9.5.0) writes exactly such a field, in the NumericUtils sortable layout, and core's SortedNumericDocValuesField javadoc recommends that layout for doubles. The stats class decodes each stored long as raw IEEE bits (Double.longBitsToDouble). The two layouts agree exactly when the sign bit is clear, so positive values are exact and every negative value is read as a different number: DoubleField -5.0 and 3.0 give min -0.8749999999999999, sum 2.125 and mean 1.0625 instead of -5.0, -2.0 and -1.0, silently, with default settings. The upstream test writes positive values only, so it cannot see it. Every release from 6.4.0 (documented recipe) and 9.5.0 (DoubleField) to 10.6.0 and main is affected. Decoding with NumericUtils.sortableLongToDouble fixes it; it changes results for anyone who wrote negative values as raw bits, so it needs a migration note (or a decoder argument).

1. The sortable layout. NumericUtils.doubleToSortableLong(value) is sortableDoubleBits(Double.doubleToLongBits(value)) (NumericUtils.java:47-48), and sortableDoubleBits(bits) is bits ^ (bits >> 63) & 0x7fffffffffffffffL (:84-85): for a negative number it flips every bit except the sign, for a non-negative number it changes nothing. The inverse is sortableLongToDouble (:56-57).
2. What the field writes. DoubleField sets DocValuesType.SORTED_NUMERIC (DoubleField.java:52), stores NumericUtils.doubleToSortableLong(value) (:73, and :105 in the setter) and reads its own value back with sortableLongToDouble (:90).
3. What the stats class reads (fix site). SortedNumericDocValuesStats opens getSortedNumericDocValues(field) (DocValuesStats.java:297); SortedDoubleDocValuesStats.doAccumulate reads docValueCount() values per document and decodes each with double val = Double.longBitsToDouble(sndv.nextValue()); (:436). Max, min, sum, the running mean and the variance all use that value (:437-450).
4. The NUMERIC sibling is right. DoubleDocValuesStats decodes Double.longBitsToDouble(ndv.longValue()) (:256), which matches DoubleDocValuesField, documented as encoding "via Double#doubleToRawLongBits" (DoubleDocValuesField.java:20-21, :39). The sorted-numeric class uses the same decode for a field type whose core writer is sortable.

-5.0 has raw bits 0xc014000000000000 and sortable bits 0xbfebffffffffffff; read as raw bits, 0xbfebffffffffffff is -0.8749999999999999. 3.0 is 0x4008000000000000 in both layouts. So for DoubleField -5.0 and 3.0 the class sees -0.8749999999999999 and 3.0.

A Z3 bit-vector/FP64 model over all 2^64 bit patterns proves: the two layouts coincide exactly when the sign bit is clear; every negative non-NaN value is read as a different number; the order among normal negatives is reversed; -0.0 and every negative subnormal but the largest-magnitude one are read as NaN (that one as -Infinity), and -Infinity as a negative subnormal; the bit flip sortableDoubleBits is its own inverse on all 2^64 patterns, so the sortable decode returns exactly the bits doubleToSortableLong started from (Double.doubleToLongBits, which canonicalises NaN).

• Executed: DoubleField -5.0 and 3.0 give min -0.8749999999999999, max 3.0, sum 2.125, mean 1.0625, variance 3.75390625 and stdev 1.9375 (expected -5.0, 3.0, -2.0, -1.0, 16.0 and 4.0). A SortedNumericDocValuesField written with the javadoc recipe NumericUtils.doubleToSortableLong gives the same wrong values. Raw bits (Double.doubleToRawLongBits, what the upstream test writes) give the right values. A SortedNumericSortField(DOUBLE) on the same DoubleField sorts -5.0, 3.0 correctly.
• Executed (an independent reviewer's test): DoubleField -2.5, 7.25 and -0.5 give min -7.999999999999999 instead of -2.5: a misread negative can come out smaller or larger than the true value.
• Executed (this doc's diagram probe, main, assertions on and off): DoubleField -0.0 and 3.0 give max NaN, sum NaN, mean NaN and min 3.0, because -0.0 is read as NaN.

Not part of this finding: the facet module's MultiDoubleValuesSource.fromDoubleField, which reads these fields the same way (another module and fix site), a wrong max() (with a positive maximum it is right unless a -0.0 or a negative subnormal is present, which can be read as NaN, as in the -0.0 case above; when every value is <= 0 it is wrong for another reason: LFV-155, the Double.MIN_VALUE seed at DocValuesStats.java:429), and the variance denominator (a separate question, not run: the sorted-numeric variance() divides by the document count(), :312-315, while doAccumulate adds one term per value, :445-450; the upstream test computes its expected value the same way, and the repro has one value per document).

• Who is hit. Users of SortedDoubleDocValuesStats (through DocValuesStatsCollectorManager, since 10.5.0, or DocValuesStatsCollector) whose field is a DoubleField or a SortedNumericDocValuesField written with NumericUtils.doubleToSortableLong, and whose matching documents hold any negative value: each one is decoded as a different number.
• Default vs opt-in. Default IndexWriterConfig, codec and searcher, public API only; two documents suffice. The class itself is opt-in: a lucene/misc utility, rarely used. It is final with a one-argument constructor, so it offers no decoder choice; the workaround is to write one's own subclass of the public abstract SortedNumericDocValuesStats (protected constructor and state) with a sortable decode, which both collector entry points accept.
• Silent, and hidden by positive data. No exception, with assertions on and off. All-positive data gives exact results, so tests with positive data, like upstream's, never see it.
• Scale. Any index size. With a single negative value among the matches, min() is always wrong; in ordinary data sum, mean, variance and stdev are wrong too, and the two-document repro gets all five wrong. (Rounding, or a set that holds both a value and its mis-read image, such as -5.0 and -0.8749999999999999, can leave an aggregate unchanged.)
• Likelihood: plausible. Deterministic on the first negative value once the setup is used; the setup itself is rare.

The test is the hunt's minimal repro, unchanged; it passes :lucene:misc:check -x test (google-java-format, ecj lint, forbidden APIs). Use the same test as the regression test.

Executed: gradle on main 5adff78bb168, assertions on (seed DEADBEEF) and off (seed 5EED1129): fails 1 of 1; with the fix it passes. The standalone runner gives the same failure on main with assertions on and off, as do the registry re-runs (seeds 5EED0E36, 5EED0D37). newDirectory() and newIndexWriterConfig() randomize only the directory and writer settings (the gradle run drew a random codec); an independent verifier's equivalent test with a plain IndexWriterConfig fails the same way.

On 10.5.0 and later the test runs as written. On 9.5.0 to 10.4.x, which have no DocValuesStatsCollectorManager, use DocValuesStatsCollector with searcher.search(query, collector); that variant fails the same way on 9.5.0 and 9.12.3.

cp $REPRO/TestSortedDoubleStatsDoubleField.java $LUCENE/lucene/misc/src/test/org/apache/lucene/misc/search/
cd $LUCENE && ./gradlew :lucene:misc:test --tests TestSortedDoubleStatsDoubleField -Ptests.seed=DEADBEEF -Ptests.asserts=true
# also with -Ptests.asserts=false; this test failed at both seeds run (DEADBEEF, 5EED1129)

TestSortedDoubleStatsDoubleField > testMinOfNegativeDoubleField FAILED
    java.lang.AssertionError: expected:<-5.0> but was:<-0.8749999999999999>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:2B44CD2E2B06DA91]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:555)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:685)
        at org.apache.lucene.misc.search.TestSortedDoubleStatsDoubleField.testMinOfNegativeDoubleField(TestSortedDoubleStatsDoubleField.java:47)
:lucene:misc:test (FAILURE): 1 test, 1  failure

• Contract. "Holds DocValues statistics for a sorted-numeric field storing double values" (DocValuesStats.java:421); min(): "The minimum value of the field. Undefined when count() is zero." (:111). No encoding precondition appears in the class, the package docs, the misc overview, CHANGES or MIGRATE.
• Core's recipe. "Note that if you want to encode doubles or floats with proper sort order, you will need to encode them with NumericUtils" (SortedNumericDocValuesField.java:33-37), there since the sorted-numeric type was added in 2014, before this class.
• Core's readers. SortedNumericSelector ("undo the numericutils sortability", :74-89), which SortedNumericSortField uses for every comparator (:282-368), decodes sorted-numeric doubles as sortable; so do the index sorter and Luke's doc-values view. No non-test, non-sandbox Lucene code writes raw-bit sorted-numeric doubles.
• Raw bits are stated where meant. DoubleDocValuesField ("encoding doubles as NumericDocValues via Double#doubleToRawLongBits", :20-21) and DoubleRangeFacetCounts ("This assumes that the field was indexed with DoubleDocValuesField", :53-54). DocValuesStats has no such sentence.
• Tests. TestDocValuesStatsCollector.testDocsWithMultipleDoubleValues writes Double.doubleToRawLongBits(nextVal) with nextVal = 1, 2, 3, ... (:274-290). For these values both layouts are the same bits, so the test passes 8 of 8 on stock and with the fix. No misc test runs these stats on a DoubleField.
• History. The raw decode is from LUCENE-7590 (2016, 6.4.0), whose thread never discusses the encoding. DoubleField came with #11997 (9.5.0), which did not touch DocValuesStats; there jpountz wrote that multi-valued floats and doubles "should use NumericUtil#doubleToSortableLong to present ordering within a document", proposed "only supporting multi-valued fields, so that there is a single encoding in doc values", pushed that change and wrote (quoted as written): "These new fields not only support indexing sorted numeric doc values, so that queries do not have to specify whether floats/doubles are indexed via their raw bits or as sortable ints/longs". So the doc-values type is SORTED_NUMERIC only; the fields also index a point.

The strongest "intended" case (built by the devil's-advocate verifier and the cross-model reviews; the refute-lens review rated it strong, the others moderate): LUCENE-7590 added the raw decode together with a raw-bit test, so the author meant the class as the multi-valued twin of DoubleDocValuesStats/DoubleDocValuesField, which are raw bits. In 2016 core had no sorted-numeric double field, so the encoding was the user's choice, and the javadoc recipe is conditional ("if you want ... proper sort order"), an order that matters only within a document, which the statistics ignore. The facet module's MultiDoubleValuesSource.fromDoubleField reads sorted-numeric doubles as raw bits too. The class is exact for raw-bit writers, negatives included (doubleToRawLongBits -5.0 and 3.0 give min -5.0, sum -2.0), and sorted-numeric doc values carry no encoding tag, so a decode switch changes results for them. A maintainer could answer "document that it expects raw bits".

Why it fails: nothing documents raw bits for this class, while Lucene states raw bits where it means them. Core's only sorted-numeric double field and core's recipe are sortable, core's readers of such fields decode sortable, and #11997 made the new fields sortable on purpose. The upstream test pins nothing observable. Core's standard double field gets wrong statistics today, with no warning. The argument bears on how to fix it (a migration note, or a decoder argument), not on whether DoubleField users get wrong numbers. Verdict: a defect for DoubleField users, an undocumented mismatch between core's sortable writer and this reader; whether the class's contract becomes sortable or raw (documented) is the maintainers' call, and the residual risk that they prefer documenting raw bits is why the priority sits in the lower part of Medium.

One fix was built and validated in a scratch worktree of 5adff78bb168; the alternatives are listed with their trade-offs.

In DocValuesStats.SortedDoubleDocValuesStats.doAccumulate, decode each value with NumericUtils.sortableLongToDouble, the inverse of what DoubleField and core's recipe write, and add one javadoc sentence naming the expected encoding (+7/-2 with the import). The NUMERIC DoubleDocValuesStats keeps its raw decode, which matches DoubleDocValuesField.

/**
 * Holds DocValues statistics for a sorted-numeric field storing {@code double} values. Values
 * must be encoded with {@link NumericUtils#doubleToSortableLong(double)}, as {@link
 * org.apache.lucene.document.DoubleField} does.
 */
public static final class SortedDoubleDocValuesStats extends SortedNumericDocValuesStats<Double> {
  ...
        double val = NumericUtils.sortableLongToDouble(sndv.nextValue()); // was Double.longBitsToDouble

Trade-offs

• Behaviour change. Applications that wrote raw IEEE bits of negative values into a SortedNumericDocValuesField (what the upstream test does, with positive values only) have their negative values decoded exactly today and misread after the fix, symmetric to the bug, so their statistics can change: an independent reviewer's raw-bit -2.5, 7.25 and -0.5 give min -7.999999999999999 under the fix (executed). Non-negative raw-bit data is unaffected. Needs a CHANGES entry and a MIGRATE note; whether to ship it on a release branch is a maintainer call.
• Optionally switch the upstream test's Double.doubleToRawLongBits to NumericUtils.doubleToSortableLong and add a negative value.
• Cost: still one decode per value, no allocation; the sortable decode adds a fixed XOR, shift and mask per value. No benchmark was run.

What the checks validated

• Gradle, JDK 25, scratch worktree of 5adff78bb168 (removed afterwards): the test fails on stock with assertions on (DEADBEEF) and off (5EED1129), and passes with the fix, on and off.
• With the fix: the test plus TestDocValuesStatsCollector, 9 tests, 0 failures; the whole :lucene:misc:test task, 317 tests (9 skipped), passes; :lucene:misc:check -x test and :lucene:misc:renderJavadoc pass.
• The decode line alone was validated the same way by the finder, an independent reproducer and a cross-model review, with the standalone runner.

Add a constructor SortedDoubleDocValuesStats(String field, LongToDoubleFunction decoder), with the sortable decode as default (or, where results must not change, the raw one), and name the expected encoding in the javadoc. The facet module already offers this shape: MultiDoubleValuesSource.fromField(field, LongToDoubleFunction decoder).

Trade-offs

Raw-bit writers can keep their results; more API surface. Not built.

State that the field must hold raw IEEE bits and that DoubleField is not supported.

Trade-offs

No behaviour change, but core's standard double field could then not be summarized by this class at all, against core's own recipe. Not recommended; not validated.

Method: non-formal (an executed differential probe; the Z3 bit-vector model only proved the trigger set afterwards).

• Round 23 of the hunt planned pattern-transfer checks: take the shape of a confirmed bug (LFV-1111, a numeric join that reads sortable DoubleField values as raw bits) and test the property "one logical type stored in two doc-values encodings must decode to the same value on every consumer" on other readers. Its own probe (2026-11-02, main e1faeba08bde, assertions on and off, also 10.6.0) fed the same values -5.0 and 3.0 to SortedDoubleDocValuesStats through three writers: DoubleField and the javadoc recipe gave min -0.8749999999999999, the raw-bit control the right -5.0. A second pattern-transfer probe found it again.
• The hunt unit that judged it reduced it to a public-API JUnit, ran it on main, 10.6.0 and 9.5.0, modelled the trigger set in Z3 (the raw decode of the sortable layout is the identity exactly when the sign bit is clear) and checked the fix against the upstream test; a history step traced it to LUCENE-7590 and #11997.
• An independent reproducer, cross-model reviews (OpenAI GPT-5.6), a devil's-advocate verifier, an adjudication, an audit and a final repro check then re-ran it.
• An earlier sighting (2026-10-26, while LFV-155 was being verified) noticed this mis-decode, but it was not followed up then.