DocValuesStats for doubles reports max() = 4.9E-324 when every value is zero or negative

DocValuesStats.DoubleDocValuesStats and DocValuesStats.SortedDoubleDocValuesStats (lucene/misc) compute count, missing, min, max, sum, mean, variance and stdev of a double doc-values field over the documents a query matches, through DocValuesStatsCollectorManager (since 10.5.0) or DocValuesStatsCollector. Both start the running maximum at Double.MIN_VALUE (DocValuesStats.java:251, :429), which in Java is the smallest positive double, 4.9E-324, not the most negative one. A value replaces the maximum only if it compares above it, so when every collected value is zero or negative max() returns 4.9E-324: a query for the winter temperatures {-12.5, -3.0, -7.25} gets count 3, sum -22.75 and min -12.5, but max 4.9E-324 instead of -3.0, while a match-all query over the same index, which also collects the summer values, reports the right max, 21.0. The javadoc leaves max() undefined only when count() is zero. The mirror mistake makes min() return Double.MAX_VALUE when every value is +Infinity. The long classes start at Long.MIN_VALUE / Long.MAX_VALUE and are correct. Nothing throws and nothing is written; every release since 6.4.0 is affected.

• DocValuesStats (lucene/misc, package org.apache.lucene.misc.search): per-field statistics over the documents a query matches: count(), missing(), min(), max(), and for numeric fields sum(), mean(), variance(), stdev(). DoubleDocValuesStats reads a DoubleDocValuesField; SortedDoubleDocValuesStats reads a multi-valued SortedNumericDocValuesField holding raw double bits.
• Two entry points. DocValuesStatsCollectorManager with IndexSearcher.search(Query, CollectorManager) (since 10.5.0, one stats object per slice, merged in reduce()), and DocValuesStatsCollector with the deprecated IndexSearcher.search(Query, Collector).
• Double.MIN_VALUE is, per the JDK javadoc, "the smallest positive nonzero value of type double, 2^-1074", that is 4.9E-324. The most negative finite double is -Double.MAX_VALUE. For long, by contrast, Long.MIN_VALUE is the most negative value.

1. The constructor stores the starting min and max (DocValuesStats.java:39-43). DoubleDocValuesStats passes super(field, Double.MAX_VALUE, Double.MIN_VALUE) (:251), and SortedDoubleDocValuesStats the same pair (:429). The long classes pass Long.MAX_VALUE, Long.MIN_VALUE (:213, :384).
2. For each collected value, doAccumulate replaces max only when Double.compare(val, max) > 0 and min only when Double.compare(val, min) < 0 (:255-262; per value in the sorted class, :433-442).
3. Every value <= 0, including 0.0 and -0.0, compares below 4.9E-324. So when every collected value is <= 0 the max never moves and max() returns 4.9E-324. One positive value is at least 4.9E-324 and makes the max right.
4. On the min side only Double.MAX_VALUE, +Infinity and NaN fail to move the min, so min() is wrong (Double.MAX_VALUE) when every value is +Infinity, and also for all-NaN input or input of only +Infinity and NaN.
5. The trigger is per query: DocValuesStatsCollector accumulates only the documents the query matches (DocValuesStatsCollector.java:56-64).
6. The concurrent path cannot repair it: reduce() merges the per-slice stats into a fresh object (DocValuesStatsCollectorManager.java:60-68), and merge() skips empty slices, copies the first non-empty one and keeps the larger max (DocValuesStats.java:168-203, sorted :335-374). A non-empty slice of values <= 0 already holds 4.9E-324.

• {-5.0, -3.0, -4.0}, match-all: count 3, min -5.0, sum -12.0, mean -4.0, max 4.9E-324 (expected -3.0).
• Same index with summer 21.0 and 18.5: match-all max 21.0 (right); the winter filter over {-12.5, -3.0, -7.25}: count 3, sum -22.75, min -12.5, max 4.9E-324.
• One document -2.0: min = sum = mean = -2.0, max 4.9E-324. Four documents 0.0: stdev 0, min 0.0, max 4.9E-324.
• SortedDoubleDocValuesStats over {-1.5, -8.0}, {-0.5}, {-4.0, -2.0, -6.0}: min -8.0, max 4.9E-324 (expected -0.5).
• Two documents +Infinity: max +Infinity, min 1.7976931348623157E308 (expected +Infinity).
• Controls: mixed signs {-5.0, 2.0, -4.0} give max 2.0; LongDocValuesStats over {-5, -3, -4} gives max -3.

• Who is hit. Users of DoubleDocValuesStats or SortedDoubleDocValuesStats on a double field whose matching values can all be <= 0: balances, profit and loss, deltas, temperatures, coordinates, or a field with a 0.0 default. Both entry points: DocValuesStatsCollectorManager (the non-deprecated path, since 10.5.0) and DocValuesStatsCollector with IndexSearcher.search(Query, Collector), which is deprecated and slated for removal in 11.0 (apache/lucene#16918, #12892).
• Default vs opt-in. Opt-in by using the misc class (the separate lucene-misc artifact); no setting is involved. A plain main() probe with new IndexWriterConfig(), the default codec and new IndexSearcher(reader) shows it, also with a single document; the JUnit tests show it under the test framework's randomized writer, codec and searcher.
• Per query. A filter that selects only non-positive values is enough, while the same field over all documents may look right (match-all 21.0, winter filter 4.9E-324).
• Effect. A silent wrong number: a positive max for data that is entirely <= 0. A check such as "max > 0", a [min, max] range for a histogram or a chart axis, or max - min (12.5 instead of 9.5 in the winter example) goes wrong. With one value the statistics contradict themselves (min = mean = sum = -2.0, max 4.9E-324). Nothing throws, nothing is written, the matched documents do not change; in this max() case count, missing, sum, mean, variance, stdev and min stay right.
• How likely. The class looks rarely used: the public searches found only two java-user threads that mention it, and a read-only code search of the public default branches of Solr, Elasticsearch and OpenSearch found no use of it (Solr has its own class with the same simple name). That is a search result, not a usage census; private applications are not visible.
• The min() side needs, for non-NaN values, every collected value to be +Infinity (all-NaN input, or only +Infinity and NaN, also leaves min() at Double.MAX_VALUE): real, but contrived. In the all-+Infinity case mean, variance and stdev are NaN as well (Infinity - Infinity), with or without a fix.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<-1.0> but was:<4.9E-324>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:555)
       org.junit.Assert.assertEquals(Assert.java:685)
       org.apache.lucene.misc.search.TestDoubleDocValuesStatsNegativeMax.testMaxOfSingleNegativeValue(TestDoubleDocValuesStatsNegativeMax.java:45)

• Contract. "The minimum value of the field. Undefined when {@link #count()} is zero." and "The maximum value of the field. Undefined when {@link #count()} is zero." (DocValuesStats.java:111, :116). The class javadoc: "Holds DocValues statistics for a numeric field storing {@code double} values" (:243). DoubleDocValuesField accepts any double. No javadoc, CHANGES entry (lucene/CHANGES.txt:8628 LUCENE-7590, :986 #15653), MIGRATE note, LUCENE-7590 comment (#8641, 42 comments) or #15653 review comment (41 inline) restricts the values to positive numbers or mentions the start values.
• The siblings. LongDocValuesStats and SortedLongDocValuesStats start at Long.MAX_VALUE / Long.MIN_VALUE (:213, :384), the true extremes, and give max -3 for {-5, -3, -4}. The double classes were written next to them with the look-alike pair, which means something else for doubles.
• The upstream test. TestDocValuesStatsCollector.testDocsWithDoubleValues (:159-211) compares with java.util.DoubleSummaryStatistics, whose max starts at Double.NEGATIVE_INFINITY (JDK DoubleSummaryStatistics.java:71-72) and gives -3.0 for {-5, -3, -4}. The test indexes only values counting up from 1.0 (nextVal, :165) and uses 0 as its own "missing or deleted" marker (getPositiveValues keeps v > 0, :440-441), so no value <= 0 ever reaches an assertion. It is an analogue for finite values only: its accept uses Math.min / Math.max (DoubleSummaryStatistics.java:144-145), which make both extremes NaN once a NaN is seen, while DocValuesStats orders values with Double.compare, so {NaN, -1.0} gives min -1.0 and max NaN.
• Self-consistency. Upstream testOneDoc asserts max = min = sum = mean = 1 for one long value (:93-100); one double -2.0 gives min = sum = mean = -2.0 but max 4.9E-324.
• History. The pair has been there since the first commit of the class (LUCENE-7590, 2016) and was never discussed. The same mistake, Double.MIN_VALUE as a starting max, was fixed as a bug in another project's own aggregators (elastic/elasticsearch#110586, and a review fix in #42035); neither involves Lucene's class.

Strongest case for "intended" (built by a devil's advocate and by a reviewer from another model family, OpenAI GPT-5.6; both rated it weak): (a) upstream's tests only ever index positive doubles, which could be read as an unwritten positive domain; (b) the start values are sentinels and the result is "undefined" anyway; (c) Lucene uses Float.MIN_VALUE as a max start value elsewhere (TopGroupsCollector.java:80, :171, BlockGroupingCollector.java:274); (d) it is a misc-module utility and the original search path is deprecated; (e) the replacement IndexSearcher.search(Query, CollectorManager) is marked @lucene.experimental (IndexSearcher.java:742-750).

Why it fails: (a) the positive-only data is a test convenience (0 is the missing marker) and the test's own oracle has -Infinity semantics; (b) "undefined" covers only count() == 0, and every failing case has count() >= 1; (c) those start values track scores, and Lucene requires scores to be non-negative ("Queries are now required to produce positive scores", LUCENE-7996), while doc values can be any double, and the long classes in the same file use the true extremes; (d) the non-deprecated DocValuesStatsCollectorManager path, used as its javadoc shows, fails the same way, 80 of 80 runs over 20 randomized iterations; (e) an experimental API may change its signature, but that does not make a wrong max() correct. Verdict: not intended, a defect.

Options A, B and C were built and validated with gradle in a scratch worktree at a9779ef3c193 (A and B also with the independent repros); option D was not run. A is the minimal fix; B and C also define the all-NaN corner case, and C keeps today's values for an empty stats object.

Two lines in DocValuesStats.java, the constructors of DoubleDocValuesStats (:251) and SortedDoubleDocValuesStats (:429).

     public DoubleDocValuesStats(String field) {
-      super(field, Double.MAX_VALUE, Double.MIN_VALUE);
+      super(field, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY);
     }
 ...
     public SortedDoubleDocValuesStats(String field) {
-      super(field, Double.MAX_VALUE, Double.MIN_VALUE);
+      super(field, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY);
     }

Trade-offs

• NaN corner. Double.compare orders NaN above +Infinity, so with every value NaN, min() stays +Infinity (today Double.MAX_VALUE); max() is NaN either way. With NaN mixed with numbers every option gives max NaN and min the smallest number. The javadoc does not define NaN behaviour.
• Empty stats (count() == 0, documented as undefined) now report +Infinity / -Infinity instead of Double.MAX_VALUE / 4.9E-324; worth a CHANGES line.
• Cost: none by reading (the same comparisons as today); not benchmarked.
• Not changed (by any option here): with repeated +Infinity values mean, variance and stdev are NaN (Infinity - Infinity at :263-266), before and after the fix.

What the checks validated

• Gradle, JDK 25, scratch worktree at a9779ef3c193: the new test 6 of 6 (assertions on and off; 120 of 120 over 20 iterations); TestDocValuesStatsCollector 8 of 8; the whole misc module, 24 classes, 321 tests, 9 skipped, 0 failures (assertions on, and in the independent fact check also off); :lucene:misc:check -x test (google-java-format, ecjLint, forbiddenApis) passes.
• Standalone runner with the patched class shadowed: the original 4-test JUnit 4 of 4, an independent reproducer's collector JUnit 7 of 7 and CollectorManager JUnit 3 of 3, the devil's advocate's JUnit 4 of 4.
• The same two-line change on the 6.4.0 release jars returns the right values for every case of the history probe.

In DoubleDocValuesStats.doAccumulate the conditions become count == 1 || Double.compare(val, max) > 0 (and the same for min); in SortedDoubleDocValuesStats.doAccumulate, valuesCount == 0 || ... (+6/-6 with option A's two lines).

       double val = Double.longBitsToDouble(ndv.longValue());
-      if (Double.compare(val, max) > 0) {
+      if (count == 1 || Double.compare(val, max) > 0) {
         max = val;
       }
-      if (Double.compare(val, min) < 0) {
+      if (count == 1 || Double.compare(val, min) < 0) {
         min = val;
       }

Trade-offs

Also gives min = max = NaN for all-NaN input, consistent with Double.compare order. A first-value check on both comparisons in each per-value loop (cost not measured) and a slightly bigger diff, for a corner case the javadoc does not define.

What the checks validated

Same gradle runs as option A (the new test, TestDocValuesStatsCollector, the misc module: 321 tests, 0 failures; check -x test), the original and the devil's advocate's JUnits, and a NaN probe: all-NaN gives min = max = NaN (option A: min +Infinity; stock: min Double.MAX_VALUE).

Option B's four guards (count == 1 || ... in DoubleDocValuesStats.doAccumulate, valuesCount == 0 || ... in SortedDoubleDocValuesStats.doAccumulate) without option A's constructor change (+4/-4).

       double val = Double.longBitsToDouble(ndv.longValue());
-      if (Double.compare(val, max) > 0) {
+      if (count == 1 || Double.compare(val, max) > 0) {
         max = val;
       }
-      if (Double.compare(val, min) < 0) {
+      if (count == 1 || Double.compare(val, min) < 0) {
         min = val;
       }

Trade-offs

Every non-empty input without NaN gets the right extremes and all-NaN gives min = max = NaN, while an empty stats object (count() == 0, documented as undefined) keeps today's Double.MAX_VALUE / 4.9E-324, so nothing changes there. The same added checks as option B; the empty-stats values stay the confusing constants.

What the checks validated

Gradle, JDK 25, scratch worktree at a9779ef3c193: the new test 6 of 6, the original 4-test JUnit (gradle copy) 4 of 4, TestDocValuesStatsCollector 8 of 8 and the NaN probe 2 of 2 (assertions on); the whole misc module with these classes added, 26 classes, 327 tests, 9 skipped, 0 failures (assertions off); :lucene:misc:check -x test passes.

Fixes every finite case.

Trade-offs

By reading, not run: a field whose values are all -Infinity would still report max -Double.MAX_VALUE, the same class of mistake one step further out.

Method: bounded-exhaustive translation validation (the semi-formal tier: a hypothesis formed while modelling, then confirmed by the tool), in a hunt unit on concurrent collector managers.

• The harness (ManagerExhaustive.java) runs the real collector managers on every segment-size shape of 1-3 segments x 1-6 documents (258 shapes), with two generated value patterns and four fixed queries, under every distinct slicing that IndexSearcher.slices gives over the parameter ranges it enumerates (3,750), sequentially and with a 3-thread pool, and compares each result with a brute-force reference computed from the matching hits. It proves nothing beyond those bounds.
• While reading DocValuesStatsCollectorManager#reduce, the finder noticed the start values and added a doc-value-stats spec: a long field and an all-negative double field, with reference min/max starting at +/-Infinity. DVSTATS-double/result failed 30,000 times with assertions on and 30,000 times with them off; the long spec matched. With the unit's fixes shadowed, the harness reports no failure.
• The finder reduced it to a public-API JUnit on the stock main jars. Then: a review by a model from another family (OpenAI GPT-5.6) that executed its own test, an independent reproducer, a read-only "works as designed" review by the same model family, a devil's advocate, an adjudication, a history sweep, a prior-report search and a security scope check.