DoubleDocValuesStats.mean() is NaN instead of Infinity when a value follows an infinite value

DocValuesStats.DoubleDocValuesStats and SortedDoubleDocValuesStats (lucene/misc) document mean() as "The mean of all values of the field." and update it one value at a time with mean += (val - mean) / count. Once a value is +Infinity or -Infinity the running mean is infinite, and the next value computes Infinity - Infinity: the mean of {+Infinity, 1.0} is NaN, while sum() and max() of the same object are +Infinity and sum() / count() is +Infinity. {1.0, +Infinity} gives +Infinity, {+Infinity, +Infinity} gives NaN, and since 10.5.0 a searcher that puts the segments in different slices can return +Infinity for the same index. In the executed runs nothing throws. Every release from 6.4.0 (where the classes appeared) to 10.6.0 is affected.

1. Per value. accumulate(doc) increments count and calls doAccumulate(count) for each collected document with a value (DocValuesStats.java:63-70).
2. The running update (DoubleDocValuesStats.doAccumulate, DocValuesStats.java:255-267):

   sum += val;
   double oldMean = mean;
   mean += (val - mean) / count;               // :265
   variance += (val - mean) * (val - oldMean);

SortedDoubleDocValuesStats.doAccumulate does the same for each value of a document, with ++valuesCount and mean += (val - mean) / valuesCount (:443-450).

1. Worked steps for {+Infinity, 1.0}. Value 1: mean = 0 + (Inf - 0) / 1 = Inf. Value 2: (1.0 - Inf) / 2 = -Inf, so mean = Inf + (-Inf) = NaN. For {+Inf, +Inf} the second step is Inf + (Inf - Inf) / 2 = Inf + NaN = NaN. Once NaN, the mean stays NaN. If the infinite value comes last ({1.0, +Inf}), the last step is 1.0 + (Inf - 1.0) / 2 = Inf, the right answer.
2. What stays right. sum (:263) and max (:257-259) are +Infinity and count is 2.
3. Why an executor can disagree (10.5.0 and later). DocValuesStatsCollectorManager.reduce() merges every collector's stats into a fresh object (DocValuesStatsCollectorManager.java:58-68). The first merge copies the mean unchanged (DocValuesStats.java:175-181); each later merge rebuilds it as (sum() + otherStats.sum()) / totalCount (:184-185; sorted variant :353-355), which keeps +Infinity. A plain IndexSearcher has one slice for all segments (IndexSearcher.java:226-231), so one collector and the running mean; with one segment per slice, {+Inf} and {1.0} land in different collectors and the merge gives +Infinity. This is a question of slice layout, not of the executor alone: by default an executor groups up to 5 small segments (up to 250,000 documents) into one slice (IndexSearcher.java:106-108, :323-325), so two small segments still share one collector; the executed manager tests override slices() to give each segment its own slice.

• On a plain searcher: {+Inf, +Inf} -> NaN; {-Inf, 5.0} -> NaN (expected -Infinity); {1.0, +Inf} -> +Infinity while {+Inf, 1.0} -> NaN; SortedDoubleDocValuesStats over docs {+Inf}, {1.0} -> NaN and over one doc {+Inf, +Inf} -> NaN; {Inf, 1, 2} -> NaN, {1, 2, Inf} -> +Infinity, {Inf} -> +Infinity; {Inf, -Inf} -> NaN, which is right.
• With an executor and one segment per slice (forced by overriding slices()), two segments {+Inf} and {1.0}: +Infinity, while the plain searcher gives NaN for the same index.

• Not part of this bug: variance()/stdev() (NaN with infinite input, before and after a fix); the finite overflow {Double.MAX_VALUE, -Double.MAX_VALUE} -> mean -Infinity (the same line, extreme magnitudes); the initial min/max values (LFV-155).
• SortedDoubleDocValuesStats decodes raw double bits (Double.longBitsToDouble, :436), as the upstream test writes them; for positive values raw and NumericUtils sortable bits are identical, so the positive examples do not depend on that choice. The single-valued DoubleDocValuesField case is the clean trigger.
• Within one document, SortedDoubleDocValuesStats sees the values sorted by Long.compare of their stored bits (SortedNumericDocValues.java:22), not in insertion order. With raw double bits +Infinity sorts after every positive finite value, so one document {+Inf, 1.0} is collected as 1.0, +Inf and is right (by reading); one document {+Inf, +Inf} gives NaN, and so does one document {-Inf, 1.5}, since negative values sort first (the harness re-run).

• Who is exposed. Users of these classes through DocValuesStatsCollector (every release since 6.4.0) or DocValuesStatsCollectorManager (10.5.0 and later), with the default IndexWriterConfig, codec and searcher, whenever the field holds an infinite value (an overflowed computation stored as is, or an explicit sentinel) and the same collector collects another value after it: a later document in doc-id order or, for the sorted variant, a later value of the same document.
• Effect. Silent wrong result: mean() is NaN instead of +Infinity or -Infinity, and anything computed from it is NaN too. Which answer you get depends on document order and, since 10.5.0, on the slice layout (one slice versus one segment per slice; with an executor the default slicing still groups up to 5 small segments into one slice). In the executed runs: no exception, no hang, and the stats classes only read the index.
• Not affected. The infinite value collected last by its collector; values that mix +Infinity and -Infinity (NaN is right there); the long variants; count(), sum() and, in these examples, max() (with only non-positive values max() has its own bug, LFV-155).
• Releases. Every release from 6.4.0 to 10.6.0 and main. Priority low.

The two documents go into one segment at close() in insertion order, and a plain IndexSearcher uses one collector. The seed still randomizes the test framework's environment: the default codec, which new IndexWriterConfig() inherits, the directory implementation, the similarity, the locale and the time zone. The wrong mean comes from the stats class's arithmetic, and both seeds run fail the same way (DEADBEEF with codec Lucene104, C0FFEE with Asserting(Lucene104)). The test needs DocValuesStatsCollectorManager, so it compiles on 10.5.0 and later; on older releases the same check runs through IndexSearcher#search(Query, Collector) with a DocValuesStatsCollector.

cp $REPRO/TestDocValuesStatsInfiniteMean.java $LUCENE/lucene/misc/src/test/org/apache/lucene/misc/search/
cd $LUCENE && ./gradlew :lucene:misc:test --tests TestDocValuesStatsInfiniteMean -Ptests.seed=DEADBEEF -Ptests.asserts=true
# also with -Ptests.asserts=false

TestDocValuesStatsInfiniteMean > testMeanOfInfinityThenOne FAILED
    java.lang.AssertionError: expected:<Infinity> but was:<NaN>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:867C5475491EFF6]:0)
        ...
        at org.apache.lucene.misc.search.TestDocValuesStatsInfiniteMean.testMeanOfInfinityThenOne(TestDocValuesStatsInfiniteMean.java:48)
:lucene:misc:test (FAILURE): 1 test, 1  failure

# the same with -Ptests.asserts=false, on the standalone runner (-ea, -da) and on the 10.6.0 release jars (-ea, -da)
# with the recommended fix: :lucene:misc:test (SUCCESS)

• Contract. mean(): "The mean of all values of the field." (DocValuesStats.java:144-147, :306-309), with no restriction to finite values. The only numeric caveat is on sum(): "Note that if the values are large, the sum might overflow." (:160-163, :327-330).
• The upstream test's oracle. TestDocValuesStatsCollector checks mean() against DoubleSummaryStatistics.getAverage() (:199-202, :318-322), which returns +Infinity for these values; the test only indexes the finite positive values 1, 2, 3, ... (:159-211).
• Legal input. DoubleDocValuesField stores any double with Double.doubleToRawLongBits (DoubleDocValuesField.java:38-40), and Lucene's own TestDocValuesQueries#testDocValueQueryWhenFieldNameIsAlsoUsedForPoints indexes Double.NEGATIVE_INFINITY and Double.POSITIVE_INFINITY into it (TestDocValuesQueries.java:1021-1048).
• History. LUCENE-7590 (#8641) introduced the running update; the only numeric limit discussed was sum overflow ("Should we care about overflow, or only document it?"), never NaN or infinity. The #15653 review does not mention non-finite values either.

Strongest case (the intended-behaviour check, the adjudication): the running update is the textbook online (Welford-style) recurrence for a running mean and variance. Apache Commons Math 3.6.1 uses the same recurrence in its storeless Mean and documents: "Note that Double.NaN may also be returned if the input includes NaN and / or infinite values." In a reference run its SummaryStatistics.getMean() returns exactly Lucene's values, order dependence included, and its DescriptiveStatistics / StatUtils.mean() return NaN for any input with an infinity. Lucene's javadoc does not spell out IEEE semantics, IndexSearcher#search(Query, CollectorManager) is @lucene.experimental, and finite means already differ by rounding between the serial and the executor path.

Why it fails: none of that is in Lucene (no javadoc caveat, test, CHANGES or MIGRATE entry, or review comment), and Lucene's own code and test point to +Infinity (sum(), max(), merge(), the DoubleSummaryStatistics oracle). An order-dependent answer, and NaN for {+Inf, +Inf}, is an artifact of Infinity - Infinity. NaN versus +Infinity is not rounding, and the serial failure needs no experimental API: it reproduces through the plain DocValuesStatsCollector on 9.12.3 and earlier. Verdict: not intended, a defect. A maintainer may still prefer a javadoc caveat; it would have to cover the order dependence and the executor disagreement.

Two code fixes were built and run in scratch worktrees of 645d4fe14a20 with gradle; they differ in the guard and in the wording of the new comment. A javadoc-only change and an always-sum / count mean were considered and not built.

While the running mean is finite the update is today's: a finite mean plus an infinite step is the right +Infinity or -Infinity (NaN for a NaN value). Only after the mean has become non-finite does the next value use sum / n, the formula merge() uses. For the infinity examples above that is the DoubleSummaryStatistics answer in either document order, and in the independent reproduction's manager test the plain and the one-segment-per-slice searcher agree. It is only as good as sum: once finite values near Double.MAX_VALUE have overflowed sum, sum / n can be NaN. The same change goes into SortedDoubleDocValuesStats with valuesCount.

       sum += val;
       double oldMean = mean;
-      mean += (val - mean) / count;
-      variance += (val - mean) * (val - oldMean);
+      if (Double.isFinite(mean)) {
+        mean += (val - mean) / count;
+        variance += (val - mean) * (val - oldMean);
+      } else {
+        // once the mean is infinite, (val - mean) can be Infinity - Infinity = NaN
+        mean = sum / count;
+        variance = Double.NaN;
+      }

Trade-offs

• Inputs whose running mean stays finite (finite values not near +/-Double.MAX_VALUE) take today's path, so their results do not change. When finite values do overflow the running mean, later values now get sum / n instead of NaN: {MAX_VALUE, -MAX_VALUE, 1.0} gives 0.3333333333333333, as DoubleSummaryStatistics does, where stock gives NaN. In the finder's exhaustive harness all 2,044 stock cases with NaN for a finite exact mean change: 1,972 become right and 72 become +/-Infinity; no result that stock gets right changes.
• variance becomes NaN once the mean is non-finite, which is what stock already returns.
• One Double.isFinite check per value; no benchmark was run.
• Not covered: the finite overflow {Double.MAX_VALUE, -Double.MAX_VALUE} -> -Infinity, as on stock; and values collected after finite values have overflowed sum and an infinity of the other sign has followed (72 harness cases still give NaN where the exact mean is an infinity, e.g. MAX_VALUE, MAX_VALUE, -Infinity, -0.0, where sum is already NaN when the last value arrives).

What the checks validated

• gradle, stock + test: fails with assertions on and off; + this fix: the test passes.
• gradle, + this fix: all lucene/misc tests, 317 tests (9 skipped), 0 failures, assertions on; the test and TestDocValuesStatsCollector for 10 iterations (90 tests), 0 failures, assertions off; checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass.
• Standalone runner with the patched file shadowed: the test (assertions on and off), the independent reproduction's serial test (7 tests: the sorted variant, {-Inf, 5.0}, the order dependence) and manager test (plain versus a one-segment-per-slice executor searcher), and upstream TestDocValuesStatsCollector for 20 iterations (160 tests) pass.
• A plain-Java model of the recurrences over 9 inputs: the DoubleSummaryStatistics answer on 5 of the 6 inputs with an infinity; on {MAX_VALUE, MAX_VALUE, -Infinity} it keeps stock's -Infinity, the exact answer, where DoubleSummaryStatistics and the alternative give NaN.
• The finder's exhaustive harness with this fix shadowed (re-run by the fact check): SDOUBLE 0 serial mean mismatches; DOUBLE 3,186, all from finite values near +/-Double.MAX_VALUE (72 NaN where the exact mean is an infinity, 3,114 finite overflow), against 39,734 on stock. Not run: the full Lucene test suite, nightly tests, benchmarks.

The same change with if (Double.isFinite(val) && Double.isFinite(mean)), so an infinite value arriving while the mean is finite also takes the sum / n branch.

Trade-offs

Same answers on all the tests, but only as good as sum: when finite values have already overflowed sum (values near Double.MAX_VALUE) and an infinity of the other sign follows, it gives NaN: {MAX_VALUE, MAX_VALUE, -Infinity} -> NaN, where stock and the recommended fix give -Infinity (DoubleSummaryStatistics gives NaN too). Shown by the plain-Java model and through the Lucene class in the harness re-run: 348 cases with NaN where the exact mean is an infinity, against 72 for the recommended fix.

What the checks validated

• gradle: all lucene/misc tests (317, 9 skipped, 0 failures, assertions on); the test and TestDocValuesStatsCollector for 10 iterations (assertions off); :lucene:misc:check -x test.
• The independent reproduction's serial and manager tests pass with it shadowed.

Document on mean() that the result may be NaN when the values include an infinity or NaN, as Apache Commons Math does. Not built.

Trade-offs

Smallest change, but it leaves the order dependence ({1.0, +Inf} gives +Infinity), NaN for {+Inf, +Inf} and the serial-versus-executor disagreement, and it documents an answer that differs from DoubleSummaryStatistics, the reference the upstream test uses.

Drop the running mean for mean(). Not built and not modelled.

Trade-offs

It is the formula merge() already uses and removes the order dependence for the infinity examples, but the sum itself overflows for large finite values ({Double.MAX_VALUE, Double.MAX_VALUE} would give +Infinity instead of Double.MAX_VALUE; DoubleSummaryStatistics gives Infinity there too), finite results change by rounding, and variance still needs the running mean.

Method: bounded-exhaustive translation validation (semi-formal), a hunt unit (never-named public API sweep: facet, join, grouping, misc). The harness DvStatsExhaustive.java runs the real DoubleDocValuesStats and SortedDoubleDocValuesStats over small value alphabets (DOUBLE: NaN, +/-0.0, +/-Infinity, Double.MIN_VALUE, +/-Double.MAX_VALUE, 1.5, -2.5; SDOUBLE: NaN, -0.0, +/-Infinity, 1.5, -2.5) on every index of 1-3 segments (DOUBLE: 153,439 indexes, 879,571 searches; SDOUBLE: 112,140 indexes, 490,056 searches), serially and through DocValuesStatsCollectorManager under 1-3 slices and an intra-segment split, against an exact reference with IEEE semantics for non-finite values. On stock, 34,648 of the 39,734 DOUBLE serial mean mismatches and all 22,040 SDOUBLE ones are NaN where the reference is an infinity; the other 5,086 DOUBLE ones are the finite overflow near +/-Double.MAX_VALUE (not part of this bug). With the recommended fix SDOUBLE has no serial mean mismatch and DOUBLE keeps 72 NaN-for-infinity cases, all after finite values near Double.MAX_VALUE had overflowed sum, plus 3,114 finite-overflow cases; with the finder's sketch, 348 and 3,114. (The split by kind is the fact check's re-run of the unchanged harness.) The finder then wrote a public-API JUnit on the stock main jars; this doc's test is a one-method form of it.