DocValuesStatsCollectorManager returns a wrong mean when merge() reads a wrapped long sum

DocValuesStatsCollectorManager (lucene/misc, new in 10.5.0) computes LongDocValuesStats or SortedLongDocValuesStats once per slice and merges the slices with DocValuesStats.merge(). The merge rebuilds the mean as (sum() + other.sum()) / n, with n the combined number of values, and for the long classes sum() is a plain long that wraps silently past Long.MAX_VALUE, as its javadoc warns. With large but ordinary values the merged mean is wrong once a sum it reads has wrapped: 24 epoch-nanosecond timestamps in 6 segments give mean() = 2.243809386908707E17 instead of 1.7616096115E18, and another slicing of the same values gives -5.44E17. variance() and stdev() are wrong too. A searcher without an executor returns the exact mean, because serial collection keeps a running mean that never reads the sum. count(), missing(), min() and max() stay right and nothing throws. Affected: 10.5.0, 10.5.1, 10.5.2, 10.6.0 (branch_10_6 4ae946e5a9b9) and main; 10.4.0 and older have no manager and no merge().

• Slice: a group of segments that one thread searches. Without an executor a searcher has one slice (IndexSearcher.java:226-231); with one, slices() sorts the segments by size and closes a slice once it has 5 segments or more than 250,000 documents, and a segment of more than 250,000 documents gets a slice of its own (:106-108, :323-324, :341-390). A slice of small segments can so hold up to about 500,000 documents.
• Collector manager: creates one collector per slice, then reduces them into one result. DocValuesStatsCollectorManager.reduce starts from a fresh stats object and calls merge() once per collector (DocValuesStatsCollectorManager.java:60-68).

1. Collecting a value (DocValuesStats.java:217-229) adds it to private long sum (sum += val, :225) and updates a running mean, mean += (val - mean) / count, and a running sum of squared deviations (:226-228). The running mean never reads the sum, so each slice's mean is right.
2. NumericDocValuesStats.merge computes the combined mean from the sums (:184-185): double combinedMean = (sum().doubleValue() + otherStats.sum().doubleValue()) / totalCount; For LongDocValuesStats the sum is a long that has wrapped. SortedNumericDocValuesStats.merge does the same over valuesCount (:353-355).
3. The pooled variance (:186-193) adds (mean - combinedMean)^2 * count for both sides, so a wrong combined mean also makes variance() and stdev() wrong.
4. LongDocValuesStats.merge then adds the sums as longs (:237-240), so the merged sum keeps wrapping. One slice is always exact: the first merge only copies it (:175-181). With 2 or more non-empty slices and values of one sign, the final mean is wrong exactly when the last slice's sum, or the running sum of the slices merged before it, leaves the long range: in the 6-slice row below no slice overflows, the running sum does. With mixed signs the wraps can cancel (slices [MAX, MAX] and [MIN, MIN] give the exact mean -0.5), and a merge that read a wrapped sum can leave a wrong variance under a right final mean.

24 documents, ts = 1,761,609,600,000,000,000 + i * 1,000,000,000 for i = 0..23 (2025-10-28T00:00:00Z plus one second per document). The true sum is 42,278,630,676,000,000,000, about 4.6 times Long.MAX_VALUE. Exact mean 1.7616096115E18, variance 4.79E19, stdev 6.92E9 (6.9 seconds).

In every row count() is 24, min() and max() are right, and sum() is 5385142528580896768 (the true sum modulo 2^64; its javadoc says it "might overflow"). In the 2-slice row the 20-value slice's sum, 35,232,192,190,000,000,000, wraps twice to -1,661,295,957,419,103,232; the 4-value slice's sum, 7,046,438,486,000,000,000, fits; (-1.661e18 + 7.046e18) / 24 = 2.2438e17. SortedLongDocValuesStats on a LongField gives the same 2.243809386908707E17 with the default slicing.

• Who is hit. Users of the lucene/misc statistics through DocValuesStatsCollectorManager on an IndexSearcher built with an executor, in 10.5.0 or later. The manager is the usage the class javadoc shows (DocValuesStatsCollectorManager.java:27-34), and IndexSearcher.search(Query, Collector), the only route before 10.5.0, is deprecated in favour of search(Query, CollectorManager) (IndexSearcher.java:634-638). The route is opt-in: CollectorManager, search(Query, CollectorManager) and the executor constructors are @lucene.experimental (CollectorManager.java:42; IndexSearcher.java:199, :216, :748), and the stats classes live in lucene/misc. The default IndexWriterConfig, codec and slicing suffice; the repro sets NoMergePolicy only to fix the segment count.
• The trigger (values of one sign; mixed signs can cancel). The reduce merges 2 or more slices that have values, and one slice's long sum, or the running long sum of the slices merged before the last one, leaves the long range.
  • Epoch-nanosecond timestamps (about 1.76e18): the 6th value overflows; 5 x 1.76e18 = 8.8e18 still fits.
  • Epoch-millisecond timestamps (about 1.76e12): a slice of small segments (at most about 500,000 documents) cannot overflow its own sum; a single segment of more than about 5.24 million values, which gets a slice of its own, can; and the running sum overflows once the slices merged before the last one hold more than about 5.24 million values (9.22e18 / 1.76e12), for example 21 slices of 250,000 values.
  • Small values cannot trigger it on a single-valued field: an index holds at most IndexWriter.MAX_DOCS = 2,147,483,519 documents (IndexWriter.java:208), and that many values of magnitude at most 4,294,967,554 (Long.MAX_VALUE / MAX_DOCS) fit in a long. Typical prices or counts are far below that.
• What goes wrong. mean(), variance() and stdev() are silently wrong, far too small or negative depending on the slicing. count(), missing(), min() and max() are right; the index and the matched documents are unaffected.
• Hard to notice. The same query gives the exact mean without an executor, and also with one when everything fits in one slice. Tests with small values or small indexes pass.
• Not part of this bug. DoubleDocValuesStats uses the same formula, so two values near Double.MAX_VALUE in two partitions give Infinity (the logged example splits one segment into two slices); extreme. The fix removes that overflow but changes other non-finite double results (see the fixes). SortedNumericDocValuesStats.variance() divides by documents, not values (DocValuesStats.java:312-315), which the upstream test pins.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<2.0E18> but was:<-1.0744573456182586E18>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:555)
       org.junit.Assert.assertEquals(Assert.java:685)
       org.apache.lucene.misc.search.TestDocValuesStatsMergedMean.testMeanWithExecutor(TestDocValuesStatsMergedMean.java:57)

• The contract. merge(): "After the call, this instance reflects the combined statistics of both segments as if they had been collected together." (DocValuesStats.java:76-83). mean(): "The mean of all values of the field." (:144-147, :306-309), with no caveat. Only sum() warns: "Note that if the values are large, the sum might overflow." (:160-164, :327-331).
• History. The wrapping long sum and its caveat date from LUCENE-7590 (295cab7216c, 2016-12-15), next to a running mean that never used it (ad7152ad473, 2016-12-11). In that issue (#8641) the author asked "Should we care about overflow, or only document it?" about the sum; there was no merge then. #15653 (0210af757c54, 10.5.0) added merge() with the sum-based mean, the first code that makes mean() depend on the sum. Its review (41 inline comments) discusses generics, visibility, a doubleValue() precision issue in the min/max comparison and the sum() accessor, never an overflowing sum in the mean.
• Tests. TestDocValuesStatsCollector runs the manager on newSearcher (which may add an executor) with small sequential long values: 1 to at most 100 in testDocsWithLongValues (TestDocValuesStatsCollector.java:105-118, :134-138), and 1 to at most about 500 (up to 100 documents with 1-5 values each) in testDocsWithMultipleLongValues (:213-228, :246-250). Neither can reach an overflow.
• Execution-dependent results. The same index and query give 1.76E18, 2.24E17 or -5.44E17 depending on whether the searcher has an executor and how segments fall into slices. merge() promises collection-equivalent statistics, and the serial path is exact.

The strongest case for "intended": sum() documents that it may overflow, and java.util.LongSummaryStatistics.getAverage(), which the upstream tests use as their mean oracle (TestDocValuesStatsCollector.java:145-148, :256-260), is (double) getSum() / getCount() with an unchecked long sum ("This implementation does not check for overflow of the count or the sum"). For the repro data getAverage() is exactly 2.243809386908707E17, the concurrent result. Read that way, a wrapping sum-based mean is the JDK's own semantics. A maintainer could add that the concurrent route is @lucene.experimental and lives in lucene/misc, that a double near 1.76e18 is only accurate to 256, and that the deprecated single-collector route still gives the exact mean.

Why it fails: Lucene's mean has never had those semantics. Serial collection, the only route before 10.5.0 and still the route without an executor, returns the exact mean for the same data, and mean() has no overflow caveat. The concurrent result is not even the JDK value in general: it changes with the slicing (-5.44E17 for 4 slices of 6, exact for 1 slice). The tests' oracle is a convenience for values of at most about 500 and asserts nothing about overflow, and merge() promises statistics "as if they had been collected together". An error of about 1.5e18, or a negative mean of positive timestamps, is not rounding, and the experimental tag warns that the API "might change in incompatible ways in the next release", not that results may be wrong. The argument supports a low priority or better documentation, not the current result. Verdict: not intended, a defect.

One fix was built and validated with gradle; the alternatives are sketches. sum() and its documented overflow stay as they are in every option except C.

In NumericDocValuesStats.merge and SortedNumericDocValuesStats.merge, compute combinedMean = mean1 * (n1 / n) + mean2 * (n2 / n) from the running means, which do not depend on the wrapped sum and are accurate to double precision (not exact), instead of (sum1 + sum2) / n. The pooled variance formula is unchanged and becomes right with the right mean. Land it with a decision on the double side effect (trade-offs).

int totalCount = this.count() + otherStats.count();
// weight the two means instead of dividing the summed sums: a long sum may have overflowed
double combinedMean =
    this.mean * ((double) this.count() / totalCount)
        + otherStats.mean() * ((double) otherStats.count() / totalCount);
// SortedNumericDocValuesStats.merge: the same with valuesCount

Trade-offs

• Precision. The merged mean is as accurate as the per-slice running means, about the serial path's precision, not exact: the 4-slice probe prints 1.7616096114999997E18, and for the extreme slices [MAX, MAX] and [MIN, MIN] the fix gives 0.0 where the exact (and stock) mean is -0.5, because both slice means round to ±2^63. The merged variance agrees with the exact one to about 2.5e-8 relative in the 6-slice probe (4.7916667861E19 vs 4.7916666667E19), from rounding of means near 1.76e18.
• Double classes. They share NumericDocValuesStats.merge. The weighted form keeps an infinite partial mean infinite and removes the Double.MAX_VALUE overflow of the summed sums. But it passes on each slice's running mean, and the double running mean has a separate bug: it becomes NaN when a value follows an infinite one, which the summed sums hid in some slicings. In the bounded-exhaustive double run the multi-slice mean mismatches go from 23,850 to 53,914 with this fix (its logged examples are "got NaN want Infinity"); the serial ones stay 39,734. Nor does the fix make the concurrent double mean equal the serial one: one segment [+Infinity, 0.0] gives NaN serially, two slices [+Infinity] and [0.0] give Infinity, with and without the fix. Treat this as a condition for landing: fix that running mean in the same change, or say in the PR that non-finite double results change and that the running-mean bug is separate.
• Cost and compatibility. Two multiplications and one more division instead of two doubleValue() calls per merge; no API or state change.

What the checks validated

• Gradle, lucene/misc only, scratch worktree of a9779ef3c193, seed DEADBEEF: the repro passes 2 of 2 with -Ptests.asserts=true and =false (stock: 2 of 2 fail); TestDocValuesStatsCollector passes 160 of 160 with -Ptests.iters=20 (stock: 8 of 8); the whole :lucene:misc:test has 24 classes, 317 tests, 0 failures, 9 skipped; tidy makes no change; ecjLintMain, ecjLintTest, checkGoogleJavaFormat, forbiddenApisMain and forbiddenApisTest pass.
• Classpath runner: the independent 4-test file passes 4 of 4 (-da); the probe prints the exact mean for 2, 4 and 6 slices.
• Bounded-exhaustive runs: 0 mismatches for the long classes (56,162 and 102,639 without the fix), over every index of 1-3 segments with 1-2 documents each (at most 4 documents, 3 for the sorted class), values from fixed sets with Long.MIN_VALUE and Long.MAX_VALUE, every partition of the segments into slices and one intra-segment split.

The same idea in the form Chan et al. use for pooled statistics; it may round a little less when the two means are close (not measured here).

Trade-offs

Not validated here. For the double classes an infinite this.mean, or two equal infinite means, gives NaN (inf - inf), while a finite this.mean with an infinite other mean stays infinite, so NumericDocValuesStats.merge needs a guard.

Track the sum without wrapping, as two longs or a BigInteger once it overflows, and keep the sum-based mean.

Trade-offs

Also fixes sum(), but changes the state and cost of every stats object, and sum()'s overflow is documented behaviour, not this bug. Not validated.

Extend the sum() caveat to mean(), variance() and stdev().

Trade-offs

Not recommended: the values are wrong by orders of magnitude, depend on the slicing, and the serial path is exact.

Method: bounded-exhaustive translation validation (the semi-formal tier). A hunt unit sweeping public APIs that no earlier study had named (facet, join, grouping, misc) ran a harness against the stock main jar.

• A planning probe first saw an epoch-nanosecond mean of -5.44e17 instead of 1.76e18 on a searcher with an executor.
• The harness then ran all six DocValuesStats classes through DocValuesStatsCollectorManager against an exact BigDecimal reference, over every index of 1-3 segments with 1-2 documents each (also with a deleted document) and every value choice from alphabets with Long.MIN_VALUE, Long.MAX_VALUE and an epoch-nanosecond timestamp: no executor, every set partition of the segments into slices, and an intra-segment split.
• Long class: 43,720 indexes and 249,424 checks, 56,162 mismatches. Sorted long class: 47,649 indexes and 207,753 checks, 102,639 mismatches. All are mean or variance under a multi-slice reduce; none is serial or one-slice; none remains with the fix.
• It was then reduced to a public-API JUnit with the default IndexSearcher(reader, executor) slicing.