QualityStats.average(...).getPrecisionAt(n) extrapolates an already averaged precision: mean P@10 0.4 instead of 0.6

Lucene's benchmark module can score a search run the way TREC does. QualityBenchmark returns one QualityStats per judged query, and QualityStats.average(stats) combines them into one object whose getters are the means over the judged queries (those with at least one relevant document). For precision at n, average() computes and stores the correct mean for every cutoff 1 to 20, but it also sets the object's result count to the mean result count. getPrecisionAt(n) then treats the average like a single query that returned fewer than n hits: for every n above the mean result count it ignores the stored mean and extrapolates a second time. Two queries with 10 and 2 relevant hits give an averaged P@10 of 0.4 instead of 0.6. The value is silently wrong (too low, too high, or 0 when the mean result count is below 1); per-query values, MAP, recall and MRR are right, and the SUMMARY that log() prints never shows an affected cutoff. Only code that reads the averaged P@n through the API sees it, in the opt-in benchmark module. The code has been this way since the package was added (LUCENE-836, first released in 2.3.0).

• Class javadoc: "Results of quality benchmark run for a single query or for a set of queries" (QualityStats.java:24).
• average(): "Create a QualityStats object that is the average of the input QualityStats objects" (:189-194).
• getPrecisionAt(n): "Return the precision at rank n", for any n from 1 to MAX_POINTS = 20 (:104-109), with no restriction to n <= getNumPoints().
• getMRR(): "the Mean reciprocal rank over the queries or RR for a single query" (:267-268): the getters of an averaged object are means over the queries.

1. One query. addResult(n, ...) sets numPoints = n and stores the precision at n in pAt[n] for n <= 20 (:92-99); pAt[0] is "unused" (:72). When the query returned fewer than n results, getPrecisionAt(n) returns (numPoints * pAt[(int) numPoints]) / n (:115-116), which is relevant hits found / n. Correct.
2. The average. For each query with judgements (:206), average() adds stat.getPrecisionAt(j) to avg.pAt[j] for every j = 1..20 (:214-216), with that extrapolation already included, and adds stat.numPoints to avg.numPoints (:209). Then avg.numPoints /= m (:224) and avg.pAt[j] /= m (:228-230). Now avg.pAt[j] is the correct mean P@j for every j, and avg.numPoints is the mean result count M.
3. The second extrapolation. avg.getPrecisionAt(n) with n > M takes the one-query branch and returns M * avg.pAt[floor(M)] / n (:115-116), discarding the stored mean avg.pAt[n].
4. Not affected. log() prints P@i only for i < (int) numPoints (:162), so the SUMMARY shows only cutoffs below M, where the branch does not run. MAP, recall and MRR are averaged directly.

Query 1 returns 10 results, all relevant; query 2 returns 2, both relevant. M = (10 + 2) / 2 = 6 and avg.pAt[6] = (1.0 + 2/6) / 2 = 0.667, so for n > 6 the returned value is 6 x 0.667 / n = 4 / n.

14 of the 20 cutoffs are wrong; the SUMMARY prints P@1 to P@5 (Num Points 6.000), all right.

With f = floor(M) and R_i(k) the relevant hits of query i in its top min(k, length) results, the returned value is (M / f) x mean R_i(f) / n and the correct one is mean R_i(n) / n. They agree when every query returned the same number of results. With a whole-number M they agree exactly when no query has a relevant hit at ranks f + 1 to n; with a fractional M they differ except in special cases where the two sides happen to be equal (lists [relevant, relevant] and [relevant] give M = 1.5 and every cutoff right; [relevant, not relevant] and [relevant] give P@2 0.75 instead of 0.5 with no relevant hit after rank 1). The error goes both ways. When M < 1, f = 0 and the branch reads the never-filled avg.pAt[0], so every P@n is 0. (Earlier records said the value is right unless a query has a relevant hit between rank f + 1 and n; that holds only for a whole-number M or M < 1: with 4 and 7 results, M = 5.5, neither query has a relevant hit at rank 6 and P@6 is still 0.458 instead of 0.417.)

• Opt-in tool. lucene/benchmark's search-quality package is a developer evaluation tool; nothing in indexing or search uses it.
• Only programmatic reads. QueryDriver (trec/QueryDriver.java:101-102) and the package's documented sample (package-info.java:68-69) call only average() and log(), and log() prints only the cutoffs below the mean result count, so the printed SUMMARY never shows an affected cutoff. A program that reads avg.getPrecisionAt(n) (as upstream TestQualityRun does for n = 1..20) gets the wrong value silently.
• When. The judged queries return different numbers of results and n is above their mean. The default maxResults is 1000 (QualityBenchmark.java:64; QueryDriver and the sample also use 1000), which keeps the mean above 20 whenever the judged queries return more than 20 hits on average (broad queries on a larger collection; in upstream TestQualityRun it is 247.6), but not with setMaxResults below 20, a small collection, very selective queries, or queries with no hits.
• Size of the error. 0.4 instead of 0.6 (P@10, the example), 0.4125 instead of 0.5625 (P@8, an end-to-end run with setMaxResults(8)), 0.458 instead of 0.417 (P@6), and 0 at every cutoff when the mean result count is below 1. Per-query values, MAP, recall and MRR are right.
• How likely: rare. A maintainer who runs relevance experiments, Robert Muir (@rmuir), has said they use the module's QueryDriver to run the queries and send its output to trec_eval (#9048, 2017; also #3263, 2010), which bypasses this path.

allRelevant(hits) builds one query's stats the way QualityBenchmark does: new QualityStats(hits, 0) (hits relevant documents exist) and addResult(rank, true, 0) for ranks 1..hits. The second query's own getPrecisionAt(10) is 2/10 = 0.2, so the mean P@10 is 0.6, which average() stores; the averaged object returns 0.4.

The same numbers come out end to end: index 10 documents "apple" and 2 "banana", all judged relevant, and run QualityBenchmark with setMaxResults(10) for the two queries (the finder's longer test). The minimal test is also the regression test for the fix.

cp $REPRO/TestQualityStatsAveragePrecisionAt.java $LUCENE/lucene/benchmark/src/test/org/apache/lucene/benchmark/quality/
cd $LUCENE && ./gradlew :lucene:benchmark:test --tests org.apache.lucene.benchmark.quality.TestQualityStatsAveragePrecisionAt -Ptests.seed=DEADBEEF -Ptests.asserts=true
# also with -Ptests.asserts=false

TestQualityStatsAveragePrecisionAt > testAveragePrecisionAtIsMeanOfPerQueryValues FAILED
    java.lang.AssertionError: expected:<0.6> but was:<0.4>
:lucene:benchmark:test (FAILURE): 1 test, 1  failure

# the same with -Ptests.asserts=false, through the standalone runner on main (-ea, -da), on the 10.6.0 release jars (-ea, -da) and on the 10.5.2 and 9.12.3 release jars (-da)
# with the recommended fix: :lucene:benchmark:test (SUCCESS): 1 test

• Contract. The class holds results "for a single query or for a set of queries" (QualityStats.java:24); average() creates "a QualityStats object that is the average of the input QualityStats objects" (:189-194); getPrecisionAt(n) accepts any 1 <= n <= 20 (:104-109); getMRR() is "the Mean reciprocal rank over the queries" (:267-268).
• average() computes the right answer. It fills avg.pAt[j] with the mean P@j for every j = 1..20 (:214-216, :228-230); the stored means above the mean result count are never returned, and nothing documents why they should be discarded.
• trec_eval. Its P_n counts missing ranks as non-relevant per query and reports the mean over queries; the per-query getPrecisionAt follows it, and the exhaustive check found 0 per-query mismatches.
• Upstream test. TestQualityRun asserts 1.0 > avg.getPrecisionAt(j) for j = 1..20 (TestQualityRun.java:149-153), with maxResults = 1000 (:54, :85), so the branch never runs there and the assertion holds either way.
• History. The branch and average() came together in LUCENE-836 (98fa2d898d1, 2007); no javadoc, CHANGES entry, test or comment on LUCENE-836 (#1911) describes extrapolating an average.

The strongest case (the cross-model review, the intended-behaviour check and the refute-lens cross-model review): the averaged object is a synthetic query whose length is the mean result count, so cutoffs above it are extrapolated like a short query's, and log() agrees by printing only cutoffs below numPoints. The reading is coherent but undocumented and weak.

Why it fails: average() computes and stores the true mean for every cutoff, which would be pointless under that reading; a fractional length is not a query, and for a mean below 1 the formula returns 0 at every cutoff; the javadoc calls the object an average of the inputs; nothing documents synthetic-query semantics. log() hiding the cutoffs explains why nobody noticed, not that it is intended. The cross-model reviews rated the argument moderate, and every reviewer rejected it.

Validated in a scratch worktree of 645d4fe14a20 with gradle (seed DEADBEEF), with the repro test added to lucene/benchmark. The hunt's checks had already validated the same change against its longer tests, the exhaustive harness and upstream TestQualityRun.

In QualityStats.java (1 file, +4/-1): a private flag that average() sets on its result, checked in getPrecisionAt.

   private long docNamesExtractTime;
+  // true for the result of average(): pAt[] then already holds the mean P@n for every n
+  private boolean isAverage = false;
@@ getPrecisionAt(int n)
-    if (n > numPoints) {
+    if (!isAverage && n > numPoints) {
       return (numPoints * pAt[(int) numPoints]) / n;
@@ average(QualityStats[] stats)
     QualityStats avg = new QualityStats(0, 0);
+    avg.isAverage = true;

Trade-offs

• Smallest change; per-query objects behave exactly as before.
• getNumPoints() of the average still returns the mean result count, and log() is unchanged, so the SUMMARY output is identical.
• Edge cases unchanged: with no input stats every P@n stays 0; with no judged query (possible only with assertions off) the values stay NaN.

What the checks validated

• Stock + test: fails with assertions on and off (expected:<0.6> but was:<0.4>).
• + fix: the test passes with assertions on and off.
• + fix: org.apache.lucene.benchmark.quality.* (upstream TestQualityRun's 2 tests and the new test) passes with assertions on and off; the whole lucene/benchmark module (19 suites, 91 tests, 0 skipped) passes with assertions on.
• :lucene:benchmark:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis) passes.
• Earlier, in the hunt's checks: the finder's 2 tests, the independent reproduction's 4 tests, the refute reviewer's own test and the exhaustive harness (0 of 1,181,520 averaged P@n values wrong) pass with the same change.
• Not run: the module with assertions off, the full Lucene suite, nightly tests.

Keep the number of filled pAt slots, min(numPoints, 20) for a query and 20 for an average, and extrapolate only past it.

Trade-offs

Same behaviour as the flag, slightly more general, a few more lines.

Say in the javadoc that, for an averaged object, getPrecisionAt(n) with n > getNumPoints() is extrapolated from the mean result count.

Trade-offs

No behaviour change, but it documents a value that is not a mean and is 0 when the mean result count is below 1.

• Method (semi-formal). Translation validation of the stock jar: every ordered pair of relevance lists of length 0..5 (15,840 sets), every ordered triple of length 0..3 (3,375) and 39,861 random sets of 1-8 queries with 0-30 results (59,076 sets with a judged query in all), each compared at all 20 cutoffs with a reference written from trec_eval's definitions. Per query: 0 mismatches in 2,950,908 checks. Averaged: 341,559 of 1,181,520 P@n values (59,076 sets x 20 cutoffs) wrong, none when n <= the mean result count or when all lists have the same length; MAP, recall and MRR 0 mismatches; with the fix, 0 wrong. A mutant reference was detected (vacuity check).
• First signal. A planning probe of the benchmark quality metrics against trec_eval found 6 of 20 averaged cutoffs wrong; the hunt unit re-ran and characterised it, then reduced it to a public-API JUnit.