SubmissionReport writes scores of 1000 or more as "1,200.5000", which trec_eval reads as 1.0

lucene/benchmark's search-quality package writes a TREC run file, one line per hit (qid Q0 docno rank score tag), through SubmissionReport. QueryDriver documents that file as the "output submission file for trec_eval". The score is formatted with NumberFormat.getInstance(Locale.ROOT) and 4 fixed decimals, and grouping is never turned off, so any score of 1000 or more is written with a thousands comma: 1200.5 as 1,200.5000. trec_eval reads the score with C atof(), which stops at the comma and returns 1.0, ignores the rank column and re-sorts each query's hits by the parsed score. A run Lucene ranked [1200.5, 900.25] is therefore evaluated as [900.25, 1.0], and every metric is computed on that order, with no error. It needs hit scores of 1000 or more (boosts, FeatureField or function scores, custom similarities); QueryDriver's default title-only BM25 queries stay far below, and through its own parser only a contrived query repeating one word 400 times got there. Lucene's ranking, the index and the in-process QualityStats numbers are not affected. Every release since 2.3.0 (2008) has it, through 10.6.0 and main; the fix is one line, nf.setGroupingUsed(false).

• QueryDriver usage: "submissionFile: output submission file for trec_eval" (QueryDriver.java:48); it writes the file with the stock SubmissionReport (:59-63).
• SubmissionReport javadoc: "Create a log ready for submission" (SubmissionReport.java:30). QualityBenchmark names two configurations, "Against existing judgements" and "For submission (e.g. for a contest)" (QualityBenchmark.java:32-41).
• trec_eval documents the line as qid iter docno rank sim run_id, with "similarity sim (a double)"; "the rank field is ignored here; internally ranks will be assigned by sorting by the sim field with ties broken determinstically (using docno)" (get_trec_results.c:17-26, its spelling).

1. The constructor builds NumberFormat.getInstance(Locale.ROOT) and sets 4 minimum and maximum fraction digits (SubmissionReport.java:48-50). Nothing calls setGroupingUsed, so the formatter is the pattern #,##0.0000 with grouping on and group size 3 (executed on JDK 25: 999.5 -> 999.5000, 1200.5 -> 1,200.5000, 1234567.0 -> 1,234,567.0000).
2. report() writes, per hit, the query id, Q0, the padded doc name, the 0-based rank, nf.format(sd[i].score) and the run name, separated by " \t " (:62-86, the score at :82).
3. trec_eval parses the field with text_results_ptr->sim = atof(lines[i].sim); (get_trec_results.c:210) and sorts by it (form_res_rels.c:111-113, comp_sim_docno at :266-275). No trec_eval source file calls setlocale, so atof runs in the "C" locale every C program starts in, and it reads the longest numeric prefix: 1,200.5000 is 1.0. Three independent C checks read grouped scores (1,200.5000, 1,200.0000, 1,000.5000) as 1.0.

Hits docA (1200.5) and docB (900.25) are written 1,200.5000 and 900.2500; trec_eval reads 1.0 and 900.25 and ranks docB first. In general a score of 1000 or more reads as its leading digit group (1 to 999): every score from 1000 to 1999.9999 becomes 1.0, and those hits are then ordered by document name, not by Lucene's ranking.

• Who. People who evaluate relevance with the quality package and an external scorer: the QueryDriver command line, QualityBenchmark with a SubmissionReport, or a direct report() call. Robert Muir described this workflow on PR #71 (2021): "generate submission.txt, then i run trec_eval etc from commandline myself".
• When. Only for hits scoring 1000 or more. Measured reach with public API: a BoostQuery of 10000 through QualityBenchmark scored 1139.5098 (written 1,139.5098); FeatureField linear queries with 10 query terms of impact 120 each scored 1200, written 1,200.0000, with no boosts, the custom-parser route the package docs suggest; QueryDriver's own SimpleQQParser with default BM25 got there only with a contrived query, the word needle repeated 400 times over 1000 documents of which one contains it: each copy adds the one-term score 2.956, so 400 x 2.956 = 1182.4163 (200 copies: 591.2). Whether realistic long queries of distinct terms reach 1000 was not measured. Ordinary title queries score far less. For scale, upstream TestQualityRun (title queries, BM25, the 578-line Reuters test file) writes 4,952 submission lines whose highest score is 4.9965.
• Effect. With trec_eval: silent. Each affected hit is read as its leading digit group (1 to 999) and sinks below every hit whose parsed score is higher; scores from 1000 to 1999.9999 all tie at 1.0 and are ordered by document name; and the metrics are computed on that order. Strict parsers (Java Double.parseDouble, Python float()) reject the line instead.
• Not affected. Lucene's search results, the index, and the metrics QualityBenchmark computes itself with a Judge.
• Releases. 2.3.0 (2008) through 10.6.0 and main. Likelihood: rare (a reasoned assessment, not a measured usage rate).

The index holds one document only because report() reads the stored doc name. SubmissionReport uses Locale.ROOT, so the test framework's random default locale does not matter; the seed only satisfies the framework. The assertion pins the existing 4-decimal format without grouping, which both fix options produce, so the same test is the regression test.

The trec_eval consequence (atof("1,200.5000") = 1.0, then a re-sort by score) is not asserted: it comes from trec_eval's source and three independent C atof() checks. Nobody ran the trec_eval binary itself.

mkdir -p $LUCENE/lucene/benchmark/src/test/org/apache/lucene/benchmark/quality/utils
cp $REPRO/TestSubmissionReportScoreFormat.java $LUCENE/lucene/benchmark/src/test/org/apache/lucene/benchmark/quality/utils/
cd $LUCENE && ./gradlew :lucene:benchmark:test --tests org.apache.lucene.benchmark.quality.utils.TestSubmissionReportScoreFormat -Ptests.seed=DEADBEEF -Ptests.asserts=true
# also with -Ptests.asserts=false

TestSubmissionReportScoreFormat > testScoreColumnHasNoGroupingSeparator FAILED
    org.junit.ComparisonFailure: expected:<1[]200.5000> but was:<1[,]200.5000>
:lucene:benchmark:test (FAILURE): 1 test, 1  failure

# the same with -Ptests.asserts=false, through the standalone runner on main (-ea, -da) and on the 10.6.0 and 10.5.2 release jars
# with setGroupingUsed(false): :lucene:benchmark:test (SUCCESS): 1 test, assertions on and off

• The documented purpose is trec_eval. QueryDriver.java:48 (added by LUCENE-2058, commit subject "specify trec_eval output file from commandline"); SubmissionReport.java:30 "Create a log ready for submission"; QualityBenchmark.java:32-41 "For submission (e.g. for a contest)". LUCENE-1116 (2008) made the line exactly the 6-column TREC run format.
• The reader needs a plain number. trec_eval: "similarity sim (a double)" (get_trec_results.c:21), parsed with atof (:210). 1,200.5000 is not a plain decimal number: strict parsers such as Java Double.parseDouble reject it, and C atof reads only its leading numeric prefix and returns 1.0.
• Grouping was inherited, not chosen. LUCENE-836 (98fa2d898d18, 2007) built NumberFormat.getInstance() and set only the fraction digits; the same commit called nf.setGroupingUsed(true) in QualityStats.log (today QualityStats.java:145-148), the human-readable summary. LUCENE-4199 (2012, "fix more default Locale uses") only changed getInstance() to getInstance(Locale.ROOT). The issue threads #1911, #2193 and #3134 never discuss the score format.
• No test pins it. The only test use, TestQualityRun.java:84, creates a SubmissionReport only when VERBOSE and never parses the output.
• Not claimed. The 4-decimal rounding can also tie scores that differ by less than 1e-4, which trec_eval then orders by document name; that is a weaker, separate point.

The strongest case for "works as designed" (the intended-behaviour verifier and the cross-model reviews, each rating it moderate): the class calls its output a "log" and says "Extend this class and override report(...) to create different reports" (SubmissionReport.java:30-31); in LUCENE-836 (#1911) the author wrote that the "format of submission log can be modified, by extending a logger class"; lucene/benchmark is a developer tool with no compatibility promise; ordinary BM25 scores stay below 1000; and no test pins the format.

Why it fails: the shipped QueryDriver uses the stock class and documents its output as the trec_eval submission file, so the default is the format, not a sample to subclass. Nothing documents a score range. The only explicit format choice in the package (grouping on) is for the human-readable log. And the trigger is legal API that the package docs invite ("Create sophisticated Lucene queries by supplying a different Quality query parser"). The extension point is still a fair reason the defect stayed latent: a user who needed another format could override report(...).

Both options were built in a scratch worktree of 645d4fe14a20 and run with gradle. Option A changes nothing for non-negative scores below 1000, which is why it is recommended; a maintainer may still prefer the explicit format string of option B.

One line in the SubmissionReport constructor (1 file, +1).

     nf = NumberFormat.getInstance(Locale.ROOT);
     nf.setMaximumFractionDigits(4);
     nf.setMinimumFractionDigits(4);
+    nf.setGroupingUsed(false); // the score column must be a plain number for trec_eval
   }

Trade-offs

Smallest change. Every non-negative score below 1000 is written byte for byte as today (0 differences in 1,000,000 random float scores in [0, 1000)); larger scores lose only the commas. Existing run files without such scores stay identical. The comparison and the test cover finite non-negative scores; Lucene scorers may not produce negative ones (LUCENE-7996). A negative score in a hand-built TopDocs also only loses its commas, and infinity and NaN are written as today.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle: the repro fails on stock and passes with the fix, assertions on and off.
• The whole lucene/benchmark module: 91 tests, 0 failures.
• Upstream TestQualityRun with -Ptests.verbose=true, which writes a submission report: passes, and its 4,952 submission lines are identical to stock's.
• :lucene:benchmark:check -x test (format, ecjLint, forbiddenApis; main and test sources, the new test included) passes.

Drop the NumberFormat field, its import and the constructor lines, and write String.format(Locale.ROOT, "%.4f", sd[i].score) in report() (1 file, +1/-6).

-              + nf.format(sd[i].score)
+              + String.format(Locale.ROOT, "%.4f", sd[i].score)

Trade-offs

No formatter state, and the intent is explicit. But String.format rounds half-up where DecimalFormat rounds half-even, so exact ties change in the fourth decimal: 305 of 1,000,000 random float scores below 1000 come out different from today (584.28125 gives 584.2813 instead of 584.2812), and about 8.8% of float scores between 1000 and 1,001,000 differ from option A. Harmless for trec_eval, but a few run files change in the last digit. It also writes infinity as Infinity where today's formatter writes ∞ (NaN stays NaN).

What the checks validated

• Same worktree: the repro passes; the quality tests with -Ptests.verbose=true pass with the same 4,952 submission lines as stock; :lucene:benchmark:check -x test passes; the standalone runner passes with assertions on and off.
• Not run: the whole module with this option.

• Hunt unit. Its main method checked the quality package's metrics against trec_eval's definitions; a side step of its second method read the callers (QualityBenchmark, SubmissionReport) and compared the written run line with trec_eval's reader. Found 2026-10-28.
• Witness. A public-API JUnit on the stock main jar (two tests: a direct report of [1200.5, 900.25] read back the way atof does, and QualityBenchmark with a BoostQuery), not a model.
• Method tier. Non-formal: code reading and executed tests.