The benchmark quality package overview's TREC sample calls a 4-argument QualityBenchmark.execute that no release has ever had

The overview of org.apache.lucene.benchmark.quality (an exported package of the published benchmark module) introduces its only code block as "a sample code used to run the TREC 2006 queries 701-850 on the .Gov2 collection". The block sets int maxResults = 1000; and calls qrun.execute(maxResults, judge, submitLog, logger). QualityBenchmark has only execute(Judge, SubmissionReport, PrintWriter) and takes the result count from setMaxResults(int) (default 1000). Copied into a project, the call fails with one javac error, "method execute in class QualityBenchmark cannot be applied to given types; required: Judge,SubmissionReport,PrintWriter; found: int,Judge,SubmissionReport,PrintWriter", on main and on every release from 2.3.0 to 10.6.0. Copied verbatim, javac stops even earlier, because the DirectoryReader.open(directory) line ends with :.

The API change was deliberate: LUCENE-1116 (January 2008; its issue description speaks of moving "control over max-docs-to-retrieve from benchmark constructor to a setter method", though in the code the count was the first argument of execute) added setMaxResults(int) and updated QueryDriver and TestQualityRun, but not the page. 2.3.0, the package's first release, already had the change, so no release ever shipped a compiling sample. A 2012 pass that fixed out-of-date API usage in the sample rewrote its reader lines, left the call and introduced the :. Nothing in Lucene behaves wrongly; only the page is stale.

QualityBenchmark has one execute: public QualityStats[] execute(Judge judge, SubmissionReport submitRep, PrintWriter qualityLog) (line 96). It searches with the field maxResults (line 106), documented "maximal number of results to collect for each query. Default: 1000." (lines 63-64) and set by setMaxResults(int) (line 167). A class-file scan of 22 release benchmark jars (2.3.0 to 10.5.2) and main finds no execute(int, ...); all 142 release tags that have the package (2.3.0 to 10.6.0) carry the stale line.

trec/QueryDriver.java:96-98 and TestQualityRun.java:82-86: construct the QualityBenchmark, call qrun.setMaxResults(maxResults), then qrun.execute(judge, submitLog, logger).

Javadoc does not compile <pre><code> blocks, and the block has no {@link} that javadoc would check. The tests use the current API.

Any defect of the API itself; the free name directory, normal for a fragment. The sample's 1000 happens to equal the default, so deleting the first argument alone gives the intended count by accident; for any other value the reader must find setMaxResults.

• Who. Readers of the quality package overview who copy its sample. A 2009 java-user@ reply pointed a user to this page as the package's usage documentation. No query, document or index file reaches it.
• Default vs opt-in. The benchmark module is a separate, opt-in artifact. No configuration or special input is involved: the copied sample always fails.
• Scale. Every release from 2.3.0 (2008) to 10.6.0, and main (executed on 9 releases and main; the source of all 142 release tags read); no release ever shipped a compiling sample.
• Effect. A compile error, not wrong behaviour. javac names the 3-argument signature, and setMaxResults is in the class's method list.
• Not claimed. Runtime defects of the same package are separate findings (LFV-935, LFV-936, LFV-937, LFV-940).

The three lines are int maxResults = 1000;, SubmissionReport submitLog = null; and QualityStats stats[] = qrun.execute(maxResults, judge, submitLog, logger); (package-info.java lines 44, 64 and 65 at 645d4fe14a20).

javac writes its diagnostic to stderr; the test asserts that the compiler returns 0, so the JUnit failure reads expected:<0> but was:<1>. Sample.java:8 is the execute line.

The whole block, extracted mechanically from the page, fails first with ';' expected at the line-41 statement; with that : read as ;, the same single execute error remains (the finder's whole-block test and an independent reproducer's test agree).

No separate regression test is suggested: the test copies the page's lines, so it would not notice a later edit of the javadoc, and the javadoc build does not compile this block (compile-checking javadoc examples is the open wish LUCENE-8488, #9534). If a reviewer wants one, it is this same test with the fixed lines, which passes. TestQualityRun already covers the real API.

# the command above, main 645d4fe14a20, seed DEADBEEF, -ea (-da identical); lines in order, temp-dir prefix shortened, ... = test-framework notes / remaining runner frames
.../tempDir-001/Sample.java:8: error: method execute in class QualityBenchmark cannot be applied to given types;
    QualityStats stats[] = qrun.execute(maxResults, judge, submitLog, logger);
                               ^
  required: Judge,SubmissionReport,PrintWriter
  found:    int,Judge,SubmissionReport,PrintWriter
  reason: actual and formal argument lists differ in length
1 error
...
There was 1 failure:
1) testOverviewExecuteCallCompiles(org.apache.lucene.benchmark.quality.TestQualityPackageSample)
java.lang.AssertionError: expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([DEADBEEF:B8371FF647951C59]:0)
	...
	at org.apache.lucene.benchmark.quality.TestQualityPackageSample.testOverviewExecuteCallCompiles(TestQualityPackageSample.java:50)
	...
FAILURES!!!
Tests run: 1,  Failures: 1

# the same test on the 10.6.0 release jars, -da: tests_run=1 failures=1, the same single diagnostic

• Offered as working code. "Here is a sample code used to run the TREC 2006 queries 701-850 on the .Gov2 collection" (line 36), the package overview's only code block, with no pseudo-code marker.
• History. 98fa2d898d1 (LUCENE-836, #1911, 2007-07-27) added the package and the sample together, and the sample matched the then-existing execute(int maxResults, Judge, SubmissionReport, PrintWriter); it compiles at that commit. 9e65cd554f2 (LUCENE-1116, #2193, 2008-01-03), whose issue description (not the commit message) lists "move control over max-docs-to-retrieve from benchmark constructor to a setter method (API change)" (in the parent's code the count was the first argument of execute, not a constructor argument), introduced setMaxResults(int) and the 3-argument execute, and updated QueryDriver and TestQualityRun, not the page. The sample compiles at the parent 40eb1cd53fd and fails at 9e65cd554f2.
• A later fix pass missed it too. 647b5091697 (2012-09-12, "code snippet improvements: fix out of date api usage ...", on trunk; its branch_4x twin d8d5bf5e07d is the one in 4.0.0) rewrote the sample's reader lines for 4.0, kept the execute call and introduced the :.
• No record accepts the old form. The recorded searches (as of 2026-11-01) found no issue, PR, list thread or Q&A post that uses or defends the 4-argument call; Stack Overflow answers about the package use the 3-argument execute.

Strongest case for "not a bug": the block is illustrative; it uses an undeclared directory and has the : typo, so nobody can expect it to compile as is. The API change was deliberate, the error is loud and names the right signature, and the module is a developer tool whose real entry point is QueryDriver.

Why it fails: the undeclared directory is context a reader supplies, but no context makes a removed signature compile. A deliberate API change that leaves the package's only sample stale is a documentation defect, not intended behaviour; the same change updated the two other call sites. The packages are exported and published, and the page has no disclaimer. Every verifier and the adjudication called it real; it stays a Low documentation defect.

One fix was validated, with the diff applied to 645d4fe14a20; the alternatives were not run.

Javadoc only, lucene/benchmark/src/java/org/apache/lucene/benchmark/quality/package-info.java lines 41 and 65; mirrors QueryDriver.java:96-98.

- *     IndexReader ir = DirectoryReader.open(directory):
+ *     IndexReader ir = DirectoryReader.open(directory);
@@
- *     QualityStats stats[] = qrun.execute(maxResults, judge, submitLog, logger);
+ *     qrun.setMaxResults(maxResults);
+ *     QualityStats stats[] = qrun.execute(judge, submitLog, logger);

Trade-offs

None for users: documentation only, and no CHANGES entry is strictly needed. The block keeps its shape. It stays a fragment (it assumes directory and the imports, and closes neither the reader nor the files it opens), so a committer may reasonably prefer the QueryDriver pointer, a compiled {@snippet}, or deleting the sample. Backport: on 10.6.0 the block opens with <pre class="prettyprint"> (main's <pre><code class="language-java"> came with #15573), so the same two edits need new diff context.

What the checks validated

• The whole block extracted from the patched file (which differs from 645d4fe14a20 only on lines 41 and 65), compiled by a one-test check: passes 1 of 1 on main with assertions on and off and on the 10.6.0 jars. The stock block fails the same check (';' expected), so the check detects the bug.
• The minimal test with the fixed lines passes on main.
• ./gradlew :lucene:benchmark:test --tests "org.apache.lucene.benchmark.quality.*" -Ptests.seed=DEADBEEF: TestQualityRun, 2 tests, 0 failures.
• ./gradlew :lucene:benchmark:checkGoogleJavaFormat and :lucene:benchmark:renderJavadoc pass, and the rendered package page shows qrun.setMaxResults(maxResults); and the 3-argument execute.

Shorter, and cannot go stale again.

Trade-offs

The overview loses its only inline example of the API; QueryDriver is a main() with argument parsing, a less direct read.

Makes the sample compile without touching it.

Trade-offs

Reverses a deliberate 2008 API change (LUCENE-1116) and adds API surface only to match stale docs.

The generic wish of LUCENE-8488 (#9534, open), for example via the javadoc {@snippet} tag backed by compiled code.

Trade-offs

Would stop this class of drift, but it is a build-infrastructure change, not part of this fix.

• First signal. A probe compiled the page's sample (only the : read as ;) with javac against the stock main 645d4fe14a20 and 10.5.2 jars and got one error, the execute call (2026-10-29).
• Hunt. A hunt unit for javadoc examples in the analysis, benchmark and backward-codecs modules reduced it to one in-process javac JUnit test and ran a class-file presence scan of every signature the sample calls over 22 release benchmark jars (2.3.0 to 10.5.2) and main.
• Verification. An independent reproducer, a reviewer from another model family (OpenAI GPT-5.6) with a refute and an intended-behaviour lens, a devil's-advocate check, an adjudication and an audit (details under Independent checks).
• History. The introducing commit was found by building the quality package from the sources of each commit around the API change and compiling the sample against it.