Four Lucene test methods never run: their names start with "tes" or "Test" instead of "test"

Lucene's test runner treats a method as a test only if it is public, non-static, takes no arguments and its name starts with lowercase test, or if it carries @Test. Four methods in Lucene's own test sources miss that by one letter: TestXYRectangle.tesInvalidMinMaxX and tesInvalidMinMaxY, TestOperations.TestOptionalOptionalAcceptsState0 and TestCommonGramsFilter.TestFirstAndLastStopWord. Every test run skips them silently: gradle runs 5, 23 and 10 tests for the three classes instead of 7, 24 and 11. Renamed, the two TestXYRectangle methods fail, because they look for 5 > 4 in a message that reads minX must be lower than maxX, got 5.0 > 4.0. No shipped code is wrong; the real loss is that XYRectangle's reversed-bounds IllegalArgumentException has had no running test since it was added in 2020.

LuceneTestCase registers two method providers, @TestMethodProviders({LuceneJUnit3MethodProvider.class, JUnit4MethodProvider.class}) (LuceneTestCase.java:114). The first keeps a method only if m.getName().startsWith("test"), it is public and non-static, and it takes no parameters (LuceneJUnit3MethodProvider.java:39-42); the second keeps @Test methods. The class javadoc says the same: "Any test method with a testXXX prefix is considered a test case. Any test method annotated with [org.junit.Test] is considered a test case." (LuceneTestCase.java:90-91). Anything else is silently ignored.

None has @Test, @Ignore, @AwaitsFix, @Nightly or a comment saying it is disabled, and no code calls them.

XYRectangle(float minX, float maxX, float minY, float maxY) throws new IllegalArgumentException("minX must be lower than maxX, got " + minX + " > " + maxX) (and the minY twin) from a plain if (XYRectangle.java:36-44). Java prints the float 5 as 5.0, so the message is got 5.0 > 4.0 and contains("5 > 4") is false. LUCENE-9218 made the bounds floats in the same commit that added the exception and these two tests, so the checks never matched in any version.

Lucene's build turns off error-prone's JUnit3TestNotRun ("we don't use junit3") and JUnit4TestNotRun ("noisy") (ErrorPronePlugin.java:542, :548); per error-prone's documentation the former looks only at junit.framework.TestCase subclasses anyway, and whether the latter would flag these methods under Lucene's RandomizedRunner is unknown (neither was tested). The JUnit 5 base class checks test* methods for @Test (LuceneTestCaseJupiter.java:608-634), but only for Jupiter tests and only for names that already start with test.

• Who. Lucene developers and CI. No user configuration or API is involved, and test sources are not in the released jars.
• What is lost. The two TestXYRectangle methods are the only tests that build an XYRectangle with reversed bounds; every other new XYRectangle(...) in the test trees passes ordered bounds. If that check regressed (for example back to an assert, as it was before 8.5.0), no running test would notice.
• What is not lost. TestFirstAndLastStopWord ("the of" gives ["the_of"]) repeats the case that the running testCommonGramsQueryFilter already checks with two common words in the other order ("of the" gives ["of_the"], TestCommonGramsFilter.java:105). TestOptionalOptionalAcceptsState0 adds nothing beyond the running testOptionalAcceptsState0 and testOptionalOptional. Both dead bodies pass when run under new names, so they hide no product bug.
• Scale. Two name scans find exactly these four: the finder's over every lucene/**/src/test tree, sandbox included (public no-arg void methods whose names start with tes, tset, tets, tst, Test, TEST or Tset but not with test), and an independent verifier's over the non-sandbox trees (Test[A-Z]*, tes[^t]*, tset*, tets*, tst*, est[A-Z]*). They look for near-miss spellings only. Public no-arg methods that never run on purpose do exist, for example the compile-only TestAnalyzers._testStandardConstants (LUCENE-1150 comment, @SuppressWarnings("unused")).
• Library behaviour. Correct: XYRectangle rejects reversed bounds with assertions on and off; Operations.optional and CommonGramsQueryFilter work.

The test compiles the checkout's own, unmodified TestXYRectangle, TestOperations and TestCommonGramsFilter and calls Request.aClass(c).getRunner().getDescription() for each: that builds Lucene's RandomizedRunner (from LuceneTestCase's @RunWith) and lists the tests it would run, without running them. Every declared public, non-static, no-arg method that is not in that list is reported. Apart from their test methods the three classes declare no public no-arg instance methods, so the list should be empty (true for these three classes; it is not a general rule). d.isSuite() covers -Dtests.iters > 1, where the runner groups each method's repetitions under a suite named after the method (checked with 5 iterations).

The test classes of two modules are compiled together, so the command runs outside gradle, against the checkout's built jars of lucene-core, -codecs, -test-framework and -analysis-common plus junit 4.13.2, randomizedtesting-runner 2.9.2 and hamcrest 3.0 ($DEPS, the versions in gradle/libs.versions.toml).

Without the test: on a stock checkout, ./gradlew :lucene:core:test --tests org.apache.lucene.geo.TestXYRectangle reports 5 tests for a class with 7 test-shaped methods, and --tests 'org.apache.lucene.geo.TestXYRectangle.tesInvalidMinMaxX' fails the build with "1 test task executed but no tests matched the provided --tests filters?".

No separate regression test is suggested: once renamed, the four methods are running tests themselves. The minimal test passes with the fix.

# the command above, main 645d4fe14a20, seed DEADBEEF, -ea (-da identical)
java.lang.AssertionError: expected:<[]> but was:<[TestCommonGramsFilter#TestFirstAndLastStopWord, TestOperations#TestOptionalOptionalAcceptsState0, TestXYRectangle#tesInvalidMinMaxX, TestXYRectangle#tesInvalidMinMaxY]>
FAILURES!!!
Tests run: 1,  Failures: 1

# gradle, stock checkout
<testsuite name="org.apache.lucene.geo.TestXYRectangle" tests="5" skipped="0" failures="0" errors="0"
<testsuite name="org.apache.lucene.util.automaton.TestOperations" tests="23" skipped="0" failures="0" errors="0"
<testsuite name="org.apache.lucene.analysis.commongrams.TestCommonGramsFilter" tests="10" skipped="0" failures="0" errors="0"
$ ./gradlew :lucene:core:test --tests 'org.apache.lucene.geo.TestXYRectangle.tesInvalidMinMaxX'
> 1 test task executed but no tests matched the provided --tests filters?

# gradle, tes -> test only (message checks unchanged)
TestXYRectangle > testInvalidMinMaxX FAILED
    java.lang.AssertionError
        at org.apache.lucene.geo.TestXYRectangle.testInvalidMinMaxX(TestXYRectangle.java:33)
TestXYRectangle > testInvalidMinMaxY FAILED
    java.lang.AssertionError
        at org.apache.lucene.geo.TestXYRectangle.testInvalidMinMaxY(TestXYRectangle.java:44)
7 tests completed, 2 failed
# the messages those checks look at: minX must be lower than maxX, got 5.0 > 4.0 / minY must be lower than maxY, got 5.0 > 4.0

• Written as tests. tesInvalidMinMaxX / Y carry the javadoc "maxX must be gte minX" / "maxY must be gte minY"; TestFirstAndLastStopWord carries "Test CommonGramsQueryFilter when first and last words are stopwords."; all four bodies are assertions only (expectThrows, assertMinimalDFA, assertTokenStreamContents), and no code calls them. Lucene's test code disables tests on purpose with 49 active @AwaitsFix / @Ignore annotations; none of the four has a marker.
• History shows typos. LUCENE-9218 (4a54ffb553ad, apache/lucene-solr#1252, 2020) created TestXYRectangle with the two tes methods next to four correctly named siblings (testNaN, testPositiveInf, testNegativeInf, testEqualsAndHashCode). #14218 (de1ed71261d5, 2025, "Clean up more dead states creation by Automata/Operations"; its commit message lists "split some large tests into smaller tests for easier management") split the running testOptional into four methods; three got test, one got Test. TestFirstAndLastStopWord came with SOLR-1674 (2009) and kept its name through the move into Lucene (LUCENE-2413, 2010) and the 2021 class rename (LUCENE-8626).
• Precedent. LUCENE-2037 (db067550e89, 2010) added a check because "i found lots of tests that simply forgot to add @Test and were never ran"; a 2010 commit (d1ecb22f8a9) renamed tesStopwordsParsing to testStopwordsParsing; LuceneTestCaseJupiter enforces @Test on test* methods "so that they're not silently ignored" (LuceneTestCaseJupiter.java:610).

Strongest case for "not a bug": the runner and XYRectangle work exactly as documented; LuceneTestCase never promises to run every public no-arg method, and a public method may be a helper; the exact exception text is not a contract; two of the four methods are redundant. The first cross-model reviewer (refute lens) returned "not real" on these grounds, at medium confidence and partly on procedure (it could not run Java).

Why it fails: nobody claims the runner is wrong; the defect is in the test sources. The same reviewer wrote that "static review nevertheless strongly indicates accidental test-name typos" and that the renames and message fixes "look appropriate". A helper would not carry test javadoc and assertion-only bodies, and nothing marks the methods as disabled. The exact text matters only because the tests' own assertion checks it. Redundancy narrows the impact but does not make the names intended. Verdict: a minor, test-only defect (adjudication).

One fix was validated with gradle in a scratch worktree of 645d4fe14a20; the alternatives were not run.

Rename tesInvalidMinMaxX, tesInvalidMinMaxY, TestOptionalOptionalAcceptsState0 and TestFirstAndLastStopWord to testInvalidMinMaxX, testInvalidMinMaxY, testOptionalOptionalAcceptsState0 and testFirstAndLastStopWord, and in TestXYRectangle change both contains("5 > 4") to contains("5.0 > 4.0"). 3 files, +6/-6, test sources only.

--- TestXYRectangle.java
-  public void tesInvalidMinMaxX() {
+  public void testInvalidMinMaxX() {
-    assertTrue(expected.getMessage().contains("5 > 4"));
+    assertTrue(expected.getMessage().contains("5.0 > 4.0"));
   (the same two changes in tesInvalidMinMaxY)
--- TestOperations.java
-  public void TestOptionalOptionalAcceptsState0() {
+  public void testOptionalOptionalAcceptsState0() {
--- TestCommonGramsFilter.java
-  public void TestFirstAndLastStopWord() throws Exception {
+  public void testFirstAndLastStopWord() throws Exception {

Trade-offs

None for users: no library code changes. Both parts are needed: renaming without the message change makes TestXYRectangle fail 2 of 7. No CHANGES entry is strictly needed.

What the checks validated

• gradle --no-daemon, seed DEADBEEF, OpenJDK 25.0.3: TestXYRectangle, TestOperations and TestCommonGramsFilter run 7, 24 and 11 tests (stock 5, 23, 10), 0 failures, with -Ptests.asserts=true and false.
• Closest suites, 0 failures: org.apache.lucene.geo.* (17 classes, 195 tests, 7 skipped), org.apache.lucene.util.automaton.* (15 classes, 239 tests, 2 skipped), org.apache.lucene.analysis.commongrams.* (3 classes, 19 tests).
• The minimal test passes against the fixed sources with assertions on and off.
• Renaming only: TestXYRectangle 7 tests, 2 failures (TestXYRectangle.java:33, :44).

assertEquals("minX must be lower than maxX, got 5.0 > 4.0", expected.getMessage()) and the minY twin.

Trade-offs

Stricter, but ties the tests to the full wording; the recommended option keeps the original substring-style check and is the smaller diff.

TestFirstAndLastStopWord and TestOptionalOptionalAcceptsState0 add no coverage that running tests do not already give.

Trade-offs

Defensible, but renaming is a smaller, less debatable change and keeps the explicit cases their authors wrote. The TestXYRectangle pair must be renamed either way.

A check in LuceneTestCase, like the Jupiter guard (LuceneTestCaseJupiter.java:608-634), that fails on public no-arg instance methods named tes... or Test... but not test....

Trade-offs

A heuristic: a legitimate public helper with such a name would trip it (the near-miss scans found none besides these four). Turning on error-prone's JUnit3TestNotRun would likely not help, since it looks only at junit.framework.TestCase subclasses; whether JUnit4TestNotRun would flag these methods in classes run by Lucene's RandomizedRunner was not tested.

• Context. A hunt unit swept Lucene for input checks that exist only as assert. Reading TestXYRectangle to see how XYRectangle's bounds checks are tested, the finder noticed that tesInvalidMinMaxX and tesInvalidMinMaxY cannot run.
• Widening. The finder's scan of every lucene/**/src/test tree, sandbox included, for public no-arg void methods whose names start with tes, tset, tets, tst, Test, TEST or Tset but not with test found exactly four. An independent verifier's scan of the non-sandbox test trees with its own patterns (Test[A-Z]*, tes[^t]*, tset*, tets*, tst*, est[A-Z]*) found the same four. Both match name patterns only.
• Confirmation. The unmodified classes ran 5, 23 and 10 tests with none of the four; renamed copies ran 42 tests with only the two XYRectangle checks failing; the corrected checks passed.
• Tier. Non-formal: the first signal was reading a test and the widening was a text scan; no model or bounded check was involved.