Six test classes on main lost their @Test annotations in #16028 and no longer run

#16028 ("Misc junit4 test cleanups", main, 2026-05-02) removed 1,334 @Test lines from test* methods in 203 test files. For LuceneTestCase subclasses that is harmless, because LuceneTestCase also runs test* methods by name. Six of the classes do not extend LuceneTestCase, on purpose, and for them JUnit runs only @Test methods. So 18 test methods never run on main, and the build reports success: TestRuntimeDependenciesSane and TestCandidateMatcherVisibility report no tests, TestClassloadingDeadlock, TestModularLayer and TestScripts each report one passing entry named after the class with no method run, and the opt-in TestHunspellRepositoryTestCases fails with "No runnable methods". No shipped code, API or index format changes and no release has it; main and the 11.0 branch branch_11x do. Restoring the 6 imports and 18 annotations (the reverse of #16028 in these files) brings the 18 methods back: 17 run and pass, and TestScripts' GUI test is reported as skipped, as it was before #16028, because the build never passes its tests.gui option to the test JVM.

LuceneTestCase is @RunWith(RandomizedRunner.class) with @TestMethodProviders({LuceneJUnit3MethodProvider.class, JUnit4MethodProvider.class}) (LuceneTestCase.java:113-114). LuceneJUnit3MethodProvider selects public, non-static, no-argument methods named test* (LuceneJUnit3MethodProvider.java:39-42), skipping methods overridden or shadowed lower in the class hierarchy (:35-36); JUnit4MethodProvider selects @Test methods. Every class #16028 touched apart from the six below reaches LuceneTestCase.

RandomizedRunner without a @TestMethodProviders annotation falls back to JUnit4MethodProvider, which takes only @Test methods (an independent verifier read this from the randomizedtesting-runner 2.9.2 bytecode; the Gradle runs show the effect). The distribution tests' base class says so itself: "the default {@code LuceneTestCase} configuration setup is not used (you have to annotate test for JUnit, for example)" (AbstractLuceneDistributionTest.java:39-42).

• The build selects classes by file name, not by scanning for tests: setScanForTestClasses(false), include("**/Test*.class") (TestsAndRandomizationPlugin.java:545-546), and on main runs JUnit 4 through the vintage engine (:568-569, since #15949).
• On that engine a plain JUnit 4 class with no @Test yields no test; a RandomizedRunner class with no test method yields one passing entry named after the class (JUnit XML: <testcase name="TestClassloadingDeadlock" classname="Gradle Test Run :lucene:core:test" time="0.005"/>), and none of its methods run (nor its @BeforeClass: an independent check ran TestModularLayer without a distribution path, which its @BeforeClass rejects, TestModularLayer.java:69-74, and it still "passed"); Parameterized fails with "No runnable methods".
• Lucene turns off Gradle's per-task no-match failure (FailOnNoMatchingFilteredTestsPlugin.java:98) and fails only when a --tests filter matched nothing anywhere, a case it describes as "usually caused by a typo in the filter pattern" (:36-38). An unfiltered run reports nothing.

In these six files bfa16d51c21 removed exactly 6 import org.junit.Test; lines and 18 @Test lines, and nothing else. The six files are unchanged since, apart from one comment in TestClassloadingDeadlock (#16027).

• Who: Lucene developers and CI on main and branch_11x. Users of Lucene are not affected: #16028 changed only test sources.
• What is no longer checked on main:
  • TestRuntimeDependenciesSane (4): the tests of a modular project really load JUnit, Lucene core and the test and main source sets as named modules (added with LUCENE-10328, which fixed that module path).
  • TestClassloadingDeadlock (1): a forked JVM whose 20 threads initialise Codec, PostingsFormat, DocValuesFormat, TestSecrets, IndexWriter and SegmentReader at once must exit within 60 s. #15864 (2026-03-25, "Fix circular dependency deadlock in TestSecrets initialization") extended it; 38 days later it stopped running on main.
  • TestModularLayer (8): the binary distribution's module names, versions, exported and opened packages, service providers and multi-release jar. With TestScripts (1, the Luke launch check) these are the only test classes of lucene/distribution.tests, so the GitHub workflow dedicated to that module (run on every pull request and push to main) runs no test method.
  • TestCandidateMatcherVisibility (3): monitor's CandidateMatcher methods are callable from another package. Its compile-time part still holds; its three runtime calls no longer run.
  • TestHunspellRepositoryTestCases: the conformance run against Hunspell's own test data, documented as -Phunspell.repo.path (lucene/analysis/common/build.gradle:244), now fails whenever it is run with that property (without it, the class is skipped by its assumption, as in every version). Loud, not silent; no GitHub workflow sets the property.
• Default or opt-in: every default Gradle run, local or CI, no longer runs 16 methods in four classes (4 + 3 + 1 + 8). TestScripts' one method is also no longer found, but the build skipped it on every run already before #16028: the build registers tests.gui (on by default for CI builds) but never passes it to the test JVM, so even -Ptests.gui=true or CI=true leaves it skipped (an older, separate matter, from #14764; not claimed here). The Hunspell runner only matters when someone opts in.
• Scale: 18 methods; main since 2026-05-02, and branch_11x, the 11.0 branch cut from main on 2026-10-28, which has the six files unchanged; no release. A fix needs a backport to branch_11x to reach 11.0.0.
• No hidden regression found: with the annotations restored, every method that ran passes (17 of 18; the build skips TestScripts' GUI test), and the deadlock guard passed 20 of 20 forked runs earlier.

TestCandidateMatcherVisibilityRuns sits next to the class it runs, so main's own test source is compiled unmodified. It shows the plain JUnit 4 case, one of the six; the other five show the same cause in the build (see "The finding in 30 seconds"). Nothing random; the seed is only for the build.

No new file is needed to see it: on main, ./gradlew :lucene:core.tests:test --tests org.apache.lucene.core.tests.TestRuntimeDependenciesSane fails with "no tests matched the provided --tests filters?", while with the annotations restored it runs 4 tests ((SUCCESS): 4 tests); 10.6.0 keeps all 4 @Test annotations.

As a regression test: this same test (google-java-format wants the two assertEquals arguments on one line).

cp TestCandidateMatcherVisibilityRuns.java $LUCENE/lucene/monitor/src/test/org/apache/lucene/monitor/outsidepackage/
cd $LUCENE && ./gradlew :lucene:monitor:test --tests org.apache.lucene.monitor.outsidepackage.TestCandidateMatcherVisibilityRuns -Ptests.seed=DEADBEEF
# -Ptests.asserts=true and =false give the same result

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true (the same with =false)
TestCandidateMatcherVisibilityRuns > testItsThreeTestMethodsRun FAILED
    org.junit.ComparisonFailure: expected:<run=[3 failures=[]]> but was:<run=[1 failures=[initializationError(
    org.apache.lucene.monitor.outsidepackage.TestCandidateMatcherVisibility): Invalid test class
    'org.apache.lucene.monitor.outsidepackage.TestCandidateMatcherVisibility':
      1. No runnable methods]]>
:lucene:monitor:test (FAILURE): 1 test, 1  failure

# with the six classes' @Test restored: :lucene:monitor:test (SUCCESS): 1 test

# how the six classes look in the same build (not the repro):
# --tests ...core.tests.TestRuntimeDependenciesSane, or ...outsidepackage.TestCandidateMatcherVisibility:
> 1 test task executed but no tests matched the provided --tests filters?
# --tests org.apache.lucene.index.TestClassloadingDeadlock:
:lucene:core:test (SUCCESS): 1 test      (the entry is named TestClassloadingDeadlock; testDeadlock never runs)
# :lucene:distribution.tests:test:
:lucene:distribution.tests:test (SUCCESS): 2 tests      (entries TestModularLayer and TestScripts, no method runs)
# -Phunspell.repo.path=<a test directory> --tests ...hunspell.TestHunspellRepositoryTestCases:
TestHunspellRepositoryTestCases > initializationError FAILED
    org.junit.runners.model.InvalidTestClassError: ... 1. No runnable methods

• The classes: TestRuntimeDependenciesSane.java:22 "Intentionally not a subclass of {@code LuceneTestCase}."; TestClassloadingDeadlock.java:51-52 "This test does not extend LuceneTestCase to prevent static class initialization when spawned as a subprocess"; AbstractLuceneDistributionTest.java:39-42 "... (you have to annotate test for JUnit, for example)"; TestHunspellRepositoryTestCases.java:32-34 "checks all Hunspell's test data".
• The build: lucene/analysis/common/build.gradle:244 registers hunspell.repo.path as "Path to hunspell repo path for TestHunspellRepositoryTestCases".
• The migration notes: lucene/MIGRATE.md:45-46 "These apply if you extend LuceneTestCaseJupiter. LuceneTestCase (JUnit4) is unchanged."; for Jupiter, :48-50 "Methods must be annotated with @Test. Method prefix test* is not sufficient."
• The PR: body "This removes spurious Test annotations and some junit4-specific syntax."; commit message "Drop @Test annotation on methods that are test* prefixed." Merged 13 minutes after it was opened, with no review or comment. No CHANGES.txt or MIGRATE.md entry retires these tests; 10.6.0, branch_10_6 and branch_10_5 keep every annotation, as did branch_10x until it was decommissioned (#16911, 2026-10-29).

The strongest case (built on purpose by an intended-behaviour checker): the removal was deliberate, as the commit message says; nothing user-facing changes, so strictly it is not a Lucene bug; TestCandidateMatcherVisibility is a compile-time check that still compiles; the Hunspell runner is opt-in, run by no workflow, and fails loudly; and Lucene's no-match check is meant to catch filter typos, not empty classes.

Why it fails: the deliberate step is "drop @Test where test* already runs", which is false for exactly these six classes, and nobody chose to retire their tests. Their own comments, MIGRATE.md and the build option say they should run, and with the 18 annotations back all 18 are found again and 17 run and pass. For five of the six the loss is silent, including the whole distribution test module. It is a test-coverage regression and a small follow-up to #16028, not a product bug.

Option A was validated with Lucene's Gradle build in a scratch worktree of 645d4fe14a20. B and D are alternatives that change more; C is an optional guard against the same mistake recurring. The hunt's first fix sketch covered only three classes; the fix must cover all six.

git diff bfa16d51c21 bfa16d51c21^ -- the six files: TestRuntimeDependenciesSane +5, TestModularLayer +9, TestCandidateMatcherVisibility +4, TestClassloadingDeadlock, TestScripts and TestHunspellRepositoryTestCases +2 each. It applies cleanly to 645d4fe14a20.

--- a/lucene/core.tests/src/test/org/apache/lucene/core/tests/TestRuntimeDependenciesSane.java
+++ b/lucene/core.tests/src/test/org/apache/lucene/core/tests/TestRuntimeDependenciesSane.java
 import org.apache.lucene.index.IndexWriter;
+import org.junit.Test;
 
 /** Intentionally not a subclass of {@code LuceneTestCase}. */
 public class TestRuntimeDependenciesSane {
+  @Test
   public void testExternalDependenciesAreModules() {
# ... and the same for the other 17 test* methods of the six classes

Trade-offs

None for production code: test sources only.

What the checks validated

• Gradle, seed DEADBEEF: the repro passes with assertions on and off.
• Selected by name: TestCandidateMatcherVisibility 3 tests, TestRuntimeDependenciesSane 4, TestClassloadingDeadlock 1 (testDeadlock), TestHunspellRepositoryTestCases 1 (a one-case test directory); all pass.
• :lucene:distribution.tests:test on the assembled distribution: 9 tests, 0 failures, 1 skipped (8 of 8 TestModularLayer pass; testLukeCanBeLaunched is @RequiresGUI and is skipped on every run, also with -Ptests.gui=true and with CI=true, because the build does not pass tests.gui to the test JVM).
• Unfiltered :lucene:monitor:test (185 tests, 1 skipped) and :lucene:core.tests:test (7), and the org.apache.lucene.analysis.hunspell.* tests (158, 13 skipped): 0 failures.
• checkGoogleJavaFormat passes for core, core.tests, distribution.tests and analysis-common; monitor fails only on the repro file (one line to join).

Add @TestMethodProviders({LuceneJUnit3MethodProvider.class, JUnit4MethodProvider.class}) to the RandomizedRunner classes, or make the plain classes extend LuceneTestCase.

Trade-offs

Works against why the classes exist: they avoid LuceneTestCase on purpose, and the distribution tests must not depend on Lucene classes. It also does nothing for the plain and Parameterized classes unless they change runner.

What the checks validated

Not run.

A build or test-framework check that fails when a Test* class yields no test method, or that flags public void test*() methods with neither @Test nor LuceneTestCase.

Trade-offs

More build machinery; it must allow abstract bases, the Hunspell runner's assumption skip and the RandomizedRunner placeholder entry. Would have caught #16028 at once; not needed to fix this bug.

What the checks validated

Not run.

Use @org.junit.jupiter.api.Test, the direction main is taking with #16027.

Trade-offs

A larger change: Parameterized becomes @ParameterizedTest, and the RandomizedRunner features these classes use (RandomizedContext, the @RequiresGUI test group, @ThreadLeakScope) need Jupiter equivalents.

What the checks validated

Not run.

• Trigger: running TestHunspellRepositoryTestCases with hunspell.repo.path pointing at Hunspell 1.7.2's test data (hunt unit on Hunspell conformance) gave InvalidTestClassError: No runnable methods.
• Cause: git log on the class led to bfa16d51c21 (#16028). The finder scanned every class the commit touched for classes with neither LuceneTestCase ancestry nor @Test and found two plain JUnit 4 classes.
• Widening: an independent reproducer ran each touched class through the JUnit Platform launcher, as the build does, and found the three RandomizedRunner classes that also lost their tests.