PorterStemFilter's and StempelFilter's javadocs recommend LowerCaseTokenizer, removed in 8.0.0, and PorterStemFilter's example does not compile

PorterStemFilter (analysis-common, the classic English stemmer that EnglishAnalyzer uses) and StempelFilter (analysis-stempel, Polish) stem what they are given and do not lower-case it, so both class javadocs tell you to "use LowerCaseFilter or LowerCaseTokenizer farther down the Tokenizer chain". PorterStemFilter's javadoc then shows a 7-line MyAnalyzer whose tokenizer is new LowerCaseTokenizer(version, reader). LUCENE-8498 removed LowerCaseTokenizer in 8.0.0 (MIGRATE: "use a LetterTokenizer followed by a LowerCaseFilter") and missed these two files, so the advice names a class that Lucene has not shipped since, and the example fails with cannot find symbol: class LowerCaseTokenizer on every release from 8.0.0 to 10.6.0 and on main. Its (Version, Reader) constructor call had already stopped compiling in 5.0.0. With LetterTokenizer and LowerCaseFilter the example compiles and stems "Running Dogs" to [run, dog]. Documentation only: no code is wrong and nothing wrong ever runs.

PorterStemFilter.java:26-42:

 * Transforms the token stream as per the Porter stemming algorithm. Note: the input to the stemming
 * filter must already be in lower case, so you will need to use LowerCaseFilter or
 * LowerCaseTokenizer farther down the Tokenizer chain in order for this to work properly!   // :28
 *
 * <p>To use this filter with other analyzers, you'll want to write an Analyzer class that sets up
 * the TokenStream chain as you want it. To use this with LowerCaseTokenizer, for example, you'd  // :31
 * write an analyzer like this: <br>
 *
 * <pre><code class="language-java">
 * class MyAnalyzer extends Analyzer {
 *   {@literal @Override}
 *   protected TokenStreamComponents createComponents(String fieldName) {
 *     Tokenizer source = new LowerCaseTokenizer(version, reader);                             // :38
 *     return new TokenStreamComponents(source, new PorterStemFilter(source));
 *   }
 * }
 * </code></pre>

StempelFilter.java:29-31: "Note: the input to the stemming filter must already be in lower case, so you will need to use LowerCaseFilter or LowerCaseTokenizer farther down the Tokenizer chain in order for this to work properly!"

• These four lines (PorterStemFilter.java:28, :31, :38, StempelFilter.java:30) are the only mentions of LowerCaseTokenizer in any .java file under lucene/ (git grep).
• MIGRATE.md:1496-1500: "LowerCaseTokenizer and LowerCaseTokenizerFactory have been removed ... Instead, use a LetterTokenizer followed by a LowerCaseFilter." CHANGES.txt:6692 lists LUCENE-8498 in the "Lucene 8.0.0" section (:6611). (MIGRATE.md has no 7.x-to-8.0 heading, so the entry sits under "Migration from Lucene 8.x to Lucene 9.0".)
• The replacement is public: LetterTokenizer() (LetterTokenizer.java:34), LowerCaseFilter(TokenStream) in core, PorterStemFilter(TokenStream) (PorterStemFilter.java:56).
• Nothing in the build checks it: code inside javadoc <pre> blocks is not compiled.

1. 2001 / 2010: the "LowerCaseFilter or LowerCaseTokenizer" sentence enters PorterStemFilter (initial import) and StempelFilter (LUCENE-2298). Valid.
2. 2012-09-12: 647b5091697 ("code snippet improvements: fix out of date api usage ...") writes the example with new LowerCaseTokenizer(version, reader). Valid; 4.10.4 compiles it.
3. 2014: LUCENE-5388 (ab0e85fafed; branch_5x backport 55594bad9b0) removes Reader from tokenizer constructors and edits this snippet's createComponents line, but not its tokenizer line. From 5.0.0 the example does not compile; the class the prose names still exists (deprecated from 7.6.0).
4. 2018: LUCENE-8498 (c0d2975970d) deletes LowerCaseTokenizer and updates SimpleAnalyzer, StopAnalyzer, CharTokenizer, MIGRATE and CHANGES, but neither of these files. From 8.0.0 the advice names a missing class.
5. 2020-2026: later edits keep the stale text. LUCENE-9570 reformats both javadocs (8ef6a0da568, 7f8b7ffbcad), #15573 changes the example's markup, and #16063 removes a final modifier from incrementToken(), outside the javadoc.

• Who. Readers of the class javadoc of PorterStemFilter (the classic English stemmer in analysis-common, used by EnglishAnalyzer) or StempelFilter (the Polish stemmer in analysis-stempel, used by PolishAnalyzer). The example is read as usage advice: a 2012 dev@ thread, "tokenizer in porterStemmer", quotes it (while it was still valid).
• What they get. Advice naming a Lucene class absent from 61 releases (8.0.0-10.6.0) and main, and an example that does not compile on any release since 5.0.0 (compiled on 5.0.0, 7.7.3, 8.0.0, 9.12.3, 10.5.2, 10.6.0 and main; the same snippet on the others).
• What they do not get. Any runtime effect: both filters behave as their lower-case precondition says, and the compile error is loud.
• Priority. Low; likelihood "plausible" (anyone who copies the example hits it, every time).

Public API only, single thread. The class check is the smallest form of the defect: compiling the example's lines verbatim inside a wrapper class that supplies the imports and declares version and reader leaves exactly one error, the missing class, on main, 10.6.0, 10.5.2, 9.12.3 and 8.0.0, and a scan of the 38 jars built from main and of the cached 10.5.2 release jars finds no entry of that name.

The same file fails unchanged against the 10.6.0 and 10.5.2 release jars. Pass -Ptests.asserts explicitly: without it the build picks the mode from the seed (on for about 75% of seeds, TestsAndRandomizationPlugin.java:240-248), and for DEADBEEF in analysis-common it is off; the bare command fails the same way.

Regression test: none suggested. The fix stops naming the class rather than bringing it back, so this test keeps failing after the fix (checked), and Lucene does not compile javadoc code.

cp $REPRO/TestPorterStemFilterJavadocTokenizer.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/en/
cd $LUCENE
./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.en.TestPorterStemFilterJavadocTokenizer -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false instead (same failure)

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (-Ptests.asserts=false: the same failure)
TestPorterStemFilterJavadocTokenizer > testRecommendedLowerCaseTokenizerExists FAILED
    java.lang.AssertionError: expected the LowerCaseTokenizer the javadoc recommends, but: java.lang.ClassNotFoundException: org.apache.lucene.analysis.core.LowerCaseTokenizer
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:ED6645F24C759196]:0)
        at org.apache.lucene.analysis.en.TestPorterStemFilterJavadocTokenizer.testRecommendedLowerCaseTokenizerExists(TestPorterStemFilterJavadocTokenizer.java:31)

        Caused by:
        java.lang.ClassNotFoundException: org.apache.lucene.analysis.core.LowerCaseTokenizer
            at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
            ...
1 test completed, 1 failed

# the example's lines compiled verbatim in a wrapper class with imports, version and reader declared (main 645d4fe14a20):
example does not compile: [example-line-4: cannot find symbol   symbol:   class LowerCaseTokenizer   location: class Outer.MyAnalyzer]
PorterStemFilter prose names LowerCaseTokenizer, which is not in these jars
StempelFilter prose names LowerCaseTokenizer, which is not in these jars

• Removed on purpose, with a replacement. MIGRATE.md: LowerCaseTokenizer "combined tokenization and filtering in a way that broke token normalization, so they have been removed. Instead, use a LetterTokenizer followed by a LowerCaseFilter." It was deprecated in 7.6.0 (99df2bd2f35) and deleted in 8.0.0 (c0d2975970d).
• Missed by the removal. c0d2975970d changed 32 files; PorterStemFilter.java and StempelFilter.java are not among them, and the 10 comments of #9544 never mention them. Both files are byte-identical in 7.7.3 and 8.0.0.
• Meant to track the API. A 2012 commit titled "code snippet improvements: fix out of date api usage" (647b5091697) rewrote this snippet, and LUCENE-5388 (2014) later updated its createComponents signature but left the tokenizer line; #12693 (2023) fixed the Analyzer class example, closing #12666.

The case for "intended": Lucene javadoc snippets are illustrative. This one is a fragment (no imports, no enclosing class) and never declared version or reader, Analyzer.java's javadoc uses fictional FooTokenizer placeholders, the build does not compile <pre> blocks, MIGRATE and CHANGES already document the replacement, and later commits edited the class without changing the advice: LUCENE-9570 reformatted both javadoc blocks (2020-2021), #15573 changed the example's markup, and #16063 removed a final modifier from incrementToken(), outside the javadoc.

Why it fails: missing imports and an undeclared variable are gaps a reader fills in; a class that was deliberately deleted is not, and nothing a reader adds makes new LowerCaseTokenizer(...) compile. The prose names it as the way to meet the filter's precondition. MIGRATE confirms the text is stale rather than excusing it, and the later edits were mechanical sweeps (LUCENE-9570 "code reformatting", #15573 "Replace prettyprint with code tag for prism", #16063 "Remove redundant final modifiers"). Every verifier, the executing cross-model reviewer and the adjudication rejected the argument; what remains true is that the impact is small, which is why it is Low.

One fix was validated with gradle in a scratch worktree of 645d4fe14a20; the alternatives were not run.

Exactly the replacement MIGRATE.md prescribes for LowerCaseTokenizer, so the example keeps its meaning (split on letters, lower-case, stem) and no longer needs the undeclared version or reader. 2 files, +7/-8, comments only.

- * filter must already be in lower case, so you will need to use LowerCaseFilter or
- * LowerCaseTokenizer farther down the Tokenizer chain in order for this to work properly!
+ * filter must already be in lower case, so you will need to use LowerCaseFilter farther down the
+ * Tokenizer chain in order for this to work properly!
- * the TokenStream chain as you want it. To use this with LowerCaseTokenizer, for example, you'd
- * write an analyzer like this: <br>
+ * the TokenStream chain as you want it. To use this with LowerCaseFilter, for example, you'd write
+ * an analyzer like this: <br>
- *     Tokenizer source = new LowerCaseTokenizer(version, reader);
- *     return new TokenStreamComponents(source, new PorterStemFilter(source));
+ *     Tokenizer source = new LetterTokenizer();
+ *     return new TokenStreamComponents(source, new PorterStemFilter(new LowerCaseFilter(source)));
  (StempelFilter.java: "LowerCaseFilter or LowerCaseTokenizer" -> "LowerCaseFilter")

Trade-offs

• No behaviour, API or test change; trivially reviewable.
• Keeps the old "farther down the Tokenizer chain" wording; the chain must be tokenizer -> LowerCaseFilter -> stemmer, so a reword such as "apply LowerCaseFilter before this filter" is optional.
• LetterTokenizer and LowerCaseFilter are plain text in the <pre> block, not {@link}s, like the names they replace; renderJavadoc accepts them, and the control test checks the example with its own imports.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle --offline, OpenJDK 25.0.3, seed DEADBEEF.
• Stock: the one-test repro fails with assertions on and off.
• With the diff: git grep LowerCaseTokenizer -- 'lucene/**/*.java' finds nothing; a script confirmed the patched example equals, line for line, the MyAnalyzer of a control test that asserts "Running Dogs" -> [run, dog].
• :lucene:analysis:common:test --tests 'org.apache.lucene.analysis.en.*' with the control: 22 tests, 0 failures, 1 skipped (TestKStemmer.testCreateMap, @Ignored sample code), including TestPorterStemFilter (4) and TestPorterStemFilterFactory (2); :lucene:analysis:stempel:test, the whole module: 19 tests, 0 failures.
• renderJavadoc, checkGoogleJavaFormat and ecjLintMain of both modules pass; the rendered PorterStemFilter.html no longer mentions LowerCaseTokenizer.
• The one-test repro still fails after the fix, as expected: only comments change.
• Not run: the whole analysis-common test task or ./gradlew check.

EnglishAnalyzer and PolishAnalyzer build StandardTokenizer -> ... -> LowerCaseFilter -> ... -> the stemmer.

Trade-offs

Closer to today's practice, but it changes the example's tokenizer (Unicode word breaks instead of letters), which the old text never meant.

Keep only the corrected prose.

Trade-offs

Smallest text, but loses the one hand-written chain in the class docs; PorterStemFilterFactory's javadoc already shows a factory-based chain.

For example {@snippet} with compiled sources.

Trade-offs

One systematic way to stop this kind of drift (a resolvable {@link}, an API-name lint or a focused documentation test could also catch a removed class), but a build-tooling change far beyond a three-line fix.

• Hunt. The finder extracted the class-javadoc example verbatim from the source and compiled it with javax.tools against the stock main jars: errors for LowerCaseTokenizer and the two undeclared names version and reader. Declaring those two leaves the class as the only error. An API-presence scan over release tags dated the removal, and a grep over the non-sandbox sources found the StempelFilter sentence.
• Minimized to the one-test class check of the repro section.
• Tier. Non-formal: compiling documentation code and scanning jars, not a check against exact semantics.