The TokenStream API tutorial in the analysis package overview has seven examples that do not compile

The org.apache.lucene.analysis package overview (core package-info.java) is Lucene's tutorial for writing analyzers, token filters and attributes, and TokenStream's javadoc points readers to it for example code. Seven of its examples do not compile on main or on any release since 6.0.0: they call StandardAnalyzer(Version), WhitespaceTokenizer(Version) and FilteringTokenFilter(Version, TokenStream), removed in 5.0, and new LengthFilter(true, source, 3, Integer.MAX_VALUE), a signature gone since 4.4; one subclasses StandardAnalyzer, which is final; and the custom PartOfSpeechAttributeImpl lacks reflectWith(AttributeReflector), abstract since 6.0. The APIs changed on purpose, and no edit of the page since has fixed the stale calls. Written against the current API (no Version argument, new LengthFilter(source, 3, Integer.MAX_VALUE), a small Analyzer instead of a StandardAnalyzer subclass, a reflectWith method, and a final tagging filter so the tutorial also runs with assertions on), every example compiles and the tutorial prints the page's outputs.

Errors that come only from the page's placeholders (Version.LUCENE_XY, which the page asks you to substitute, and names it leaves undeclared) are not counted.

• StandardAnalyzer.java:34: public final class StandardAnalyzer, constructors (CharArraySet) :46, () :51, (Reader) :61.
• WhitespaceTokenizer.java:34-66 (analysis-common): (), (AttributeFactory), (int), (AttributeFactory, int).
• LengthFilter.java:44 (analysis-common): only (TokenStream in, int min, int max). FilteringTokenFilter.java:38: only (TokenStream in).
• AttributeImpl.java:34, :93, :99: clear(), reflectWith(AttributeReflector) and copyTo(AttributeImpl) are abstract. Analyzer has no Version API left.

• MyAnalyzer takes a Version (:609-613), but the last step calls new MyAnalyzer() (:903).
• :834-836 says the attribute "implements its abstract methods clear() and copyTo()".
• public static class PartOfSpeechTaggingFilter (:842) is not final. TokenStream requires final subclasses or a final incrementToken(), "checked when Java assertions are enabled" (TokenStream.java:74-76), so even the compile-fixed tutorial throws AssertionError under -ea.
• The four printed outputs list a token new (:658, :684, :889, :934) that the input text (:622, :901) does not contain.

• Who. Developers learning the TokenStream API from Lucene's own tutorial, the page TokenStream's javadoc points to ("You can find some example code for the new API in the analysis package level Javadoc", TokenStream.java:68).
• Evidence of use. Stack Overflow question 40052828 (2016-10-14): a 6.2.1 user quoted example 1 and asked why it does not compile; the answer explained that the Version argument was removed in 5.0.
• Default vs opt-in. No configuration: it is the page as published in the core module's javadoc.
• Scale. Seven of the page's example blocks (the others are fragments, illustrative classes with undefined names, or compile as they are; the last step's main, :898-925, fails in the assembled tutorial only because it calls new MyAnalyzer() against the page's own MyAnalyzer(Version), a knock-on of example 3). All seven are broken in every release from 6.0.0 to 10.6.0 and main; at least one since 3.0.0.
• Severity. Documentation only, loud, compile time. Once the compile errors are fixed, the part-of-speech steps also fail under -ea because the tagging filter is not final.

Source line 13 of the in-memory file is package-info.java:183; 14-18 are :267-271; 22-23 are :672-673 (also :617 and :876-877); 31 is :709; 36-41 are :809-831 reduced to its two overrides. The 7 errors cover the seven examples: two StandardAnalyzer(Version) calls, the final class, WhitespaceTokenizer(Version), LengthFilter(true, ...), super(version, in) and the missing reflectWith.

Why a plain JUnit run and not ./gradlew test: the in-process compiler reads the test JVM's class path, which must hold both lucene-core and lucene-analysis-common. In :lucene:analysis:common Lucene is on the module path, so the compiler sees neither jar; :lucene:core tests lack analysis-common.

The test is not needed to show the defect: one javac error per example does that. As a regression test (optional for a javadoc fix): the same method with EXAMPLES holding the fixed lines passes with assertions on and off; it cannot see the page itself, so it only guards the new lines against a future API change.

# build the jars (any build that produces them, for example ./gradlew assemble)
L=$LUCENE/lucene
CP=$L/core/build/libs/lucene-core-11.0.0-SNAPSHOT.jar:$L/codecs/build/libs/lucene-codecs-11.0.0-SNAPSHOT.jar
CP=$CP:$L/analysis/common/build/libs/lucene-analysis-common-11.0.0-SNAPSHOT.jar
CP=$CP:$L/test-framework/build/libs/lucene-test-framework-11.0.0-SNAPSHOT.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # from the Gradle cache
mkdir -p $REPRO/classes
javac -d $REPRO/classes -cp $CP $REPRO/TestAnalysisOverviewExamplesCompile.java
java -ea -Dtests.seed=DEADBEEF -cp $REPRO/classes:$CP org.junit.runner.JUnitCore org.apache.lucene.analysis.TestAnalysisOverviewExamplesCompile
# assertions off: -da instead of -ea (the same 7 errors)

# plain JUnit run, main 645d4fe14a20, seed DEADBEEF, -ea (the same with -da); abridged: the JUnit version, NOTE,
# WARNING and timing lines and most stack frames omitted, javac's lists of non-applicable candidate constructors
# shortened to [...]
There was 1 failure:
1) testExamplesCompile(org.apache.lucene.analysis.TestAnalysisOverviewExamplesCompile)
java.lang.AssertionError: expected:<[]> but was:<[line 13: no suitable constructor found for StandardAnalyzer(org.apache.lucene.util.Version)
    [...], line 14: no suitable constructor found for StandardAnalyzer(org.apache.lucene.util.Version)
    [...], line 14: cannot inherit from final org.apache.lucene.analysis.standard.StandardAnalyzer, line 22: no suitable constructor found for WhitespaceTokenizer(org.apache.lucene.util.Version)
    [...], line 23: constructor LengthFilter in class org.apache.lucene.analysis.miscellaneous.LengthFilter cannot be applied to given types;
  required: org.apache.lucene.analysis.TokenStream,int,int
  found:    boolean,org.apache.lucene.analysis.Tokenizer,int,int
  reason: actual and formal argument lists differ in length, line 31: constructor FilteringTokenFilter in class org.apache.lucene.analysis.FilteringTokenFilter cannot be applied to given types;
  required: org.apache.lucene.analysis.TokenStream
  found:    org.apache.lucene.util.Version,org.apache.lucene.analysis.TokenStream
  reason: actual and formal argument lists differ in length, line 36: PartOfSpeechAttributeImpl is not abstract and does not override abstract method reflectWith(org.apache.lucene.util.AttributeReflector) in org.apache.lucene.util.AttributeImpl]>
	at __randomizedtesting.SeedInfo.seed([DEADBEEF:456DAE84982F4242]:0)
	at org.junit.Assert.fail(Assert.java:89)
	...
	at org.apache.lucene.analysis.TestAnalysisOverviewExamplesCompile.testExamplesCompile(TestAnalysisOverviewExamplesCompile.java:96)

FAILURES!!!
Tests run: 1,  Failures: 1

• The page presents working code. "In this example we will create a WhiteSpaceTokenizer and use a LengthFilter to suppress all words that have only two or fewer characters" (package-info.java:597-598); "In this easy example a simple white space tokenization is performed. ... Here is the output:" (:647-649); "Now let's take a look at how the LengthFilter is implemented:" (:689). No "pseudo-code" or "illustrative" disclaimer anywhere on the page.
• Deliberate API changes that left the example calls stale. LUCENE-5859 (9938a39a872, 2014, "Remove Version from Analyzer constructors") did not touch it. LUCENE-4963 (8a7f2b6cc45, 2013) edited the page's listings but left both LengthFilter(true, ...) calls. LUCENE-6651 (4accc254774, 2015) made reflectWith abstract; its MIGRATE entry: "If you have implemented your own attribute, make sure to add the required method sigature [sic]. See the Javadocs for an example."
• No later edit fixed the stale calls. LUCENE-5392 (2b822c846a8, 2014-01-10) rewrote the three whitespace calls to new WhitespaceTokenizer(matchVersion), which matched trunk's API at the time, before LUCENE-5859 removed that constructor; later commits such as LUCENE-7318 (2016, the page's StandardAnalyzer link), #36 (2021, an enum modifier in the custom-attribute example) and LUCENE-9570 (reformatting) left the stale calls as they were; the last three (#14199, #14734, #15573, the last on 2026-01-15) are typo, grammar and markup fixes.
• Precedent. Upstream fixes stale javadoc examples: LUCENE-9046 (TermInSetQuery) and #16385 (per its CHANGES entry, the grouping package info) were merged; #16821 and #16823 (open issues) and #16904 (an open PR) report the same kind of defect in other package docs; LUCENE-8488 (#9534, open) asks for javadoc examples to be compiled.

The strongest case: these are illustrative fragments without imports; Version.LUCENE_XY is an explicit placeholder that tells the reader to adapt the code; and in 2012, on #4740 (LUCENE-3666), Uwe Schindler (in an IRC log posted there) advised leaving reflectWith out to keep the attribute example minimal ("too much information", "a minimum example would be ideal").

Why it fails: a reader adds missing imports and the code compiles, but no import and no Version value makes the removed constructors exist, and a final class cannot be subclassed. reflectWith was optional only while AttributeImpl had a reflection-based default; LUCENE-6651, also by Uwe Schindler, removed it, and its MIGRATE note requires the method. The page prints the output its code should produce. All verifiers and the adjudication rejected the intended-behaviour case. Verdict: a documentation defect.

One option was validated in a scratch worktree of 645d4fe14a20: A, a javadoc-only rewrite of the one file. B is a variant for the position-gap example; C lists optional text cleanups.

• Drop every Version matchVersion = Version.LUCENE_XY line and the MyAnalyzer(Version) field and constructor.
• new StandardAnalyzer() (#1); new WhitespaceTokenizer() (#3, #4, #7); new LengthFilter(source, 3, Integer.MAX_VALUE) (#4, #7).
• #2: StandardAnalyzer is final, so a small Analyzer with StandardTokenizer and LowerCaseFilter (the chain StandardAnalyzer() builds; its stop set is empty) that overrides getPositionIncrementGap, as the text above the example says.
• #5: LengthFilter(TokenStream in, int min, int max) { super(in); ... } without the @param version line; the FilteringTokenFilter listing's constructor becomes FilteringTokenFilter(TokenStream in).
• #6: add reflectWith(AttributeReflector) and name it in the sentence after the example.
• Make PartOfSpeechTaggingFilter final, and drop the stray new from the four printed outputs.

diff --git a/lucene/core/src/java/org/apache/lucene/analysis/package-info.java b/lucene/core/src/java/org/apache/lucene/analysis/package-info.java
index 4dfe81038f4..28a52a3abcd 100644
--- a/lucene/core/src/java/org/apache/lucene/analysis/package-info.java
+++ b/lucene/core/src/java/org/apache/lucene/analysis/package-info.java
@@ -179,8 +179,7 @@
  * something like:
  *
  * <pre id="analysis-workflow"><code class="language-java">
- *     Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY
- *     Analyzer analyzer = new StandardAnalyzer(matchVersion); // or any other analyzer
+ *     Analyzer analyzer = new StandardAnalyzer(); // or any other analyzer
  *     TokenStream ts = analyzer.tokenStream("myfield", new StringReader("some text goes here"));
  *     // The Analyzer class will construct the Tokenizer, TokenFilter(s), and CharFilter(s),
  *     //   and pass the resulting Reader to the Tokenizer.
@@ -263,8 +262,14 @@
  * Analyzer.getPositionIncrementGap(fieldName)}:
  *
  * <pre><code class="language-java">
- *   Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY
- *   Analyzer myAnalyzer = new StandardAnalyzer(matchVersion) {
+ *   Analyzer myAnalyzer = new Analyzer() {
+ *     {@literal @Override}
+ *     protected TokenStreamComponents createComponents(String fieldName) {
+ *       Tokenizer source = new StandardTokenizer();
+ *       return new TokenStreamComponents(source, new LowerCaseFilter(source));
+ *     }
+ *
+ *     {@literal @Override}
  *     public int getPositionIncrementGap(String fieldName) {
  *       return 10;
  *     }
@@ -606,23 +611,16 @@
  * <pre><code class="language-java">
  * public class MyAnalyzer extends Analyzer {
  *
- *   private Version matchVersion;
- *
- *   public MyAnalyzer(Version matchVersion) {
- *     this.matchVersion = matchVersion;
- *   }
- *
  *   {@literal @Override}
  *   protected TokenStreamComponents createComponents(String fieldName) {
- *     return new TokenStreamComponents(new WhitespaceTokenizer(matchVersion));
+ *     return new TokenStreamComponents(new WhitespaceTokenizer());
  *   }
  *
  *   public static void main(String[] args) throws IOException {
  *     // text to tokenize
  *     final String text = "This is a demo of the TokenStream API";
  *
- *     Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY
- *     MyAnalyzer analyzer = new MyAnalyzer(matchVersion);
+ *     MyAnalyzer analyzer = new MyAnalyzer();
  *     TokenStream stream = analyzer.tokenStream("field", new StringReader(text));
  *
  *     // get the CharTermAttribute from the TokenStream
@@ -655,7 +653,6 @@
  * demo
  * of
  * the
- * new
  * TokenStream
  * API
  * </pre>
@@ -669,8 +666,8 @@
  * <pre><code class="language-java">
  *   {@literal @Override}
  *   protected TokenStreamComponents createComponents(String fieldName) {
- *     final Tokenizer source = new WhitespaceTokenizer(matchVersion);
- *     TokenStream result = new LengthFilter(true, source, 3, Integer.MAX_VALUE);
+ *     final Tokenizer source = new WhitespaceTokenizer();
+ *     TokenStream result = new LengthFilter(source, 3, Integer.MAX_VALUE);
  *     return new TokenStreamComponents(source, result);
  *   }
  * </code></pre>
@@ -681,7 +678,6 @@
  * This
  * demo
  * the
- * new
  * TokenStream
  * API
  * </pre>
@@ -700,13 +696,12 @@
  *    * Create a new LengthFilter. This will filter out tokens whose
  *    * CharTermAttribute is either too short
  *    * (&lt; min) or too long (&gt; max).
- *    * {@literal @param} version the Lucene match version
  *    * {@literal @param} in      the TokenStream to consume
  *    * {@literal @param} min     the minimum length
  *    * {@literal @param} max     the maximum length
  *    *&#47;
- *  public LengthFilter(Version version, TokenStream in, int min, int max) {
- *     super(version, in);
+ *   public LengthFilter(TokenStream in, int min, int max) {
+ *     super(in);
  *     this.min = min;
  *     this.max = max;
  *   }
@@ -743,7 +738,7 @@
  *    * Create a new FilteringTokenFilter.
  *    * {@literal @param} in      the TokenStream to consume
  *    *&#47;
- *   public FilteringTokenFilter(Version version, TokenStream in) {
+ *   public FilteringTokenFilter(TokenStream in) {
  *     super(in);
  *   }
  *
@@ -828,18 +823,23 @@
  *   public void copyTo(AttributeImpl target) {
  *     ((PartOfSpeechAttribute) target).setPartOfSpeech(pos);
  *   }
+ *
+ *   {@literal @Override}
+ *   public void reflectWith(AttributeReflector reflector) {
+ *     reflector.reflect(PartOfSpeechAttribute.class, "partOfSpeech", pos);
+ *   }
  * }
  * </code></pre>
  *
  * <p>This is a simple Attribute implementation has only a single variable that stores the
  * part-of-speech of a token. It extends the <code>AttributeImpl</code> class and therefore
- * implements its abstract methods <code>clear()</code> and <code>copyTo()</code>. Now we need a
- * TokenFilter that can set this new PartOfSpeechAttribute for each token. In this example we show a
- * very naive filter that tags every word with a leading upper-case letter as a 'Noun' and all other
- * words as 'Unknown'.
+ * implements its abstract methods <code>clear()</code>, <code>copyTo()</code> and <code>
+ * reflectWith()</code>. Now we need a TokenFilter that can set this new PartOfSpeechAttribute for
+ * each token. In this example we show a very naive filter that tags every word with a leading
+ * upper-case letter as a 'Noun' and all other words as 'Unknown'.
  *
  * <pre><code class="language-java">
- *   public static class PartOfSpeechTaggingFilter extends TokenFilter {
+ *   public static final class PartOfSpeechTaggingFilter extends TokenFilter {
  *     PartOfSpeechAttribute posAtt = addAttribute(PartOfSpeechAttribute.class);
  *     CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
  *
@@ -873,8 +873,8 @@
  * <pre><code class="language-java">
  *   {@literal @Override}
  *   protected TokenStreamComponents createComponents(String fieldName) {
- *     final Tokenizer source = new WhitespaceTokenizer(matchVersion);
- *     TokenStream result = new LengthFilter(true, source, 3, Integer.MAX_VALUE);
+ *     final Tokenizer source = new WhitespaceTokenizer();
+ *     TokenStream result = new LengthFilter(source, 3, Integer.MAX_VALUE);
  *     result = new PartOfSpeechTaggingFilter(result);
  *     return new TokenStreamComponents(source, result);
  *   }
@@ -886,7 +886,6 @@
  * This
  * demo
  * the
- * new
  * TokenStream
  * API
  * </pre>
@@ -931,7 +930,6 @@
  * This: Noun
  * demo: Unknown
  * the: Unknown
- * new: Unknown
  * TokenStream: Noun
  * API: Noun
  * </pre>

Trade-offs

• Javadoc only, so no Lucene code changes.
• The gap example no longer shows StandardAnalyzer itself. With the default settings its index-time tokens are the same as StandardAnalyzer()'s, but it is not the same class: no setMaxTokenLength (both default to 255), no StopFilter (the default stop set is empty) and no normalize override, which matters only for query-time normalization. Replies on LUCENE-3055 (#4128) by Robert Muir and Uwe Schindler recommend writing your own Analyzer instead of subclassing StandardAnalyzer.

What the checks validated

• Page text (a checker builds nine units from eleven of the page's Java blocks, taken verbatim and assembled as the page describes; it adds imports, packages, a main for the two fragments, the attribute's own files, the tagging filter nested in MyAnalyzer, and gap and token prints for the gap unit; it compiles each unit with javac against the 645d4fe14a20 jars, runs the six with a main under -ea and compares four outputs with the page): stock page 8 of 9 units fail to compile; fixed page 9 of 9 compile, the six run, and all four printed outputs match exactly (This is a demo of the TokenStream API; This demo the TokenStream API twice; This: Noun demo: Unknown the: Unknown TokenStream: Noun API: Noun); the gap analyzer returns 10. Without final, the two part-of-speech units throw AssertionError: TokenStream implementation classes or at least their incrementToken() implementation must be final.
• Build: :lucene:core:checkGoogleJavaFormat, ecjLintMain and renderJavadoc pass (the fixed source has no matchVersion left); ./gradlew :lucene:core:test --tests "org.apache.lucene.analysis.*" with seed DEADBEEF passes (168 tests, 0 failures). The repro's fixed-lines variant passes with -ea and -da.

AnalyzerWrapper.getPositionIncrementGap delegates to the wrapped analyzer (AnalyzerWrapper.java:130-131); overriding it in the wrapper gives 10 (the devil's advocate ran this).

Analyzer standard = new StandardAnalyzer();
Analyzer myAnalyzer = new AnalyzerWrapper(Analyzer.PER_FIELD_REUSE_STRATEGY) {
  @Override protected Analyzer getWrappedAnalyzer(String fieldName) { return standard; }
  @Override public int getPositionIncrementGap(String fieldName) { return 10; }
};

Trade-offs

Keeps the exact StandardAnalyzer chain, but adds a second concept (wrapping, a reuse strategy) to a section about overriding one method; A matches the text better.

What the checks validated

Compiled and run by the devil's advocate: "control L266 gap=10".

• "The LengthFilter is part of the Lucene core" (:598): it is in analysis-common.
• The FilteringTokenFilter listing (:737-775) lacks the real class's end()/skippedPositions handling: a filter built on it reports end() position increment 0 where the real LengthFilter reports 1 (input "demo of", min length 3; the devil's advocate ran this). Show the real end() or point to the source.
• The "could begin" hint FirstTokenOfSentenceAttributeImpl (:948-968) could mention reflectWith too.

Trade-offs

Outside the compile claim; each is a one-line text change.

What the checks validated

Not run (a text change; the devil's advocate ran the end-position check only).

Method: non-formal (a compile check of documentation code). On 2026-10-28 the hunt unit for javadoc package and overview examples (siblings of #16904) compiled all 186 <pre><code class="language-java"> blocks of the 107 non-sandbox lucene/**/src/java files that have one against the stock main a9779ef3c193 jars with the JDK compiler, and flagged six blocks of the analysis package overview with API-level errors. A targeted public-API JUnit then compiled them verbatim (6 of 7 tests fail). The seventh example, a <pre id="analysis-workflow"> block the sweep's pattern skips, was found independently by the reproducer, the devil's advocate and the intended-lens cross-model reviewer. The verifiers' tests were later cut to the one-method repro.