The analysis compound package overview's examples do not compile on any release since 4.0.0, and its output table shows offsets the filters no longer emit

The org.apache.lucene.analysis.compound package overview shows how to split German and Swedish compound words with HyphenationCompoundWordTokenFilter and DictionaryCompoundWordTokenFilter, in three examples and a table of the tokens the filter produces. Copied into code, none of the three examples compiles on any release since 4.0.0: they call getHyphenationTree(Reader), new WhitespaceTokenizer(Reader) and the String[] dictionary constructors, all gone on main (the Version-less WhitespaceTokenizer(Reader) was back in 4.10.x only; today: getHyphenationTree(String) or (InputSource), new WhitespaceTokenizer() plus setReader, and a CharArraySet dictionary). Once ported, the page's loop throws IllegalStateException because it never calls reset(). With reset() it runs, but every word part carries the offsets of the whole word, for example (Rind,0,29) where the table shows (Rind,0,4): a deliberate change of 4.4.0 (LUCENE-5018, for a match version of 4.4 or later; unconditional since 5.0.0) that the table never picked up. The filters work as designed; only the page is wrong, and a javadoc-only fix solves it (validated); shrinking the block is an alternative (not run).

The overview opens with "A filter that decomposes compound words you find in many Germanic languages into the word parts. This example shows what it does:" (package-info.java:19-20 at 645d4fe14a20), the input Rindfleischüberwachungsgesetz Drahtschere abba and an output table (:40-64): (Rindfleischüberwachungsgesetz,0,29), (Rind,0,4,posIncr=0), (fleisch,4,11,posIncr=0), (überwachung,11,22,posIncr=0), (gesetz,23,29,posIncr=0), (Drahtschere,30,41), (Draht,30,35,posIncr=0), (schere,35,41,posIncr=0), (abba,42,46). The section "Examples" (:139) holds three methods in a <pre><code class="language-java"> block (:141-195).

javac reports 5 errors first (2 for getHyphenationTree(java.io.Reader), 3 for WhitespaceTokenizer(java.io.StringReader)); with those two calls fixed it reports String[] cannot be converted to CharArraySet twice.

A tokenizer reads from a placeholder reader until reset() swaps in the reader given to setReader (Tokenizer.java:100-104). Reading the placeholder throws IllegalStateException("TokenStream contract violation: reset()/close() call missing, ...") (Tokenizer.java:111-118).

Each part is a CompoundToken, which copies the current token's offsets (CompoundWordTokenFilterBase.java:163-165):

      // offsets of the original word
      this.startOffset = CompoundWordTokenFilterBase.this.offsetAtt.startOffset();
      this.endOffset = CompoundWordTokenFilterBase.this.offsetAtt.endOffset();

So "Rindfleischüberwachungsgesetz" (offsets 0-29) yields (Rind,0,29), (fleisch,0,29), ... and "Drahtschere" (30-41) yields (Draht,30,41), (schere,30,41). Lucene's own TestCompoundWordTokenFilter.testDumbCompoundWordsSE (:147), the page's example 3 with the same name, dictionary and input, asserts start offsets 0, 0, 0, 8, 8, 8, ... and end offsets 7, 7, 7, 16, 16, 16, ... (:207-215).

The term list itself depends on the grammar: the page's de_DR.xml cannot be bundled (package-info.java:88-98), and the hunt checked the terms with a hand-written 4-pattern grammar only. The offsets do not depend on the grammar.

• Who. Developers who start with the decompounding filters from the package overview, the page the javadoc shows first for org.apache.lucene.analysis.compound. It is published on the Lucene website too: an independent verifier fetched the 10.3.1 page, which still shows getHyphenationTree(reader), new WhitespaceTokenizer(new StringReader and (Rind,0,4,posIncr=0), and the fact check found the same on the published 10.3.1 and 10.6.0 pages on 2026-11-01.
• Opt-in. The filters are in lucene-analysis-common and no built-in analyzer uses them (outside the compound package only the factory service registration, the TestRandomChains test and the release notes CHANGES.txt and MIGRATE.md name them). Users add them in code or through DictionaryCompoundWordTokenFilterFactory and HyphenationCompoundWordTokenFilterFactory.
• What they get. First compile errors that name the missing members, then an IllegalStateException once ported: loud, nothing wrong runs silently. The quiet part is the table: it shows per-part offsets that the filters have not produced since 4.4.0 (for a match version of 4.4 or later; at all since 5.0.0), so someone building highlighting or other offset-based features on decompounded tokens is misled about what the filter emits. How many readers rely on the table was not measured.
• Scale. Every release from 4.0.0 to 10.6.0 and main; no configuration, data size or special input is involved.

Two small test methods because the page has two independent stale parts. testExamplesCompile holds the page's "Examples" block verbatim (HTML entities decoded; only imports and a class wrapper added; a line-by-line check against the page confirms it) and compiles it in-process with javax.tools against the test JVM's class path and module path. testTableOffsets runs DictionaryCompoundWordTokenFilter on the page's word "Drahtschere" alone, so the table's (Draht,30,35) and (schere,35,41) become 0-5 and 5-11.

javac's line numbers refer to the text block: 14 and 33 are the getHyphenationTree(reader) calls (page lines 149 and 168), 17, 36 and 51 the new WhitespaceTokenizer( lines (152, 171, 186).

It is the hunt's minimal repro with one change: javac gets java.class.path plus jdk.module.path. Under Gradle the test JVM puts Lucene on the module path, so the class-path-only version fails its compile test there for an unrelated reason, even with the fix applied ("27 tests completed, 1 failed"); with the module path added it passes.

Regression test: none suggested. The page is documentation, TestCompoundWordTokenFilter already asserts the whole-word offsets, and this test embeds a copy of the page text, so it would need editing with every page change. If a committer wants one anyway, it is this same test with the corrected block and table.

# main 645d4fe14a20, seed DEADBEEF, assertions on (off: identical); javac's "is not applicable" lines shortened
TestCompoundPackageDoc
├─ testTableOffsets ✘ endOffset 1 term=Draht expected:<5> but was:<11>
└─ testExamplesCompile ✘ line 14: no suitable method found for getHyphenationTree(java.io.Reader)
       method ...HyphenationCompoundWordTokenFilter.getHyphenationTree(java.lang.String) is not applicable
       method ...HyphenationCompoundWordTokenFilter.getHyphenationTree(org.xml.sax.InputSource) is not applicable
   line 17: no suitable constructor found for WhitespaceTokenizer(java.io.StringReader)
       constructor ...WhitespaceTokenizer(org.apache.lucene.util.AttributeFactory) is not applicable
       constructor ...WhitespaceTokenizer(int) is not applicable
   line 33: no suitable method found for getHyphenationTree(java.io.Reader)
   line 36: no suitable constructor found for WhitespaceTokenizer(java.io.StringReader)
   line 51: no suitable constructor found for WhitespaceTokenizer(java.io.StringReader)
Tests run: 2, Failures: 2

# the same 2 failures with the released 10.6.0, 10.5.2 and 9.12.3 jars, and through Gradle: 2 tests, 2 failures with assertions on and 2 tests, 2 failures off

# the ported example 1 (current API), without and with reset(), on a 4-pattern test grammar
IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). ...
[(Rindfleischüberwachungsgesetz,0,29), (Rind,0,29,posIncr=0), (fleisch,0,29,posIncr=0), (überwachung,0,29,posIncr=0), (gesetz,0,29,posIncr=0), (Drahtschere,30,41), (Draht,30,41,posIncr=0), (schere,30,41,posIncr=0), (abba,42,46)]

• The page presents them as working examples. The section is headed "Examples" (package-info.java:139) with no pseudo-code marker, and the table is introduced with "This example shows what it does" (:20). In January 2026 the block was re-tagged as highlighted Java, <pre><code class="language-java"> (#15573, 378ca8d1bbf, Dawid Weiss).
• Not missing boilerplate. The errors name removed members: getHyphenationTree(Reader), WhitespaceTokenizer(Reader) and the String[] dictionary constructors. The loop breaks the documented consumer workflow ("The consumer calls reset()", TokenStream.java:54).
• The changes that broke it never touched the page (git show --stat): LUCENE-2781 (a58c26978f5), LUCENE-3530 (a91efbedd1b), LUCENE-5388 (ab0e85fafed on trunk; on branch_5x inside 55594bad9b0, the LUCENE-5944 backport commit), LUCENE-5235 (cdb6ea3c022) and LUCENE-5018 (3e893624367 / 2e92d089832, which changed only CHANGES.txt, CompoundWordTokenFilterBase and its test). Main's CHANGES.txt lists LUCENE-5018 under 4.4.0 ("... don't update offsets anymore"), LUCENE-5235 under 4.6.0 and LUCENE-5388 under 5.0.0. A 2012 commit titled "code snippet improvements: fix out of date api usage" (647b5091697) changed only this block's markup.
• Lucene's tests use the current API for the same examples. TestCompoundWordTokenFilter.testDumbCompoundWordsSE is the page's example 3 with a CharArraySet dictionary, asserting whole-word offsets; the hyphenation tests load their grammar with new InputSource(getClass().getResource("da_UTF8.xml").toExternalForm()).
• Maintainers fix such pages. LUCENE-4788 ("Out of date code examples", #5853) was fixed in 4.2; #16904 (stale examples on the collation page) was approved by a committer on 2026-10-29 and is still open.

Strongest case for "not a bug": every behaviour involved is deliberate (the removals, the reset() check, the whole-word offsets; per-part offsets were later proposed on #9496, LUCENE-8450, and opposed); the block is informal (test-method names, no imports, a grammar file that is not bundled); the TokenStream contract makes reset() the caller's job; and maintainers know examples rot. Robert Muir wrote on #15225 (2025-10-25): "Code examples need to be kept extremely minimal in the rare cases where it does make sense: the maintenance cost is very high and we don't have great validation for these" (their view).

Why it fails: the deliberate changes are the root cause and place the defect in the page, which is exactly the claim. Nothing marks the code as pseudo-code, the table is presented as what the filter does, and the examples are the page's only usage guidance, so a reader who copies them breaks no rule. The #15225 view argues for fewer, smaller examples: it changes the remedy (port, shrink or delete), not the verdict, and records no decision to keep stale ones. Every verifier and the adjudication reached the same conclusion.

One fix (A) was validated in a scratch worktree of 645d4fe14a20. The others were not run. No option changes library code; a CHANGES entry is the committer's call.

Javadoc only, package-info.java, +36/-28. In the table every part gets the whole word's offsets ((Rind,0,29), (fleisch,0,29), (überwachung,0,29), (gesetz,0,29), (Draht,30,41), (schere,30,41)), and the sentence under it says so: "The input token is always preserved, each word part carries the start and end offsets of the whole input token, and the filters do not alter the case of word parts." In all three examples: String[] becomes new CharArraySet(Arrays.asList(...), true), the grammar is loaded with getHyphenationTree("de_DR.xml"), the tokenizer is new WhitespaceTokenizer() plus setReader(new StringReader(...)), and the loop gets tf.reset() before and tf.end(); tf.close(); after it.

- *     String[] dict = { "Rind", "Fleisch", "Draht", "Schere", "Gesetz",
- *         "Aufgabe", "Überwachung" };
- *     Reader reader = new FileReader("de_DR.xml");
+ *     CharArraySet dict = new CharArraySet(Arrays.asList("Rind", "Fleisch", "Draht", "Schere",
+ *         "Gesetz", "Aufgabe", "Überwachung"), true);
- *         .getHyphenationTree(reader);
+ *         .getHyphenationTree("de_DR.xml");
+ *     Tokenizer tokenizer = new WhitespaceTokenizer();
+ *     tokenizer.setReader(new StringReader("Rindfleischüberwachungsgesetz Drahtschere abba"));
- *         new WhitespaceTokenizer(new StringReader(
- *             "Rindfleischüberwachungsgesetz Drahtschere abba")), hyphenator,
+ *         tokenizer, hyphenator,
+ *     tf.reset();          (before the loop; tf.end(); tf.close(); after it)

Trade-offs

• Case-insensitive on purpose: the old String[] constructors built a case-insensitive set; with false, example 1 finds only "Rind" and "Draht". Lucene's own compound tests build their dictionaries the same way.
• Grammar by file name, getHyphenationTree(String) ("the filename of the XML grammar to load"). Wrapping new FileReader(...) in an InputSource, as a first draft did, decodes with the platform charset: the pattern LUCENE-2732 deprecated. A relative name resolves against the working directory.
• getHyphenationTree(String) hands the name to the XML parser as a system id (new InputSource(hyphenationFilename)); only a relative name in the working directory was run, and with the 4-pattern test grammar, not the OFFO de_DR.xml.
• The diff is against main; the 10.6.0 page still has <pre class="prettyprint"> (it lacks #15573's markup change), so a backport needs a rebased diff.
• The ported block is still unchecked text and can drift again.

What the checks validated

• :lucene:analysis:common:checkGoogleJavaFormat, ecjLintMain and renderJavadoc pass; the rendered page shows the new calls and offsets and none of the stale ones.
• Gradle, all compound tests (org.apache.lucene.analysis.compound.*: 5 classes, 27 tests, including the fix-check copy of the repro) with seed DEADBEEF: 0 failures with assertions on, 0 with assertions off.
• The repro with the patched block and table substituted (generated from the patched file, checked line by line; it adds the imports the patched block needs, java.util.Arrays, CharArraySet and Tokenizer) passes with assertions on and off standalone: 2 tests, 0 failures.
• The patched example 1 with an offset print added, run on a 4-pattern test grammar named de_DR.xml, prints (Rindfleischüberwachungsgesetz,0,29) (Rind,0,29) (fleisch,0,29) (überwachung,0,29) (gesetz,0,29) (Drahtschere,30,41) (Draht,30,41) (schere,30,41) (abba,42,46): the corrected table's terms and offsets.

Replace the three examples with a short paragraph (the filters wrap a TokenStream, take a CharArraySet dictionary and, for the hyphenation variant, a grammar from getHyphenationTree), point to DictionaryCompoundWordTokenFilterFactory and HyphenationCompoundWordTokenFilterFactory for configuration, and keep the corrected table.

Trade-offs

Matches the view on #15225 that code examples should be minimal; readers lose a ready-to-run snippet.

Move the examples into a compiled test source and include them with the javadoc {@snippet} tag, as LUCENE-8488 (#9534, open) asks for in general.

Trade-offs

Lucene uses no {@snippet} today, so it needs build work (a snippet path) beyond this page; better as a separate, general proposal.

Correct the offsets and add the sentence about whole-word offsets; leave the examples.

Trade-offs

Smallest change for the part that can mislead quietly; leaves the compile errors.

• Signal (2026-10-28). A capped compile sweep of the non-sandbox javadoc Java blocks against main reported this page's getHyphenationTree(Reader) and WhitespaceTokenizer(StringReader) errors and listed it as a lead. A follow-up probe then compiled the block verbatim against the stock 645d4fe14a20 jars.
• Hunt (2026-10-29). The hunt wrote the one-test repro, ran it on main (-ea, -da), 10.5.2, 9.12.3 and the 10.6.0 RC1 jars, scanned every called signature in the cached release jars from 2.4.0 on, found the 2 hidden String[] errors with a staged compile, and ran the ported example (the reset() throw and the whole-word offsets) on a hand-written 4-pattern grammar.
• Checks. A cross-model refute review (OpenAI GPT-5.6, executing its own 4-test JUnit), independent reproducers, an intended-behaviour devil's advocate and an adjudication; then a minimal two-method repro.
• Tier. Non-formal: it compiles documentation code and scans jars for API members; no model checking or proof was involved.