SoraniAnalyzer, LatvianAnalyzer and GalicianAnalyzer load the `#` comment lines of their stopwords.txt as stop words

SoraniAnalyzer, LatvianAnalyzer and GalicianAnalyzer build their public default stop set from a bundled stopwords.txt with WordlistLoader.getWordSet(InputStream), the overload that "adds every line as an entry" (WordlistLoader.java:80-90), and pass no comment marker (SoraniAnalyzer.java:65-68, LatvianAnalyzer.java:64-67, GalicianAnalyzer.java:63-66). Their files mark comments with #, so the header, bibliography, category and English-gloss lines become entries: 60 of the 122 entries of SoraniAnalyzer.getDefaultStopSet(), 9 of 170 for Latvian (including a bare #) and 1 of 161 for Galician. 24 of the 28 analyzers that call getWordSet pass "#", and Lucene's own StopFilterFactory loads the same three files to 62, 161 and 160 entries, so one bundled file gives two different sets inside Lucene. The stock analyzers' tokens do not change (StandardTokenizer never emits # or a token with a space); the visible effects are the sets' size(), iteration and contains(), and dropped tokens when a set is reused with another tokenizer (with WhitespaceTokenizer the Latvian set removes a standalone #). Each analyzer has had it since it was added (Galician 3.1.0, Latvian 3.2.0, Sorani 4.7.0), through 10.5.2 and main. The fix is to pass "#" in the three calls.

• Stop words: frequent words ("and", "of") that an analyzer removes from the token stream. Each language analyzer bundles a list as a resource stopwords.txt, loads it once into a CharArraySet and returns it from the public static getDefaultStopSet(); its StopFilter removes every token that is in the set.
• The file format: one word per line; a line that starts with # is a comment. StopFilterFactory's default "wordset" format documents exactly this ("allows whole line comments beginning with the \"#\" character", StopFilterFactory.java:52-55).
• WordlistLoader (@lucene.internal) has two families of getWordSet overloads: without a comment marker, it "adds every line as an entry" (:80-90, and "Every line ... should contain only one word"); with one, it "adds every non-comment line" (:148-160), dropping lines that start with the marker.

1. SoraniAnalyzer.java:65-68, LatvianAnalyzer.java:64-67 and GalicianAnalyzer.java:63-66 call WordlistLoader.getWordSet(IOUtils.requireResourceNonNull(getResourceAsStream("stopwords.txt"), ...)): no marker.
2. BulgarianAnalyzer.java:71-75 and 23 other analyzers make the same call with "#" as a second argument; SmartChinese passes "//". These three are the only analyzers that pass nothing.
3. The three files have # lines: Sorani 69 (a 6-line header, then English glosses and category headings: # and, # which, # demonstratives, ...), Latvian 9 (a header with a bare # on line 4, and the headings # prepositions, # Conjunctions, # Particles, # modal verbs, # verbs), Galician 1 (# galican stopwords).
4. The loader trims each line and adds it, so every comment line becomes an entry. Sorani's 69 lines give 60 distinct entries because some glosses repeat.
5. The stock chains tokenize with StandardTokenizer before the StopFilter (SoraniAnalyzer.java:114-118, LatvianAnalyzer.java:112-114, GalicianAnalyzer.java:111-113). No token contains a space or is #, so no comment entry ever matches there.

Nothing is only in the factory set: the default sets are the real words plus the comments.

• Who sees it. Every caller of the public static getDefaultStopSet() of SoraniAnalyzer, LatvianAnalyzer or GalicianAnalyzer; no setting is needed. Code that lists, exports, counts, merges or displays default stop words gets the comments. The same entries come back from getStopwordSet() (StopwordAnalyzerBase.java:41-43) of an analyzer built with the default constructor, which copies the default set.
• One file, two sets. Loading the analyzer's own stopwords.txt through StopFilterFactory (for example in a CustomAnalyzer) gives 62, 161 and 160 entries; getDefaultStopSet() gives 122, 170 and 161.
• Dropped tokens only on reuse. With a tokenizer that can emit #, the Latvian set removes standalone # tokens: WhitespaceTokenizer + StopFilter on lucene # search gives [lucene, search]. With KeywordTokenizer a whole field equal to a comment line is dropped: KeywordTokenizer + StopFilter(SoraniAnalyzer.getDefaultStopSet()) on # and gives [] on main and [# and] with the fix (one executed probe, assertions on and off).
• Not affected. The stock analyzers: with the default set and with the comment-free set they give the same tokens over 139 Sorani, 175 Latvian and 164 Galician inputs (every line of each file plus #, # and and lucene # search); the stock LatvianAnalyzer gives [lucen, search] for lucene # search either way. Analyzers built with your own stop set. The 24 sibling analyzers.
• Scale. Low: public sets that contain comment lines, deterministic, in every release that has these analyzers. No report was found in the sources searched on 2026-10-27, 12 to 15 years after the analyzers were added.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<Sorani [0, Latvian 0, Galician 0]> but was:<Sorani [60, Latvian 9, Galician 1]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.ckb.TestDefaultStopSetCommentLines.testNoCommentLinesInDefaultStopSets(TestDefaultStopSetCommentLines.java:28)

• The analyzers' contract. SoraniAnalyzer.java:43, LatvianAnalyzer.java:42, GalicianAnalyzer.java:41: DEFAULT_STOPWORD_FILE is the "File containing default Kurdish/Latvian/Galician stopwords"; :47, :46, :45: getDefaultStopSet() "Returns an unmodifiable instance of the default stop words set".
• The loader's precondition. The overload they call requires "Every line of the Reader should contain only one word" (WordlistLoader.java:80-82); the comment lines are prose.
• The file format. StopFilterFactory's default "wordset" format "allows whole line comments beginning with the \"#\" character" (StopFilterFactory.java:52-55); TestWordlistLoader.testComments (:39-45) asserts that the comment-aware overload drops #comment.
• History. The comment-aware overload existed when each analyzer was added, and same-era siblings used it (in 3.1.0 Bulgarian, Armenian, Basque and Catalan passed "#"). The same file was also read as commented on the factory path: the Sorani commit shipped a byte-identical copy as the Solr example's lang/stopwords_ckb.txt, loaded by StopFilterFactory, where the # lines are comments; byte-identical Galician and Latvian copies (stopwords_gl.txt, stopwords_lv.txt) came later from the same author in SOLR-3097/SOLR-3105 (2012, first released in 3.6.0), also loaded by StopFilterFactory. The adding issues (LUCENE-2842, LUCENE-3016, LUCENE-5379), CHANGES and MIGRATE say nothing about comment entries; later refactors of the calls were mechanical, and their commit messages do not mention comments.
• No test asserts a # entry. TestSoraniAnalyzer, TestLatvianAnalyzer and TestGalicianAnalyzer check analysis output only, and a git grep over every test tree finds no test that asserts a # entry in a default set.
• Precedent. In #12291 a committer wrote "I think the stoplist loader already ignores comment lines, but, does not ignore empty lines"; the fix PR #12299 is listed as GITHUB#12291 under Bug Fixes in 9.7.0 (lucene/CHANGES.txt:3553). It cuts both ways: that fix changed WordlistLoader for every caller, which a blank entry allows, while a #-leading line can be real data for the no-comment overload.

The strongest case for "intended" (rated moderate by the independent reproducer and the cross-model reviews; the others gave no strength rating): the loader does exactly what its javadoc says; getDefaultStopSet() does not promise that every entry is a word; the calls are 12 to 15 years old and survived several refactors; the entries are inert in the stock chains; changing a public default set has a compatibility cost (the Thai stop-list change ran through a migration note, #16718/#16770).

Why it fails: the question is which overload the analyzers pick, not whether the loader works. Every sibling with the same file format picks the comment-aware one, which existed each time; the files are documented as stop-word files and Lucene's own factory path reads their # lines as comments; mechanical refactors are not acceptance; and "no effect on stock output" is a severity argument, not an intent argument, as #12299 shows. The rebuttal does not make the set's format an explicit contract, and changing a public set has a real compatibility cost. Verdict: not intended, low impact (the executed checks establish the behaviour, not the intent). A maintainer may still file the fix under Improvements.

Option A was validated through gradle; B and C are alternatives a maintainer might raise, D is a complement.

The same three-line change in SoraniAnalyzer.java, LatvianAnalyzer.java and GalicianAnalyzer.java; the call then reads exactly like BulgarianAnalyzer.java:71-75. A regression test: the repro above, or the comment-entry check added to each analyzer's own test class.

             WordlistLoader.getWordSet(
                 IOUtils.requireResourceNonNull(
                     SoraniAnalyzer.class.getResourceAsStream(DEFAULT_STOPWORD_FILE),
-                    DEFAULT_STOPWORD_FILE));
+                    DEFAULT_STOPWORD_FILE),
+                "#");

Trade-offs

• Changes three public default sets: they lose 60, 9 and 1 entries and then equal what StopFilterFactory loads; no real word is lost.
• The stock analyzers' tokens stay the same, so indexes built with them need no reindex (executed on 478 inputs; beyond them by reasoning: every comment entry contains a space or is a bare #, which StandardTokenizer never emits).
• Custom chains that reuse a default set with a tokenizer that can emit # (or, with KeywordTokenizer, a whole comment line such as # and) keep those tokens after the fix. The CHANGES line should say so; a maintainer may file it under Bug Fixes (as #12299) or Improvements.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25.0.3, seed DEADBEEF: :lucene:analysis:common:test with both repro classes added passes with assertions on: 341 classes, 1,924 tests, 0 failures, 18 skipped. That is the module's 1,909 upstream tests (among them TestSoraniAnalyzer 7, TestLatvianAnalyzer 4, TestGalicianAnalyzer 4, TestBulgarianAnalyzer 7) plus the 15 repro tests, which fail 10 of 15 on stock main.
• The same with assertions off: 341 classes, 1,924 tests, 0 failures, 18 skipped.
• Static checks: :lucene:analysis:common:check -x test with the fix (repro classes removed) passes: checkGoogleJavaFormat, ecjLint, forbiddenApis, checkLicenses (error-prone is skipped in this build).
• With the standalone runner (fix as shadow classes): the 4-test class 4 of 4 and the independent reproducer's 11 of 11 pass; the cross-model reviewers' own suites and the 15 upstream Sorani, Latvian and Galician tests pass.

Gives the same sets as A without touching code.

Trade-offs

• Loses the provenance: the Latvian source paper, the Sorani references and the English gloss above every Sorani word.
• Breaks the convention of every other bundled #-commented file and leaves the overload trap for the next edit.
• Not run.

Fixes all callers at once inside WordlistLoader.

Trade-offs

• Changes the documented "adds every line as an entry" behaviour for every caller, including StopwordAnalyzerBase.loadStopwordSet(Path|Reader) (StopwordAnalyzerBase.java:70-89), which loads user stop files: a real entry that starts with # would vanish.
• A maintainer may note that #12299 did change the loader for every caller; a blank entry has no plausible meaning, while a #-leading entry can be real data for this overload.
• Not run.

Assert for every analyzer with a bundled #-commented stopwords.txt that getDefaultStopSet() has no entry starting with #, or equals the StopFilterFactory load of the same file. No such Lucene unit test was written. The hunt ran related checks: its harness ran every entry of 47 analyzers' default sets (9,538 entries) through the analyzer's own chain and found comment survivors only in these three (8 for Latvian, whose bare # does not survive StandardTokenizer), and a loader and format audit found them to be the only #-commented files loaded without a marker.

Trade-offs

• Catches the next analyzer that picks the wrong overload; adds one test class.
• Not run as a Lucene test.

Method: bounded-exhaustive check of bundled data (the study's "semi-formal" tier), hunt unit "bundled word lists", stock main a9779ef3c193, 2026-10-27.

• Every entry of every default stop set reachable through a public static accessor of 47 analyzers (9,538 entries) was analyzed alone through its own default analyzer, plus title- and upper-case variants (25,467 checks). Each entry that the analyzer did not remove was classified by cause.
• Class "comment": Sorani 60, Latvian 8, Galician 1 (Latvian's ninth comment entry, the bare #, is removed by StandardTokenizer before the check). No other analyzer had any.
• A loader and format audit in the same unit found the ckb, lv and gl files to be the only #-commented bundled files loaded without a comment marker.
• An AI finder then wrote the public-API test and traced the cause; independent AI tools reproduced and judged it (see the checks).