With format="snowball", a UTF-8 byte order mark stays glued to the first word of a stop, keep or common-words file

StopFilterFactory, KeepWordFilterFactory, CommonGramsFilterFactory, CommonGramsQueryFilterFactory and SuggestStopFilterFactory read a words file in one of two formats: wordset, the default (one word per line, # comments), or the opt-in snowball (several words per line, | comments). Some editors save UTF-8 with a byte order mark, the bytes EF BB BF, which decode to U+FEFF. The wordset loader drops a leading U+FEFF, on purpose since 2010 (SOLR-1865). The snowball path does not: the factory decodes the file with its own reader and the parser splits on \s+, which does not match U+FEFF, so the mark stays glued to the first word. A file <BOM>the a an | articles gives the stop set [a, \uFEFFthe, an], and the cat is analyzed to [the, cat] instead of [cat]. The first keep word, common word and suggest stop word are lost the same way; when the first line is a comment or blank, or the mark is followed by a space or tab, the words survive and the set gains a stray \uFEFF entry. No error is raised. The snowball path has skipped no mark since it was added (SOLR-1860, 2012), so every Lucene release from 4.0.0-BETA to 10.5.2 and main is affected. Skipping one leading U+FEFF where the factory decodes the file fixes it.

• Words file: the words parameter of StopFilterFactory, KeepWordFilterFactory, CommonGramsFilterFactory, CommonGramsQueryFilterFactory and SuggestStopFilterFactory; format says "how the words file will be parsed" (AbstractWordsFileFilterFactory.java:38-54).
• Byte order mark (BOM): the bytes EF BB BF that some editors write at the start of a UTF-8 file. Java's UTF-8 decoder keeps them as the character U+FEFF, which is invisible when printed.

1. wordset goes through AbstractAnalysisFactory.getWordSet and WordlistLoader.getLines, which decodes the bytes itself and drops a leading U+FEFF from each line it reads while no word has been added yet (lines.isEmpty()), so a BOM at the start of the file is skipped: // skip initial bom marker (WordlistLoader.java:297-314, :302-304).
2. snowball goes through AbstractAnalysisFactory.getSnowballWordSet, documented as the "same as getWordSet(ResourceLoader, String, boolean), except the input is in snowball format" (AbstractAnalysisFactory.java:304-307). It builds its own strict UTF-8 InputStreamReader (:317-322) and hands it to WordlistLoader.getSnowballWordSet(Reader, CharArraySet) (:323).
3. That parser cuts each line at | and splits it on \s+ (WordlistLoader.java:193-207). \s matches ASCII whitespace only, so the first piece of the first line is \uFEFFthe; when the first line is a comment or blank, or the mark is followed by a space or tab, the piece is \uFEFF alone and becomes an entry of its own while the word survives (<BOM> the a an gives [the, a, an, \uFEFF]).
4. The factory loops over a comma-separated words list (AbstractAnalysisFactory.java:316-325), so every file's own mark is kept.

Without the mark, snowball gives the wordset column in the first five rows. The internal WordlistLoader.getSnowballWordSet overloads that the built-in language analyzers use keep a mark too, but none of the 147 bundled resource files starts with one.

• Who is exposed. Applications that configure one of the five factories with an explicit words file and format="snowball": directly, through CustomAnalyzer, or through a search server's schema (the factories' javadoc examples are schema snippets). The default format is wordset, so it is opt-in.
• What it takes. A words file saved as UTF-8 with a byte order mark, which the 2010 CHANGES entry that introduced the skip describes as "often created by editors such as Windows Notepad", and a word on its first line.
• What happens. A stop word is indexed and searched like any other word; a keep word's tokens are dropped; a common word forms no bigrams; a suggest stop word is not treated as one. With a comment or blank first line, or a space or tab after the mark, no word is lost and the set gains a \uFEFF entry. Nothing fails, and a printed set looks right because U+FEFF is invisible.
• Scale. Deterministic and independent of index size. The wrong set is used for every document and query analyzed with that analyzer, so any text that contains the affected word is analyzed differently. Indexes built with the wrong set keep their terms until reindexed.
• Built-in analyzers. Not affected: all 15 bundled Snowball lists start with a | From https://snowballstem.org/... comment line, and none of the 147 bundled resource files starts with EF BB BF. A copy re-saved with a mark loses no word and only gains the stray entry.
• Versions. Every release from 4.0.0-BETA to 10.5.2 and main; SuggestStopFilterFactory from 5.0.0, when it was added, and KeepWordFilterFactory from 8.11.0, when it got the format parameter.
• How likely. Rare: it needs the opt-in format, a file saved with a mark and a word on the first line, and the searches found no prior report (see "Prior reports").

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected [the], actual [\uFEFFthe]
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.core.TestSnowballStopWordsBom.testLeadingBomIsSkipped(TestSnowballStopWordsBom.java:31)

• One parameter, two parse modes of the same file. format "defines how the words file will be parsed", with wordset and snowball as the two values; neither says the file must not carry a mark (AbstractWordsFileFilterFactory.java:38-54, StopFilterFactory.java:44-60). A file with one word per line is valid in both formats, yet the same marked file gives [the, and] in one and [\uFEFFthe, and] in the other.
• Documented as the same loader except for the format. AbstractAnalysisFactory.java:304-307: "same as getWordSet(ResourceLoader, String, boolean), except the input is in snowball format". The mark belongs to the encoding, not to either format, and on this path Lucene's own code does the decoding.
• Deliberate in the default loader. SOLR-1865 (2010) added the skip with the CHANGES entry "SolrResourceLoader.getLines ignores Byte Order Markers (BOMs) at the beginning of input files, these are often created by editors such as Windows Notepad"; LUCENE-4257 moved it into WordlistLoader.getLines. LUCENE-5505 did the same for hunspell dictionaries ("HunspellStemFilter ignores BOM markers in dictionaries"). In 2010, rmuir listed that the Solr stopword loading code "ignores BOM markers" among the reasons it was better than Lucene's own loader. It is not a rule of every Lucene resource loader: the same hunt saw other loaders keep a mark.
• An omission. The snowball factory path came two years after the rule (SOLR-1860, 2012) with its own reader, and its tests use a file without a mark. No test, javadoc, CHANGES entry or issue says snowball files must not carry one; in the tests of core, analysis-common and suggest, the only input that starts with a mark is hunspell's (core's TestStandardAnalyzer lists U+FEFF only as a word-break format character).

The checkers built the strongest case for "intended": (a) read literally, the snowball grammar names only whitespace and | comments, so U+FEFF is part of the first word; (b) WordlistLoader is @lucene.internal, and its generic getWordSet(Reader) keeps a mark too; (c) the wordset skip is a code comment plus a CHANGES entry, not javadoc; (d) committers have said that, used directly, "the stopword file parser does not care about encodings at all" (uschindler, 2010) and that a mark in a Reader means "your conversion from bytes is screwed up somewhere before" (dweiss, 2017, reviewing a char-filter mapping that deleted U+FEFF from analysis input: an analogy, not a remark on resource files); (e) in the 2010 user-list thread behind the skip, rmuir answered "No." to whether a marked file is a bug, and then: "we can ignore them for the first line of the file to be more user-friendly"; SOLR-1865 is typed Improvement, and users can simply save the file without a mark.

Why we think it fails: (a) to (c) describe the internal parser; the user-facing contract is the factory's format parameter, whose two values are documented as parse modes of the same file, and the factory itself turns the bytes into chars, so by the reasoning of (d) the factory is where the mark should go (the recommended fix). (e) was about whether to handle marks at all; in the same thread Yonik Seeley called it "a Java bug" and added "we should fix if it's practical to do so, rather than passing the buck", the default words-file loader then skipped them, and snowball is the only words-file format of these factories that does not. The literal reading (a) stays arguable. Expect "improvement, not bug" as an answer. Every checker rejected the works-as-designed case; the cross-model reviews rated it moderate, the devil's advocate weak.

Options B and A were each applied in a scratch worktree of main a9779ef3c193 and run through gradle with assertions on and off; the worktree was removed afterwards. Option C was not run.

In AbstractAnalysisFactory.getSnowballWordSet (AbstractAnalysisFactory.java:322), wrap the decoding reader in a BufferedReader, read one char and reset() unless it is U+FEFF, then parse as before. WordlistLoader.getSnowballWordSet reuses a reader that is already a BufferedReader, so nothing is buffered twice.

What it changes. Only the factory path used by the five factories: one U+FEFF at the very start of each file is dropped, the case getLines covers for wordset (unlike getLines, it does not look at later lines). Files without a mark are read unchanged; an empty file still gives no words; malformed UTF-8 still fails with the same MalformedInputException (checked with a copy of this reader setup on the stock jars). The internal WordlistLoader API and the built-in analyzers are untouched.

-        try (Reader reader = new InputStreamReader(loader.openResource(file.trim()), decoder); ) {
+        try (BufferedReader reader =
+            new BufferedReader(new InputStreamReader(loader.openResource(file.trim()), decoder))) {
+          // skip initial bom marker, as the wordset format does (WordlistLoader.getLines)
+          reader.mark(1);
+          if (reader.read() != '\uFEFF') {
+            reader.reset();
+          }
           WordlistLoader.getSnowballWordSet(reader, words);
         }

Trade-offs

• Fixes the factory path only: a caller of the internal WordlistLoader.getSnowballWordSet(Reader) who passes a Reader that still carries the mark gets the glued word, which matches the committers' view that the Reader-level parser is encoding-agnostic.
• The parser's try-with-resources and the factory's both close the same BufferedReader; the second close() does nothing.
• About 6 lines plus an import change (BufferedReader in, the unused Reader out).
• Indexes built with the wrong set keep their terms until reindexed. No CHANGES entry is drafted.

What the checks validated

• Gradle in a scratch worktree of main a9779ef3c193, assertions on and off: the 4-test repro passes (stock: 3 of 4 fail), and so do the upstream TestStopFilterFactory, TestKeepFilterFactory, TestCommonGramsFilterFactory and TestCommonGramsQueryFilterFactory (3, 2, 4 and 4 tests: 17 with the repro, 0 failures, each way).
• TestWordlistLoader (4 tests) and TestSuggestStopFilterFactory (3 tests) pass.
• All of :lucene:analysis:common:test with assertions on: 340 classes, 1,913 tests (the repro included), 18 skipped, 0 failures.
• :lucene:core:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• Shadowed into the repro harness: the independent 9-test suite, which covers the stop, keep-word, common-grams and suggest-stop factories, the comment-line case and the same-set-in-both-formats check, passes 9 of 9 with assertions on and off (stock: 7 of 9 fail), and the on-disk test through CustomAnalyzer.builder(Path) passes 5 of 5.

In WordlistLoader.getSnowballWordSet(Reader, CharArraySet) (WordlistLoader.java:193-207), drop a leading U+FEFF from the first line, as getLines does for a mark at the start of a file (:302-304). This is the three-line fix (plus a comment) every checker ran.

+      boolean first = true;
       while ((line = br.readLine()) != null) {
+        // skip initial bom marker, as getLines does
+        if (first && line.length() > 0 && line.charAt(0) == '\uFEFF') line = line.substring(1);
+        first = false;

Trade-offs

Also covers the internal getSnowballWordSet(Reader) and getSnowballWordSet(InputStream[, Charset]) overloads, harmless today because no bundled list has a mark. But it changes an @lucene.internal Reader-level parser that committers have called encoding-agnostic, and a Reader whose text really starts with U+FEFF (ZERO WIDTH NO-BREAK SPACE) loses it.

What the checks validated

• Gradle in a scratch worktree of main a9779ef3c193, assertions on and off: the repro and the four upstream factory suites pass (17 tests, 0 failures, each way); TestWordlistLoader (4 tests) and TestSuggestStopFilterFactory (3 tests) pass; all of :lucene:analysis:common:test: 340 classes, 1,913 tests, 18 skipped, 0 failures (this includes the analysis-common analyzers that load a bundled Snowball list through the changed parser; the Ukrainian analyzer in the morfologik module, which also calls it, was not run); :lucene:core:check -x test passes.
• Earlier, shadowed into the repro harness by every checker: the repro 4 of 4, the independent suite 9 of 9; in the hunt's bounded-exhaustive check, a run that shadowed this fix together with a fix for an unrelated finding drops the snowball mismatches with a mark from 43,380 to 0. The fact check of this doc ran the on-disk test with this fix: 5 of 5 pass.

A shared helper that opens a resource as UTF-8 and drops one leading U+FEFF, used wherever a factory decodes a resource file itself.

Trade-offs

Would also fix the same pattern in other loaders (the ICU tokenizer's rule files are a separate confirmed report; the hunt's probe also saw it in the synonym and the Japanese and Korean user-dictionary loaders, not verified independently), so it suits a single "BOM in resource files" issue; it touches several modules and needs its own tests. Not drafted.

• Method (semi-formal): bounded-exhaustive translation validation of the resource-file formats that analysis factories load, on the stock main jars, in the hunt's unit on factory resource-file formats. Every words file of 1 to 3 lines over a per-format line alphabet, with LF, CRLF or CR line ends, with and without a trailing line end, with and without a leading mark, with ignoreCase on and off, loaded through StopFilterFactory from an in-memory ResourceLoader.
• Oracle: independent reference parsers written from the format javadoc, without BufferedReader, trim or split, plus one rule no javadoc states: both drop one leading mark first (dropBom), taken from the default format's behaviour.
• Signal. Snowball with a mark: all 43,380 generated files differ (28,548 with ASCII whitespace only, 14,832 with other whitespace, compared with the ASCII-whitespace reading that the same files without a mark match). Wordset with a mark, and snowball without one: 0 of those 43,380. Two-file lists: 35 of 35 for snowball, 0 of 35 for wordset. In a run that shadowed the parser fix together with a fix for an unrelated finding, every snowball class with a mark drops to 0.
• The finder then traced it to the separate reader of the snowball path and wrote the public-API JUnit; independent checkers followed (see "Independent checks").