A synonyms file saved as UTF-8 with a byte order mark silently loses its first rule

SynonymGraphFilterFactory and the deprecated SynonymFilterFactory decode the synonyms file with their own strict UTF-8 reader and pass it to SolrSynonymParser (the default format) or WordnetSynonymParser. Some editors save UTF-8 with a byte order mark, the bytes EF BB BF, which Java decodes to the invisible character U+FEFF. Nothing on the synonym path drops it: the Solr parser treats a line as a comment only if its first character is #, and trim() and the default whitespace rule tokenizer keep U+FEFF, so it stays glued to the first term of the first line. With couch, sofa on line 1, couch gets no synonym and sofa gains the invisible synonym \uFEFFcouch; a commented-out first line #couch, sofa becomes a live rule; in WordNet format the first word leaves its synset. With the default settings only the first line of each file is affected, and no error is raised. The word-list factories (stop words, keep words, stemmer overrides, char-filter mappings, type files) read through WordlistLoader.getLines, which skips a leading mark (SOLR-1865, 2010), and the synonym factory did too in 3.1 to 3.3; the 3.4.0 rewrite (LUCENE-3233) gave it its own reader without the skip. Every release from 3.4.0 to 10.5.2, 10.6.0 (branch_10_6) and main is affected (10 releases executed, the rest inferred from the unchanged code). Skipping one leading U+FEFF where the factories decode each file fixes it.

• Synonyms file: the synonyms parameter of SynonymGraphFilterFactory and the deprecated SynonymFilterFactory (one file or a comma-separated list), parsed by SolrSynonymParser (format=solr, the default) or WordnetSynonymParser (format=wordnet).
• Byte order mark (BOM): the bytes EF BB BF at the start of a UTF-8 file. Java's UTF-8 decoder keeps them as the character U+FEFF; Character.isWhitespace(0xFEFF) is false and "\uFEFF".trim() has length 1.

1. SynonymGraphFilterFactory.loadSynonyms (SynonymGraphFilterFactory.java:168-194) builds a strict UTF-8 decoder and, for each file, calls parser.parse(new InputStreamReader(loader.openResource(file), decoder)) (:191). The deprecated SynonymFilterFactory.loadSynonyms (SynonymFilterFactory.java:171-199) does the same (:194-196). The decoded first line of the example is \uFEFFcouch, sofa, 12 characters.
2. SolrSynonymParser.addInternal skips a line only if it is empty or its charAt(0) is # (SolrSynonymParser.java:85; javadoc :33: "Blank lines and lines starting with '#' are comments"). It splits the line on => and ,, and each piece goes through unescape(...).trim() and the rule analyzer (:96-118), by default a WhitespaceTokenizer (SynonymGraphFilterFactory.java:146), which does not split on U+FEFF. So the first term is \uFEFFcouch, and a # first line is parsed as a rule; it only has a visible effect when it contains , or =>.
3. WordnetSynonymParser.parse reads the synset id from fixed columns, line.substring(2, 11) (WordnetSynonymParser.java:53-54): with the mark, line 1 gives (10000000 instead of 100000001, so the first word starts a synset of its own.
4. The factories that read word lists and line files through AbstractAnalysisFactory.getLines / getWordSet go through WordlistLoader.getLines, which drops a leading U+FEFF before the # test: // skip initial bom marker (WordlistLoader.java:302-304). Stop words, keep words and common-grams words in the default wordset format, stemmer overrides, char-filter mappings and type lists get the skip. Not every resource loader does: the snowball words format (AbstractWordsFileFilterFactory.java:89-90 -> AbstractAnalysisFactory.getSnowballWordSet, its own reader), ICU tokenizer rule files and the Japanese and Korean user dictionaries keep the mark too (separate reports).

Line 2 and later lines are unaffected; with a comma-separated list, each file that starts with a mark loses its own first line. With the non-default tokenizerFactory=org.apache.lucene.analysis.standard.StandardTokenizerFactory, which drops a leading U+FEFF, the first rule survives, but a commented-out first rule still becomes active, and a mark followed by a blank first line fails to load (term: \uFEFF was completely eliminated by analyzer) while the same file without the mark loads.

• Who is exposed. Applications that configure SynonymGraphFilterFactory or the deprecated SynonymFilterFactory with a synonyms file: directly, through CustomAnalyzer, or through a search server's schema (the factory javadoc example is a schema snippet). The factories' defaults are affected: format=solr, the whitespace rule tokenizer, expand=true. The component is opt-in: no stock Lucene analyzer uses it; SynonymGraphFilterFactory and both parsers are @lucene.experimental, and SynonymFilterFactory is deprecated.
• What it takes. A synonyms file saved as UTF-8 with a byte order mark, which the 2010 CHANGES entry for the shared loader's skip describes as "often created by editors such as Windows Notepad", whose first line is a rule, or a # line with , or =>. A re-saved copy of a file that starts with the Apache license header qualifies: that header line has a comma.
• What happens. The first rule silently stops matching its first term (missed expansions: the couch query finds 0 of 1 documents), and its other terms gain an invisible synonym that matches nothing; a first explicit mapping never fires; a commented-out first rule becomes active and adds expansions the user disabled; in WordNet format the first word loses its synonyms. With the default settings, no error or warning (with the non-default StandardTokenizerFactory as the rule tokenizer, a mark before a blank first line makes loading fail).
• Scale. Deterministic and independent of index size: one line per file. If the filter runs at index time, terms indexed with the wrong rules stay until reindexing.
• Versions. Every release from 3.4.0 to 10.5.2, 10.6.0 (branch_10_6) and main (10 releases executed, the rest inferred from the code).
• How likely. Rare: a file saved with a mark is needed, only its first line goes wrong, and no public report was found in 15 years. The WordNet half is the weaker one (WordNet prolog files are machine-generated), so the Solr-format symptoms lead.
• Workarounds. Save the file without a mark, or start it with a blank line or a comment line without , or =>.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<[bar]> but was:<[foo]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.synonym.TestSynonymGraphFilterFactoryUtf8Bom.testBomBeforeFirstRule(TestSynonymGraphFilterFactoryUtf8Bom.java:36)

• The file's own comment rule. "Blank lines and lines starting with '#' are comments" (SolrSynonymParser.java:33). A user who writes #couch, sofa on line 1 sees a line that starts with #; with a mark, Lucene makes it a rule.
• Lucene's shared line loader drops the mark on purpose. SOLR-1865 (2010) added the skip with the CHANGES entry "SolrResourceLoader.getLines ignores Byte Order Markers (BOMs) at the beginning of input files, these are often created by editors such as Windows Notepad"; LUCENE-4257 moved it into WordlistLoader.getLines (WordlistLoader.java:302-304). Every factory that reads its file through getLines or getWordSet gets it: the same resource loader and the same bytes give a different result for synonyms.
• The synonym factory had it and lost it silently. In 3.1 to 3.3, SynonymFilterFactory read its rules with SolrResourceLoader.getLines, skip included. LUCENE-3233 (3.4.0) replaced that with a plain InputStreamReader. Its issue (41 comments), its CHANGES entries ("Improved memory usage, build time, and performance of SynonymFilterFactory"; "Added SynonymFilter ... with parsers for wordnet and solr formats") and the upgrade notes say nothing about BOM or encoding handling. Executed: 3.3.0 clean, 3.4.0 affected, 3.4.0 with luceneMatchVersion 3.3 (the old loader) clean.
• Hunspell's equivalent was a bug fix. "LUCENE-5505: HunspellStemFilter ignores BOM markers in dictionaries" is listed under "Bug fixes" (lucene/CHANGES.txt:12543, :12554); Dictionary.java:351-352: "ignore any BOM marker on first line".
• Valid input, no documented exclusion. A file with a mark is valid UTF-8, and the factories' strict decoder accepts it. No javadoc, test, CHANGES or MIGRATE entry says synonyms files must be BOM-free, and no synonym test uses a mark.

The checkers built the strongest case for "intended": (a) SynonymMap.Parser.parse takes a Reader, i.e. characters, and the Solr grammar exempts only lines whose first character is #; U+FEFF is a real character (ZERO WIDTH NO-BREAK SPACE). (b) SynonymGraphFilterFactory and the parsers are @lucene.experimental. (c) Committers have said Reader-level parsers do not deal with encodings: "If you use Lucene directly, the stopword file parser does not care about encodings at all, it simply takes a java.io.Reader" (uschindler, SOLR-1860, 2010), and "if you have a byte order mark in your character input (reader) then your conversion from bytes is screwed up somewhere before" (dweiss, LUCENE-7785, 2017, about a char-filter mapping: an analogy). (d) In the 2010 user-list thread behind the skip, rmuir answered "No." to whether a file with a mark is a bug, then proposed skipping it "to be more user-friendly"; SOLR-1865 is typed Improvement. (e) Nobody reported it in 15 years.

Why we think it fails: (a) to (c) describe the Reader-level parser; users configure the factory with a file name, and the factory itself turns the bytes into characters, so by the reasoning of (c) the factory is where the mark should go (the recommended fix). (d) was about whether to handle marks at all; the project then did, in the very loader the synonym factory used until 3.4.0, and the rewrite dropped it without a word. (e) shows rarity, not intent. The cross-model reviews rated the works-as-designed case moderate (the intended-lens review: "not ultimately persuasive"; the refute-lens review: "not conclusive"), and the devil's advocate rated it weak to moderate. The literal reading (a) stays arguable: expect "improvement, not bug".

Options F and P were each applied in a scratch worktree of main a9779ef3c193 and run through gradle with assertions on and off; the worktree was removed afterwards. Option C was not run. Pick one of F and P: together they would strip two leading U+FEFF characters.

In both loadSynonyms methods (SynonymGraphFilterFactory.java:191, SynonymFilterFactory.java:194-196), wrap the decoding reader in a PushbackReader of size 1, read one character and push it back unless it is U+FEFF, then parse as before, inside a try-with-resources. +19/-4 lines over the two files (one import each; the deprecated factory drops its unused java.io.Reader import).

What it changes. Only the factory path: one U+FEFF at the very start of each synonyms file is dropped before the parser sees it, the case getLines covers for the word-list factories. It covers every parser the factory loads, including a custom SynonymMap.Parser named in format. Files without a mark are read unchanged. SynonymGraphFilterFactory now also closes each reader itself, as the deprecated factory already did; the parser's own close() makes the second close a no-op. A decoding error in the first character is now raised by the factory's read() rather than inside parse: the same MalformedInputException, an IOException, either way (read, not run).

     for (String file : files) {
       decoder.reset();
-      parser.parse(new InputStreamReader(loader.openResource(file), decoder));
+      try (PushbackReader reader =
+          new PushbackReader(new InputStreamReader(loader.openResource(file), decoder), 1)) {
+        // skip initial bom marker, as WordlistLoader.getLines does for other resource files
+        int first = reader.read();
+        if (first != -1 && first != '\uFEFF') {
+          reader.unread(first);
+        }
+        parser.parse(reader);
+      }
     }

Trade-offs

• A direct caller of SolrSynonymParser.parse(Reader) or WordnetSynonymParser.parse(Reader) that builds its own Reader from a marked file still gets the glued term. That matches the committers' view that Reader-level parsers are encoding-agnostic, but such callers exist downstream (engines that read synonyms files with their own UTF-8 reader; checked by reading only) and would need their own skip.
• Like getLines, it cannot tell a mark from a file whose first character is meant as a ZERO WIDTH NO-BREAK SPACE (not a realistic synonym).
• Open PR #13054 (synonym map off-heap) rewraps loadSynonyms, so the two may conflict.
• Indexes built with the wrong rules keep their terms until reindexed. No CHANGES entry is drafted.

What the checks validated

• Gradle in a scratch worktree of main a9779ef3c193, seed DEADBEEF, assertions on and off: the 6-test repro passes (stock: 5 of 6 fail), the independent on-disk test passes 6 of 6 (stock: 3 of 6 fail), and all upstream synonym suites plus TestConditionalTokenFilter pass: 14 classes, 144 tests, 0 failures, each way.
• All of :lucene:analysis:common:test with assertions on: 341 classes, 1,921 tests (the two BOM test classes included), 18 skipped, 0 failures.
• :lucene:analysis:common:check -x test passes with the repro added (google-java-format, ecjLint, forbiddenApis).

In SolrSynonymParser.addInternal (before the comment test, SolrSynonymParser.java:85) and WordnetSynonymParser.parse (before substring(2, 11), WordnetSynonymParser.java:54), drop a leading U+FEFF from the first line of each parse() call, the idiom of WordlistLoader.getLines. +6 lines per parser. This is the hunt's fix; the finder, a checker and the registration check ran it in the repro harness.

+    boolean first = true;
     while ((line = in.readLine()) != null) {
+      if (first) {
+        first = false;
+        // skip initial bom marker, as WordlistLoader.getLines does
+        if (line.length() > 0 && line.charAt(0) == '\uFEFF') line = line.substring(1);
+      }
       if (line.length() == 0 || line.charAt(0) == '#') {

Trade-offs

Also covers downstream callers that build their own Reader and call the parsers directly. But it changes @lucene.experimental Reader-level parsers that committers have called encoding-agnostic, a Reader whose text really starts with U+FEFF loses it, and a custom SynonymMap.Parser named in format is not covered.

What the checks validated

• The same gradle runs, assertions on and off: the repro, the on-disk test and the upstream synonym suites pass (14 classes, 144 tests, 0 failures, each way); :lucene:analysis:common:check -x test passes with the repro added. The whole module was not re-run for this option.
• Earlier, shadowed into the repro harness by the finder and the checkers: the repro 6 of 6, 94 upstream synonym tests, an independent on-disk test 6 of 6, a root-cause variant 8 of 8; the bounded-exhaustive check drops to 0 mismatches.

A shared helper (for example in AbstractAnalysisFactory, next to getLines) that opens a resource as strict UTF-8 and drops one leading U+FEFF, used wherever a factory decodes a resource itself.

Trade-offs

Would also fix the same pattern in other loaders (snowball-format words files and ICU tokenizer rule files are separate confirmed reports, and the Japanese and Korean user dictionaries a sibling one), so it suits a single "BOM in resource files" issue; it touches several modules and needs its own tests. Not drafted.

• First signal. On 2026-10-27 a probe in the hunt's unit on factory resource-file formats (the unit that found the snowball words-file case) printed <BOM>foo,bar : foo -> [foo] for the synonym factory and recorded it as a lead for another unit; a planning probe re-ran it (8 checks, 6 failing).
• Method (semi-formal): bounded-exhaustive metamorphic check on the stock main jars, in the hunt's unit on synonym and user-dictionary loaders. Law: loading F and loading BOM+F give the same analysis of every probe text (terms, position increments, position lengths) or the same exception class. Inputs: every file of 1 to 3 lines over a 5-line Solr alphabet (equivalence, explicit mapping, comment with a comma, blank, multi-word) and a 4-line WordNet alphabet, with LF and CRLF line ends and expand true and false, through both synonym factories, plus three controls that read through the shared line loader (stop filter, stemmer override, mapping char filter).
• Signal. Solr format: 456 of 620 checks mismatch for each factory (372 with a term glued to U+FEFF, 84 where a first mapping never fires); WordNet: 112 of 336 for each factory; the three controls: 0. With the parser fix shadowed, every family: 0.
• The finder then traced it to the factories' own reader, checked the 3.4.0 history and wrote the public-API JUnit; independent checkers followed (see "Independent checks").