A suggester or spell-checker file saved as UTF-8 with a byte order mark silently loses its first entry

FileDictionary (the text-file source for Lookup.build) and PlainTextDictionary (the word-list source for SpellChecker.indexDictionary) have constructors that take an InputStream (and, for PlainTextDictionary, a Path) and decode it as UTF-8 themselves; their javadoc says "NOTE: content is treated as UTF-8". Some editors and export tools save UTF-8 with a byte order mark, the bytes EF BB BF. Java's UTF-8 decoder returns them as the invisible character U+FEFF, and the iterators copy the first line's text unchanged, so the first entry becomes \uFEFF + word. For a file <BOM>foo<TAB>5 / bar<TAB>3, the WFST, FST and TST suggesters return nothing for lookup("fo"), AnalyzingSuggester returns \uFEFFfoo, and a SpellChecker indexed from the word file <BOM>foo / bar says exist("foo") is false. A file that holds only the mark gives one phantom entry. No error is raised, and only the first line of each file is affected. Lucene's line loader WordlistLoader.getLines and hunspell skip the mark. Every release from 4.0.0-BETA (2012) to 10.6.0, and main, is affected. Skipping one leading U+FEFF in the four decoding constructors fixes it.

• The decoding constructors: FileDictionary(InputStream), FileDictionary(InputStream, String fieldDelimiter), PlainTextDictionary(Path) and PlainTextDictionary(InputStream). Each says "NOTE: content is treated as UTF-8" (FileDictionary.java:77, 104, PlainTextDictionary.java:47, 56). The Reader constructors, where the caller decodes the bytes, are not part of this report.
• Byte order mark (BOM): the bytes EF BB BF at the start of a UTF-8 file, an encoding signature, not text. Java's UTF-8 decoder returns them as U+FEFF (JDK-4508058, closed Won't Fix: application code must skip the mark itself).

1. FileDictionary(InputStream) delegates to FileDictionary(InputStream, String) (FileDictionary.java:79-81), which does in = new BufferedReader(IOUtils.getDecodingReader(dictFile, StandardCharsets.UTF_8)); (:107). PlainTextDictionary(Path) uses Files.newBufferedReader(path, StandardCharsets.UTF_8) (PlainTextDictionary.java:50) and PlainTextDictionary(InputStream) the same getDecodingReader call (:59). IOUtils.getDecodingReader builds a strict decoder and an InputStreamReader (IOUtils.java:192-199); it has no BOM handling.
2. FileDictionary.FileIterator() reads line 1 (FileDictionary.java:128), splits it on the delimiter (:133) and copies fields[0] unchanged (:138, :142, :145). PlainTextDictionary's iterator copies each whole line (PlainTextDictionary.java:84-86). Neither class nor its upstream tests handles U+FEFF or a BOM.
3. So the first entry is \uFEFFfoo. The WFST, FST and TST suggesters index that key, so no prefix of foo reaches it; AnalyzingSuggester with StandardAnalyzer matches it but returns the stored key with the mark; SpellChecker.indexDictionary indexes the word \uFEFFfoo, so exist("foo") is false.
4. Lucene's other text-file loaders skip the mark: WordlistLoader.getLines (// skip initial bom marker, WordlistLoader.java:302-304) and hunspell (Dictionary.java:246, :351-354, "ignore any BOM marker on first line"). Not uniform: WordlistLoader.getWordSet(InputStream) keeps it too (an independent reviewer from another model family, OpenAI GPT-5.6, pointed this out); WordlistLoader is @lucene.internal.

Only line 1 is affected: a U+FEFF at the start of a later line is a real character and is kept. (getLines differs slightly: it strips U+FEFF from the first line it keeps, which can be a later line when comment or blank lines come first.)

• Who is exposed. Applications that build a suggester from FileDictionary(InputStream) or FileDictionary(InputStream, String), or index a SpellChecker from PlainTextDictionary(Path) or PlainTextDictionary(InputStream). These are the public constructors with no special setting; the suggest module itself is opt-in.
• What it takes. A dictionary file saved as UTF-8 with a byte order mark, which the 2010 CHANGES entry for the line loader's skip (SOLR-1865) describes as "often created by editors such as Windows Notepad". No scale, no concurrency, no unusual setting.
• What happens. The first suggestion can never be found by typing its prefix (WFST, FST, TST); AnalyzingSuggester returns it with an invisible character that breaks equality checks downstream; the spell checker reports the first word as misspelled and suggests the marked form; an empty file saved with a mark gives one phantom entry. No error or warning, and printed output looks right.
• Scale. Deterministic and independent of dictionary size: one entry per file.
• Versions. Every release from 4.0.0-BETA (2012) to 10.6.0, and main (14 releases and pre-releases from 4.0.0-BETA to 10.5.2 executed, plus 4ae946e5a9b9, the commit 10.6.0 was released from, and main; the rest read). The (InputStream, String) and Path constructors arrived in 4.7.0 and 5.0.0; before them FileDictionary(InputStream), PlainTextDictionary(InputStream) and PlainTextDictionary(File) showed the same defect.
• How likely. Rare: it needs a marked file, only its first entry goes wrong, and no public report was found since 4.0.0.
• Workarounds. Save the file without a mark, start it with a throwaway first line, or decode the file yourself, strip a leading U+FEFF and pass a Reader.

Copy the file into lucene/suggest/src/test/org/apache/lucene/search/suggest/ and run the command. The test body is deterministic (no random(), no Directory, no analyzer), though it runs under Lucene's randomized test framework; pass -Ptests.asserts explicitly, because by default the build picks it from the seed. With the file added, :lucene:suggest:check -x test passes. PlainTextDictionary(Path/InputStream) has the same defect and the same fix; this repro shows the FileDictionary entry point only. The test is the suggested regression test as it stands.

./gradlew :lucene:suggest:test --tests "org.apache.lucene.search.suggest.TestFileDictionaryUtf8Bom" -Ptests.seed=DEADBEEF -Ptests.asserts=true

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestFileDictionaryUtf8Bom > testFirstEntryAfterByteOrderMark FAILED
    java.lang.AssertionError: expected:<[66 6f 6f]> but was:<[ef bb bf 66 6f 6f]>
        at org.apache.lucene.search.suggest.TestFileDictionaryUtf8Bom.testFirstEntryAfterByteOrderMark(TestFileDictionaryUtf8Bom.java:29)
1 test completed, 1 failed

# with the recommended fix, assertions on and off:
:lucene:suggest:test (SUCCESS): 1 test
BUILD SUCCESSFUL

• The constructors own the decoding. With the InputStream and Path constructors the caller never sees characters: Lucene picks the charset (FileDictionary.java:107, PlainTextDictionary.java:50, 59). Dawid Weiss's view on LUCENE-7785 (2017, about a char-filter mapping) points to exactly that step: "if you have a byte order mark in your character input (reader) then your conversion from bytes is screwed up somewhere before". A caller can only work around it outside these constructors: skip the three bytes on the stream first, or decode the file and use a Reader constructor.
• Lucene's other user-file loaders skip it on purpose. SOLR-1865 (2010, Robert Muir's patch) added the skip with the CHANGES entry "SolrResourceLoader.getLines ignores Byte Order Markers (BOMs) at the beginning of input files, these are often created by editors such as Windows Notepad"; it now lives in WordlistLoader.getLines (WordlistLoader.java:302-304). "LUCENE-5505: HunspellStemFilter ignores BOM markers in dictionaries" is listed under Bug fixes in 4.8.0.
• Valid input, no documented exclusion. A file with a signature is valid UTF-8, and the strict decoder accepts it. The Unicode BOM FAQ calls an initial BOM a signature. No javadoc, test, CHANGES or MIGRATE entry says these dictionaries keep a mark; upstream TestFileDictionary and TestPlainTextDictionary never feed one.
• Silent. Nothing is raised, and printed suggestions look right.

The checkers built the strongest case for "intended": (a) the javadoc literally holds, because the code does decode UTF-8 and Java's decoder keeps U+FEFF, for compatibility (JDK-4508058, Won't Fix); this "literal UTF-8 contract" reading is the independent GPT-5.6 reviewer's, and its refute pass rated the case strong. (b) The Unicode BOM FAQ advises producers not to write a BOM where the data type is known; in the same spirit Uwe Schindler wrote on SOLR-1860 (2010) "If it's documented to be UTF-8, its clear what you have to provide". (c) Michael Busch closed LUCENE-1413 (2009, PlainTextDictionary and UTF-8 files) as Invalid with the advice to pass a decoding Reader. (d) In the 2010 solr-user thread behind SOLR-1865, Chris Hostetter asked whether such files should be considered a bug; Robert Muir answered "No.", then proposed ignoring the mark on the first line "to be more user-friendly" (typed Improvement); Jan Høydahl resolved a FileDictionary format question (SOLR-9368, 2016) with a reference-guide change. (e) Lucene is not uniform: WordlistLoader.getWordSet(InputStream) keeps the mark.

Why we think it fails: (a) to (c) are about who supplies the characters; with these constructors Lucene owns the bytes-to-chars step, so the caller cannot drop the mark inside them. (d) shows the project then chose to skip marks in its loaders (Yonik Seeley in the same thread: "we should fix if it's practical to do so, rather than passing the buck"), and LUCENE-5505 filed the hunspell equivalent as a bug fix. (e) shows an inconsistency, not an intent to keep marks. Verdict: a real, low-priority defect; expect "improvement" or a javadoc note as possible answers.

Option A was applied in a scratch worktree of main 645d4fe14a20 and run through gradle with assertions on and off; the stock runs used a second scratch worktree of the same commit, and both were removed afterwards. Options B and C were not run.

A skipBom flag is set only by FileDictionary(InputStream, String) (which FileDictionary(InputStream) delegates to), PlainTextDictionary(Path) and PlainTextDictionary(InputStream). On the first iteration the reader does in.mark(1), reads one character, calls in.reset() unless it was U+FEFF, and clears the flag. +23 lines over the two files; the snippet shows the FileDictionary half, and PlainTextDictionary.FileIterator.next() gets the same block.

The skip happens at the first iteration, not in the constructor, because FileDictionary(InputStream) and PlainTextDictionary(InputStream) do not declare IOException. Files without a mark, and every Reader constructor, behave as before; a file that holds only the mark gives no entry. This is the hunt's fix sketch with one behavioural change: the FileDictionary flag is cleared after the first skip (the sketch kept it final), so a second getEntryIterator() on a partly read dictionary cannot drop a U+FEFF that starts a later line. It also drops the sketch's explicit this.skipBom = false; in FileDictionary(Reader, String), redundant once the field is not final. The one-test repro exercises only the FileDictionary half; the PlainTextDictionary half was validated with the hunt's 4-test check.

     private FileIterator() throws IOException {
+      if (skipBom) {
+        // skip an initial UTF-8 byte order mark, as WordlistLoader.getLines does
+        skipBom = false;
+        in.mark(1);
+        if (in.read() != '\uFEFF') {
+          in.reset();
+        }
+      }
       line = in.readLine();

Trade-offs

• A caller that decodes the file itself and passes a Reader still gets the glued mark. That is in line with Dawid Weiss's remark on LUCENE-7785 that a mark in a Reader means the conversion from bytes went wrong before, and with Uwe Schindler's note on SOLR-1860 that Lucene's Reader-level stopword parser "does not care about encodings at all"; but it leaves Solr's characterEncoding-without-fieldType path and its FileDictionaryFactory, which pass a Reader, uncovered.
• Like getLines, it cannot tell a byte order mark from a file whose first character is meant as ZERO WIDTH NO-BREAK SPACE (not a realistic dictionary entry).
• The four javadoc notes could add "a leading byte order mark is skipped". No CHANGES entry is drafted.

What the checks validated

• Gradle in a scratch worktree of main 645d4fe14a20, seed DEADBEEF, assertions on and off: the repro alone fails stock and passes with the fix.
• The repro, a 4-test PlainTextDictionary check from the hunt (the Path and InputStream constructors, a SpellChecker, a no-mark control; not part of the repro) and the upstream TestFileDictionary, TestPlainTextDictionary and TestSpellChecker: 5 classes, 17 tests; stock 4 failures, with the fix 0, each way.
• All of :lucene:suggest:test with the fix, assertions on: 33 classes (31 upstream), 241 tests, 7 skipped, 0 failures.
• :lucene:suggest:check -x test (google-java-format, ecjLint, forbiddenApis) passes with the fix and the repro added.
• Earlier, the hunt's sketch in the repro harness: the finder's 11 tests, an independent 14-test JUnit and the 12 upstream tests pass, and the bounded-exhaustive sweep reports 0 mismatches.

Set the flag in the Reader constructors too. WordlistLoader.getLines is no precedent for this: it takes an InputStream and decodes it itself (WordlistLoader.java:297-299), and WordlistLoader's Reader methods do not skip a mark.

Trade-offs

Also covers Solr's characterEncoding path and FileDictionaryFactory, but a Reader whose text really starts with U+FEFF loses it, against the committers' remarks that Reader-level parsers do not deal with encodings; the Reader behaviour is outside this report's claim.

Add to the four "content is treated as UTF-8" notes: "supply UTF-8 without a byte order mark".

Trade-offs

The smallest change and a plausible maintainer answer (SOLR-9368 was resolved with a documentation change), but files that editors save with a mark still lose their first entry silently.

• First signal. The hunt's round-14 planner carried the kept-BOM hypothesis of four earlier findings (word lists, ICU rule files, Japanese and Korean user dictionaries, synonyms files) over to the suggest dictionaries by reading the code: constructors that decode UTF-8 themselves versus WordlistLoader.getLines. Its executed differential probe failed 8 of 16 checks, every row with a mark.
• Method (semi-formal): bounded-exhaustive metamorphic sweep on the stock main jars, public API only. Law: loading EF BB BF + F gives the same entries (and lookups) as loading F. Inputs: every file of 1 to 3 lines over 21 FileDictionary and 7 PlainTextDictionary line kinds with 1- to 4-byte characters, LF and CRLF, with and without a trailing line end, plus lookups through WFST, FST, TST and AnalyzingSuggester.
• Signal. 81,080 of 81,080 checks on the decoding constructors mismatch (entries 80,976, mark-only files 4, lookups 100); all 158,732 controls hold (CRLF vs LF, trailing line end, InputStream vs Reader without a mark); with the fix shadowed, 0 mismatches. The same on branch_10_6. The Reader constructors also differ (79,383 checks) and are not claimed.
• The finder then wrote a public-API JUnit (11 tests), traced the history to 2012, and the repro was later minimized to one test.