Kuromoji and Nori user dictionaries keep a UTF-8 byte order mark on line 1

JapaneseTokenizerFactory and KoreanTokenizerFactory take a userDictionary file. Each factory decodes the file itself with a strict decoder (UTF-8 by default) and passes the reader to UserDictionary.open, whose loop removes # comments and skips blank lines but never removes a leading byte order mark; Java's UTF-8 decoder keeps it as U+FEFF. In Nori the first entry is stored under the key U+FEFF + 세종시, which ordinary text does not contain, so 세종시 is segmented [세, 종시] (the result without any user dictionary) instead of [세종, 시], with no error. Kuromoji fails at analyzer setup: the dictionary of its own testUserDict (line 1 is a comment) throws ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1, and an entry on line 1 throws Illegal user dictionary entry, whose two quoted strings print the same. Lucene's shared line loader for the other analysis resource files skips the mark (WordlistLoader.getLines, "skip initial bom marker", SOLR-1865), so the stop-tags file of the same module and the new Thai user dictionary accept a BOM. Present since Kuromoji 3.6.0 (2012) and Nori 7.4.0 (2018). Skipping a U+FEFF at the start of line 1 in both loops fixes it; both modules' full test suites pass with that change.

• User dictionary: the userDictionary argument of JapaneseTokenizerFactory and KoreanTokenizerFactory (a resource path, decoded with userDictionaryEncoding, default UTF-8), or a Reader passed to UserDictionary.open.
• Byte order mark (BOM): the 3 bytes EF BB BF at the start of a file saved as "UTF-8 with BOM". Java's UTF-8 decoder returns them as the character U+FEFF (JDK-4508058, Won't Fix).

1. Both factories' inform open the file, build a strict decoder (default UTF-8), wrap it in an InputStreamReader and call UserDictionary.open(reader) (KoreanTokenizerFactory.java:110-124, JapaneseTokenizerFactory.java:142-156). None of the 173 charsets of JDK 25 strips a UTF-8 BOM, so userDictionaryEncoding is no workaround (a UTF-16 file with a BOM works: that decoder consumes its BOM).
2. Nori (ko/dict/UserDictionary.java:56-65): line.replaceAll("#.*$", ""), skip if line.trim().length() == 0, else keep the line. trim() removes only characters up to U+0020, so U+FEFF stays. An entry on line 1 is stored under U+FEFF + 세종시 (:89-90, :125-131). The lookup starts at every position of the text, so the key could only match where the text itself has U+FEFF before 세종시; ordinary text does not, so for 세종시 the system dictionary decides: [세, 종시]. A comment or blank line 1 leaves a junk entry U+FEFF with no visible effect in the inputs tested.
3. Kuromoji (ja/dict/UserDictionary.java:59-69): the comment pattern ^#.*$ (:37) does not match U+FEFF + # ..., and the line is not empty, so it is parsed as a one-field entry and the constructor's values[1] (:97) throws ArrayIndexOutOfBoundsException, against the loop's own comment "Skip empty lines or comment lines" (:63). An entry on line 1 keeps U+FEFF in its surface (:96) but not in its segmentation, so the LUCENE-8933 check (:114-123) throws Illegal user dictionary entry.
4. The other resource files skip it: WordlistLoader.getLines drops a U+FEFF at the start of the first line (WordlistLoader.java:302-304, "skip initial bom marker"). AbstractAnalysisFactory.getWordSet and getLines use it, and with them the stop-tags file of JapanesePartOfSpeechStopFilterFactory and the new ThaiTokenizerFactory user dictionary. The two CJK user dictionaries read lines themselves.

Before 9.0.0 Kuromoji did not throw for an entry on line 1; like Nori, it silently ignored it (3.6.0: 朝青龍 gave [朝, 青龍] instead of [朝青龍]).

• Who is exposed. Applications that opt in to a Kuromoji or Nori user dictionary: the userDictionary argument of JapaneseTokenizerFactory / KoreanTokenizerFactory (directly, through the japanese / korean SPI names, or through a search-server schema, as both factory javadocs show), or code that passes a Reader over such a file to UserDictionary.open. The default configuration, without a user dictionary, is not affected.
• What it takes. The dictionary file saved as UTF-8 with a BOM. User dictionaries are hand-edited word lists, and editors such as Windows Notepad often write a BOM (the SOLR-1865 change note says so).
• Nori: silent. When line 1 is an entry, that entry is ignored: documents and queries with that word are segmented as if it were not in the dictionary, with no error. The other entries work. A comment or blank line 1 hides the problem (no visible effect in the inputs tested).
• Kuromoji 9.0.0 and later: loud. An ordinary BOM-prefixed file fails at analyzer setup, with ArrayIndexOutOfBoundsException (comment or blank line 1, the shape of Lucene's own test dictionary and of the Solr sample) or Illegal user dictionary entry (entry on line 1); neither names the byte order mark. Kuromoji 3.6.0 to 8.11.4 silently ignored an entry on line 1, like Nori.
• Scale. Deterministic, at setup, independent of index size: one entry (Nori) or the whole dictionary (Kuromoji).
• How likely. Rare (registry: rare, priority low): opt-in, needs the BOM, Nori loses only an entry on line 1, Kuromoji fails loudly so users re-save the file, and no report was found since Kuromoji shipped in 2012.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<세[종]> but was:<세[]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.ko.TestNoriUserDictionaryUtf8Bom.testLeadingBomKeepsFirstEntry(TestNoriUserDictionaryUtf8Bom.java:33)

• The factory owns the decoding. It turns the file's bytes into characters itself (JapaneseTokenizerFactory.java:142-156, KoreanTokenizerFactory.java:110-124); no factory argument helps: userDictionaryEncoding cannot remove a UTF-8 BOM, because no JDK 25 charset does. The only workarounds are re-saving the file without the BOM or wrapping the ResourceLoader to drop the three bytes, which a workaround probe on main confirmed; a workaround does not make the behaviour intended.
• Lucene skips the mark elsewhere. WordlistLoader.getLines (SOLR-1865, 2010: it "ignores Byte Order Markers (BOMs) at the beginning of input files, these are often created by editors such as Windows Notepad"), the hunspell dictionary (LUCENE-5505, listed under 4.8.0 "Bug fixes") and the newest tokenizer user dictionary (Thai, GITHUB#16722, 10.6.0, read through getWordSet). Of the three tokenizer user dictionaries, one skips a BOM and two do not.
• The loader's own intent. Kuromoji's loop says "Skip empty lines or comment lines"; with a BOM it instead crashes on a comment line.
• Nothing documents or tests the current behaviour. No javadoc, CHANGES or MIGRATE entry says a user dictionary must be BOM-free, UserDictionary.open has no javadoc, no Kuromoji or Nori test uses a BOM, and neither loader ever had BOM handling.
• Valid input, silent effect. A BOM-prefixed file is valid UTF-8 and Nori accepts it without a word; a silently ignored entry is the kind of leniency a committer objected to in LUCENE-8584 ("Now they can add something to the end of their dictionary and it gets ignored?").
• Classification is open. No document promises BOM handling, so upstream may call the fix an improvement (SOLR-1865 was typed Improvement).

Rated moderate by the cross-model reviews (an independent reviewer from another model family, OpenAI GPT-5.6, reading-only) and by the intended-behaviour checker. (a) UserDictionary.open(Reader) takes characters and promises no BOM handling; Java keeps U+FEFF on purpose; a committer wrote on LUCENE-7785 "if you have a byte order mark in your character input (reader) then your conversion from bytes is screwed up somewhere before". (b) LUCENE-8933 deliberately made Kuromoji require surface = segmentation, and LUCENE-8584 declined to make the loader lenient ("Its not stuff a library should be doing"). (c) KoreanTokenizerFactory is @lucene.experimental. (d) In 2010 a committer answered "No." to "should we consider that a bug?" about a BOM in a stop list, and SOLR-1865 was typed Improvement.

Why it fails: (a) on the factory path the "conversion from bytes" is Lucene's own code, so that remark argues for fixing the factory, not for keeping the behaviour (it was also about a char filter, not file loading); (b) LUCENE-8933 validates entries, it does not intend a comment line to crash, and the LUCENE-8584 objection was to silently ignored entries, which is what Nori does now; (c) experimental does not make a silent loss intended, and the Kuromoji factory is not experimental; (d) the same 2010 reply went on: "Yes, we can ignore them for the first line of the file to be more user-friendly. I'll open an issue" (SOLR-1865 followed), and in 2014 the same slip in hunspell went under "Bug fixes". What survives is the bug-or-improvement classification, not the defect, and maintainers could still reasonably prefer the factory-only fix, which keeps open(Reader) strict.

Two fixes were applied in a scratch worktree of a9779ef3c193 and run with gradle; the worktree was removed afterwards. The other options were not run.

Before the comment and blank-line checks, remove a U+FEFF at position 0 of the first line; a first comment line is then skipped as intended. The same 6 lines in ja/dict/UserDictionary.java and ko/dict/UserDictionary.java (the Nori loop is shown); the idiom of WordlistLoader.getLines:302-304, with braces (getLines drops U+FEFF until it has kept a line, this drops it from the first line read: the same for a file that starts with a BOM). It reaches every caller of the public open(Reader), including downstream code that reads the file itself.

    // text + optional segmentations
    boolean firstLine = true;
    while ((line = br.readLine()) != null) {
      // skip initial bom marker
      if (firstLine && line.length() > 0 && line.charAt(0) == '\uFEFF') {
        line = line.substring(1);
      }
      firstLine = false;
      // Remove comments
      line = line.replaceAll("#.*$", "");

Trade-offs

Only a U+FEFF at the very start of the input is removed; a later one is untouched. A dictionary whose first surface deliberately began with U+FEFF would change meaning. It changes a public Reader API, which the "a Reader holds characters" view may resist. No CHANGES entry is drafted; open PR #16272 touches both files, so the patch may conflict with it; open PR #14969 edits the Nori constructor, not the loop.

What the checks validated

• The 6 new tests and the closest upstream suites (TestKoreanTokenizerFactory 8, Nori dict.TestUserDictionary 2, TestJapaneseTokenizerFactory 9, Kuromoji dict.TestUserDictionary 7) pass with assertions on and off: 32 tests, 0 failures (stock: 4 of the 6 new tests fail).
• All of :lucene:analysis:nori:test (14 classes, 90 tests, 4 skipped) and :lucene:analysis:kuromoji:test (31 classes, 198 tests, 7 skipped), assertions on: 0 failures.
• :lucene:analysis:nori:check :lucene:analysis:kuromoji:check -x test passes with the new tests in place (google-java-format, ecjLint, forbiddenApis on main and test sources).
• Earlier, with the hunt's version of the same change: the finder's repro 5/5, the reproducer's 13/13, the version-independent probe on main clean, the bounded-exhaustive check 0 mismatches, 26 upstream dictionary and factory tests pass.

Wrap the decoding reader in a PushbackReader, read one character and push it back unless it is U+FEFF, in JapaneseTokenizerFactory.inform and KoreanTokenizerFactory.inform.

        PushbackReader reader = new PushbackReader(new InputStreamReader(stream, decoder));
        // skip initial bom marker
        int first = reader.read();
        if (first != -1 && first != '\uFEFF') {
          reader.unread(first);
        }
        userDictionary = UserDictionary.open(reader);

Trade-offs

Puts the BOM handling where the bytes are decoded and leaves UserDictionary.open(Reader) strict, which answers the "a Reader holds characters" objection; it does not help code that calls open(Reader) on its own reader. Open PR #16272 changes the same KoreanTokenizerFactory.inform line, so this may conflict with it too. A UTF-16 file is unaffected (its decoder already consumed the BOM).

What the checks validated

Assertions on: the 3 Kuromoji tests and the Nori factory and control tests pass; as designed, the Nori testUserDictionaryOpenWithBom still fails (13 tests, 1 failed), because open(Reader) is unchanged. The full modules and the static checks were not run with it.

Throw an IllegalArgumentException naming the byte order mark.

Trade-offs

Nothing is lost silently and the message is precise, but it refuses valid UTF-8 that Lucene's other resource readers accept.

Use one small helper here and in the snowball word-set loader, the ICUTokenizerFactory rule files and the synonym parsers, which keep a BOM the same way.

Trade-offs

Fixes the whole class at once but touches several modules; worth raising if upstream wants one issue.

Method (semi-formal): a bounded-exhaustive metamorphic check on the stock main jars.

• The law: loading a file F and loading BOM + F give the same analysis of every probe text (terms, position increments, position lengths), or the same exception class.
• The harness enumerated every user dictionary of 1 to 3 lines over a 4-line Kuromoji alphabet (two entries, a comment, a blank line) in search and normal mode, and a 5-line Nori alphabet (a compound entry, a noun entry, c++, a comment, a blank line), with LF and CRLF line ends. Kuromoji: 304 of 336 checks mismatch, all loud (exceptions); Nori: 126 of 310, all silent (different segmentation). Three controls that read through WordlistLoader.getLines (stop words, stemmer overrides, char mappings) give 0, and with the fix sketch every family gives 0, so the check both detects the defect and accepts the fix.
• The lead came from an earlier hunt unit that found the same BOM slip in the snowball word-set loader and probed the other readers that bypass getLines (first executed 2026-10-27). The finder then reduced it to a public-API JUnit. No model checker, solver or proof was involved.