ICUTokenizerFactory keeps the UTF-8 byte order mark of a custom rule file as rule text

ICUTokenizerFactory's rulefiles argument replaces the break rules of one script with a rule file of your own. parseRules reads that file as UTF-8, blanks the lines that start with # and passes the text to ICU's new RuleBasedBreakIterator(String), but it never removes a leading byte order mark. ICU's string API treats U+FEFF as an ordinary literal character, so the BOM becomes part of the file's first statement. If that statement is a rule, it compiles but can no longer match: the one-rule file .+ {200}; gives no token for ab cd, and Lucene's own test file KeywordTokenizer.rbbi with a BOM drops the Cyrillic and Thai tokens (4 tokens instead of 6), with no error. If it is a !! directive, ICU throws Error 66052 at line 20 column 1 (a rule syntax error) for Lucene's Latin-break-only-on-whitespace.rbbi, pointing away from the invisible cause on line 1. ICU's own compiler for this file format, genbrk, strips a BOM and its man page recommends one; Lucene's word-list and hunspell readers skip it too. Present since rulefiles arrived in 4.1.0 (SOLR-4123, 2012). Skipping a U+FEFF at the start of the first line fixes it; the whole ICU module passes with that change.

• rulefiles: an argument of ICUTokenizerFactory (ICU analysis module) that replaces the break rules of one script with your own ICU rule file, for example rulefiles="Latn:my.Latin.rules.rbbi".
• Rule status: each ICU rule ends with a number such as {200}; text a rule matches gets that status. ICUTokenizer emits only pieces with a non-zero status and skips status-0 pieces (ICUTokenizer.java:209-213).
• Byte order mark (BOM): the 3 bytes EF BB BF that many editors write at the start of a UTF-8 file. Decoded, it is the character U+FEFF.

1. inform calls parseRules once per script entry of the rulefiles map (ICUTokenizerFactory.java:108-135, the call at :119; a repeated script overwrites the earlier pair, :86-95).
2. parseRules reads the file with IOUtils.getDecodingReader(stream, UTF_8) (:141). Java's UTF-8 decoder keeps a leading BOM as U+FEFF.
3. line.startsWith("#") is false for U+FEFF# ..., so the negated test if (!line.startsWith("#")) (:144) is true and the first line is kept instead of blanked. Lines are joined with \n, so line numbers stay as in the file (:147).
4. new RuleBasedBreakIterator(rules.toString()) (:150). ICU's rule scanner skips only Pattern_White_Space; U+FEFF is not in that set, so it is a literal character. ICU still reads the rest of a # line as a comment, so U+FEFF joins the first statement, after any comment lines.
5. First statement is a rule (.+ {200};): it compiles as "U+FEFF, then .+". Ordinary text has no U+FEFF, so the rule does not match it and ICU falls back to breaking after each character with status 0. Directly in ICU4J: .+ {200}; on ab cd gives its first boundary at 5 with status 200; with U+FEFF in front, at 1 with status 0. The tokenizer skips all of it. No error.
6. First statement is a directive or a variable definition: U+FEFF !!forward; is ICU error 66052 (U_BRK_RULE_SYNTAX), reported at the start of the directive (Lucene's two files: line 20 or 44, column 1, after their comment lines); U+FEFF $Var = ...; reads as a use of an undefined variable, error 66058, reported just past the variable name (the independent reproduction's $Ws file: line 1, column 5; with $Whitespace, column 13). Neither message names the invisible character at line 1, column 1.

The default configuration (no rulefiles) loads precompiled Default.brk and MyanmarSyllable.brk (DefaultICUTokenizerConfig.java:81-84, :147) and never runs parseRules; none of Lucene's five checked-in .rbbi files starts with a BOM (the build-time sources Default.rbbi and MyanmarSyllable.rbbi, and the three test files).

• Who is exposed. Applications that opt in to custom per-script rules: ICUTokenizerFactory with rulefiles, directly, through the icu SPI name, or through a search server schema built on it (the factory javadoc shows such a schema).
• What it takes. One custom rule file saved as UTF-8 with a BOM. Editors such as Windows Notepad often write one (the SOLR-1865 change note says so), and ICU's genbrk(1) man page recommends one.
• Silent case. The first statement is a rule, the shape of Lucene's KeywordTokenizer.rbbi: no error; text that only the first rule covers gets no tokens. With a one-rule file that is all text of the script, which is then not indexed and cannot be found; an index built that way needs reindexing after the fix.
• Loud case. The first statement is a !! directive or a variable definition, the shape of two of Lucene's three test rule files: the analyzer fails to build with an ICU syntax error whose position points at that statement (a directive's line, or just past a variable's name), not at the invisible BOM at line 1, column 1. ICU's own rule guide calls stray literals "surprisingly hard to recognize and debug".
• Scale. Deterministic: the rules are compiled once, when the analyzer is built, whatever the index size. Which tokens are lost depends on the text and on which rules cover it.
• How likely. Rare (registry: rare, priority low): opt-in, needs the BOM, a file that starts with a directive or a variable fails loudly (two of Lucene's three test files have that shape), and no report was found in the searched public sources since 4.1.0 (January 2013).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[ab cd]> but was:<[]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.icu.segmentation.TestICUTokenizerFactoryRuleFileBom.testRuleFileWithUtf8Bom(TestICUTokenizerFactoryRuleFileBom.java:46)

• The factory owns the file reading. It turns bytes into a String and already pre-processes the file (blanking # lines) before handing a String to ICU (ICUTokenizerFactory.java:137-151). ICU maintainers place the encoding signature in the file reader, not the string API (ICU-21907, open: genbrk "detects and honors a BOM"; "the rbbi API for building from rules takes a UnicodeString. The code page stuff is in the genbrk tool").
• ICU's reader for this file format strips the BOM. genbrk.cpp calls ucnv_detectUnicodeSignature and skips the signature (ruleSourceC += signatureLength); its man page (CAVEATS) says "it is recommended that you write this file in UTF-8 with the BOM". (The man page's sentence that a BOM-less file is read in the operating system's codepage is stale: the code assumes UTF-8.)
• Lucene's other resource readers skip it. WordlistLoader.getLines ("skip initial bom marker", WordlistLoader.java:302-304, SOLR-1865, 2010) and the hunspell Dictionary (Dictionary.java:246, :351-352; LUCENE-5505, listed under "Bug fixes" in 4.8.0). parseRules (2012, more than two years later) never adopted it.
• Nothing documents or tests the current behaviour. The javadoc says only that rule files "are compiled by the ICU RuleBasedBreakIterator" and links ICU's syntax page (ICUTokenizerFactory.java:49-52), which says nothing about encodings. SOLR-4123's 21 JIRA comments never discuss encoding, no later commit on the file mentions a BOM, and all five checked-in .rbbi files are BOM-free.
• Classification is open. No Lucene document promises BOM handling for rule files, so upstream may call the fix an improvement: SOLR-1865 was typed Improvement.

An independent reviewer from another model family (OpenAI GPT-5.6), refute lens, executed, without network access: "not real, high", although it reproduced the silent no-token outcome. (a) The javadoc delegates the format to ICU, and RuleBasedBreakIterator(String) treats U+FEFF as a literal by design; Java's UTF-8 decoder keeps it on purpose. (b) Lucene's build-time RBBIRuleCompiler.getRules keeps a BOM the same way. (c) BOM skipping in WordlistLoader and hunspell is component-specific, and Lucene bans BOMs in its own source tree (SOLR-9295, #12937). (d) In 2010 a committer answered "No." to "should we consider that a bug?" about BOMs in Solr's resource files.

Why it fails: (a) the factory reads a file, and ICU's own file reader for this format strips the BOM; the reviewer did not have ICU's genbrk source or man page, and the same reviewer family, given the genbrk man page's CAVEATS section, rated the case real (high), intended-behaviour argument moderate. (b) RBBIRuleCompiler compiles only Lucene's own BOM-free files at build time. (c) The source-tree ban is repository hygiene, not a rule for users' files. (d) That remark predates rule files by more than two years, and the same reply went on: "we can ignore them for the first line of the file to be more user-friendly. I'll open an issue" (SOLR-1865); in 2014 the same slip in hunspell went under "Bug fixes". Two points stand: rule files are trusted configuration, and ICU-2003 (2002, Won't Fix) declined to make ICU's own rule parsers ignore such characters; both fit fixing it in Lucene's file reader (ICU-21907) but keep the classification open. What survives is the bug-or-improvement classification, not the defect.

One fix was applied in a scratch worktree of a9779ef3c193 and run with gradle, assertions on and off; the worktree was removed afterwards. The other options were not run.

Before the # test, remove a U+FEFF at position 0 of the first line. The first comment line is then blanked as intended, and every other line, and so every line number in ICU's error messages, stays as in the file. Same idiom as WordlistLoader.getLines:302-304, with braces; the hunt's sketch is the same change on one line.

    String line = null;
    boolean firstLine = true;
    while ((line = reader.readLine()) != null) {
      // skip initial bom marker
      if (firstLine && line.length() > 0 && line.charAt(0) == '\uFEFF') {
        line = line.substring(1);
      }
      firstLine = false;
      if (!line.startsWith("#")) {
        rules.append(line);
      }

Trade-offs

Only a U+FEFF at the very start of the file is removed; one later in the file, a legitimate literal, is untouched. A file that deliberately began its first rule with a literal U+FEFF would change meaning; ICU's own genbrk strips that character too. Scope: a UTF-8 BOM only (the factory decodes strictly as UTF-8, so UTF-16 files stay unsupported); a file that holds only a BOM was not tested. No CHANGES entry is drafted.

What the checks validated

• The 3 new tests and the 5 upstream TestICUTokenizerFactory tests pass, assertions on and off (stock: the 3 new tests fail).
• All of :lucene:analysis:icu:test, assertions on: 17 classes, 161 tests, 10 skipped, 0 failures.
• :lucene:analysis:icu:check -x test passes with the new test in place (google-java-format, ecjLint and forbiddenApis on main, test and tools sources, checkLicenses).
• The public-API probe with the patched classes: every BOM file gives exactly the tokens of its BOM-less copy, including the two-rule file and Latin-dont-break-on-hyphens.rbbi.
• Earlier, in the repro harness with the hunt's one-line version: the finder's repro 3/3, the reproducer's 10/10, the devil's advocate's 6/6 and the version-independent check on main pass.

if (rules.length() > 0 && rules.charAt(0) == '\uFEFF') rules.deleteCharAt(0); before compiling.

Trade-offs

One line. Lucene would no longer blank the first line, but ICU reads # ... as a comment anyway, so the result should be the same; not executed.

Throw an IllegalArgumentException naming the file and the byte order mark.

Trade-offs

Nothing is lost silently and the message is precise, but it refuses a file that ICU's own tool accepts and recommends.

Use one small helper here and in the snowball word-set loader, the synonym file parsers and the Kuromoji and Nori user dictionaries, which keep a BOM the same way.

Trade-offs

Fixes the whole class at once but touches several modules; worth raising if upstream wants one issue.

Method (semi-formal): bounded enumeration in a hunt unit on analysis-factory resource file formats, on the stock main jar.

• The harness tried 8 spellings of the rulefiles argument and 9 layouts of the same rule file: with and without a final newline, LF and CRLF line ends, comment lines, indented and trailing comments, blank lines, and a leading BOM. Each was checked through the tokenizer against the tokens of the plain BOM-less file, plus one default-configuration control: 18 checks, 2 mismatches, both BOM files (.+ {200};, alone and after a # line).
• The hypothesis came from extending the unit's earlier finding, a BOM kept by the snowball word-set loader (LFV-718), to the other resource readers that do not go through WordlistLoader.getLines.
• The finder reduced it to a public-API JUnit test; no model checker, solver or proof was involved. Writing this doc, a probe added the two-rule case, which shows that only the first rule is broken.