FileDictionary reads its fieldDelimiter as a regular expression, so a '|' delimiter makes a valid suggestion file fail to load

FileDictionary reads a text file with one suggestion per line: the suggestion, optionally followed by fieldDelimiter and a weight, and fieldDelimiter and a payload. The default delimiter is a TAB, and two public constructors take any string "to separate out the fields in a line". The iterator splits each line with line.split(fieldDelimiter) (FileDictionary.java:133 and :167 on main 645d4fe14a20), and String.split compiles its argument as a regular expression. With "|" the valid line foo bar|10 splits into 10 one-character fields and getEntryIterator() throws IllegalArgumentException: More than 3 fields in one line; . throws ArrayIndexOutOfBoundsException, + and other lone metacharacters throw PatternSyntaxException, and the anchors $ and ^ never split, so every line silently becomes one suggestion such as foo bar$10$p1 with weight 1 and no payload. TAB and plain delimiters such as ,, ;, :: or " , " work. The regex split came with the configurable delimiter in LUCENE-5337 (2013, first released in 4.7.0); every release from 4.7.0 to 10.6.0 and main is affected. Regex-style delimiters such as "\\|" or the text backslash-t work today only because of this, so a literal-split fix is a behaviour change that needs a MIGRATE note; documenting regex semantics is the alternative.

• Contract. The class javadoc (:30-59) defines an entry as suggestion, suggestion fieldDelimiter weight or suggestion fieldDelimiter weight fieldDelimiter payload (:36-40), says the default fieldDelimiter is TAB (:42) and "An entry cannot have more than two fieldDelimiter" (:52). DEFAULT_FIELD_DELIMITER = "\t": "Tab-delimited fields are most common thus the default, but one can override this via the constructor" (:62-66). Both two-argument constructors use "fieldDelimiter to separate out the fields in a line" (:91-109) and store the string as given.
• Split. The first line is split in the FileIterator constructor, String[] fields = line.split(fieldDelimiter); (:133), every later line in next() the same way (:167). String.split(regex) compiles the argument as a regular expression and drops trailing empty strings.
• Count. More than 3 fields throws IllegalArgumentException("More than 3 fields in one line") (:134-135, :168-169). 3 fields are suggestion, weight, payload; 2 are suggestion, weight (parsed with Long.parseLong, then Double.parseDouble, :207-214); otherwise fields[0] is the suggestion and the weight is 1 (:144-146, :182-184).

• TAB, ,, ;, ::, " , ", #, ~ and 10 more (no metacharacter: the regex matches only itself): parse as documented.
• | and || (alternations of empty patterns: they match the empty string between every two characters): foo bar|10 gives 10 fields and IllegalArgumentException: More than 3 fields in one line (the same with ||). With |, a 3-character line such as a|0 gives NumberFormatException instead, and a short line whose second character is a digit loads silently wrong (a1 becomes a with weight 1, a12 becomes a with weight 1 and payload 2).
• " | " ("space or space": it matches each single space): foo bar | 10 splits at every space into 4 fields, foo, bar, |, 10, and throws the same IllegalArgumentException; a | 0 gives NumberFormatException for "|".
• . (any character: every field is empty and all are dropped): on any non-empty line fields[0] on an empty array throws ArrayIndexOutOfBoundsException (:145), even for a line without a delimiter.
• +, *, ?, (, ), [, {, \ (not a valid pattern on its own): PatternSyntaxException on the first line read, even without a delimiter (an empty file reads no line and loads).
• $, ^ (anchors at the line ends: they never split): no error; the whole line is the suggestion (foo bar$10$p1), weight 1, no payload, and a suggester built from it returns that key.

The *, ?, (, ), [, {, \, ^ and || cases come from the bounded-exhaustive harness only; the others were also executed by independent tests (" | " only by them, it is not among the harness's 30 delimiters). The fact check's per-delimiter probe later ran every delimiter named here on the stock main jar, with the same results.

• Who is hit. Applications that build a suggester (Lookup.build) from a text file through FileDictionary(Reader, String) or FileDictionary(InputStream, String) with a delimiter that contains a regex metacharacter. Lucene itself has no caller of FileDictionary outside its tests. Solr's FileDictionaryFactory has passed the suggester's fieldDelimiter parameter to this constructor unchanged since the same release (SOLR-5378, 4.7.0) and still does on current Solr main.
• Default vs opt-in. Opt-in: the default TAB and delimiters without a metacharacter (,, ;, ::, " , ") are unaffected.
• What goes wrong. | (the likeliest such delimiter) makes a valid file with weights fail at build time with "More than 3 fields in one line" for a line with one delimiter (in the harness 27,927 of 27,930 valid files fail or load wrongly; the 3 others contain only the one-character suggestion a); . and + fail with low-level exceptions; $ and ^ fail silently: every line becomes one suggestion that contains the weight and payload text, with weight 1 and no payloads.
• Scale. None needed: one line is enough. Deterministic, assertions on or off.
• Workaround. Pass Pattern.quote(d) or an escaped regex such as "\\|", or use TAB.
• How likely. Plausible: whoever picks | hits it on first use, which also makes it visible during development; the silent case needs the unusual $ or ^.

Copy the test into lucene/suggest/src/test/org/apache/lucene/search/suggest/ of a Lucene checkout and run it with gradle (command below). It fails before reaching its assertions: getEntryIterator() already throws. With the literal-split fix it passes unchanged, so it doubles as the regression test.

Executed: gradle on a fresh scratch worktree of 645d4fe14a20 (fails); the standalone runner on the main jars with -ea and -da (built from the hunt's 645d4fe14a20 checkout, whose only changes are untracked files) and on the 10.6.0 release jar (fails, same exception).

cp $REPRO/TestFileDictionaryPipeDelimiter.java $LUCENE/lucene/suggest/src/test/org/apache/lucene/search/suggest/
cd $LUCENE && ./gradlew :lucene:suggest:test --tests org.apache.lucene.search.suggest.TestFileDictionaryPipeDelimiter -Ptests.seed=DEADBEEF

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF
TestFileDictionaryPipeDelimiter > testPipeDelimiter FAILED
    java.lang.IllegalArgumentException: More than 3 fields in one line
        at org.apache.lucene.search.suggest.FileDictionary$FileIterator.<init>(FileDictionary.java:135)
        at org.apache.lucene.search.suggest.FileDictionary.getEntryIterator(FileDictionary.java:114)
        at org.apache.lucene.search.suggest.TestFileDictionaryPipeDelimiter.testPipeDelimiter(TestFileDictionaryPipeDelimiter.java:29)

# standalone runner, seed DEADBEEF
main 645d4fe14a20  -ea / -da   tests_run=1 failures=1   same exception
10.6.0             -ea         tests_run=1 failures=1   same exception

• Contract. FileDictionary.java:30-59: the line grammar suggestion fieldDelimiter weight fieldDelimiter payload, "An entry cannot have more than two fieldDelimiter" (:52); the constructors use it "to separate out the fields in a line" (:91-109); the default is a literal TAB character (:66). No javadoc, package doc, CHANGES.txt or MIGRATE.md entry mentions regular expressions. Equally, nothing says the delimiter is literal.
• Tests. TestFileDictionary.generateFileEntry appends the delimiter verbatim (sb.append(fieldDelimiter), :41, :47) and expects the original fields back; testFileWithDifferentDelimiter uses " , " (:176-181).
• History. Before LUCENE-5337 the code was line.split("\t"). 99255ff0da68 (2013-11-13, "Add Payload support to FileDictionary (Suggest) and make it more configurable", now #6401) replaced it with line.split(fieldDelimiter) and added the delimiter constructors. The issue asks "to let the users configure the field delimiter"; its 13 comments never mention regular expressions or escaping. An independent port, Lucene.NET's FileDictionary, splits on the literal string.
• Effect. Read as documented, a valid file with | should load, and $ should not fold weights and payloads into the suggestion text.

The strongest case for "works as designed": regex semantics have been the actual behaviour since 4.7.0, and callers rely on them. "\\|" and "\\s*,\\s*" work today, and so does the two-character text backslash-t, which is how Solr's suggester guide writes the default: in an XML config it reaches FileDictionary as a backslash and a t and matches a TAB only because it is parsed as a regex (a 2016 solr-user config does this). A reliance test pins all three: it passes on main and 9.12.3 and fails 3 of 3 with the literal fix. SOLR-9368 resolved another FileDictionary format mismatch (comment lines) with a doc fix.

Why it fails: in SOLR-9368 the javadoc matched the code; here nothing documents regex semantics, every written description (javadoc, test generator, Solr guide wording, the Lucene.NET port) reads naturally as a literal delimiter, and nothing written suggests that | should break a valid file. The reliance limits the fix, not the finding: because callers depend on the regex behaviour, documenting it may well be the least disruptive resolution, not only a fallback; a literal split needs a MIGRATE note. Verdict: a defect, a mismatch between code and documentation.

Two options. The literal split was validated with gradle in a scratch worktree at 645d4fe14a20; the doc-only option changes no behaviour. A natural split: the literal split on main for the next major release with a MIGRATE entry, the doc fix on the 10.x branch.

In FileDictionary.java, store the delimiter as a Pattern compiled with Pattern.LITERAL in both two-argument constructors and split with it at :133 and :167. Pattern.quote(fieldDelimiter) at the call sites is equivalent.

private final Pattern fieldDelimiter;

public FileDictionary(Reader reader, String fieldDelimiter) {
  in = new BufferedReader(reader);
  this.fieldDelimiter = Pattern.compile(fieldDelimiter, Pattern.LITERAL);
}
// same in FileDictionary(InputStream, String)

String[] fields = fieldDelimiter.split(line); // was: line.split(fieldDelimiter), at :133 and :167

Trade-offs

• Literal delimiters (TAB, ,, " , ", ...) behave exactly as before.
• Behaviour change for regex-reliant callers, and a silent one: "\\t" (the text backslash-t, as Solr configs write the default), "\\|" and "\\s*,\\s*" stop splitting, and every line becomes one suggestion with weight 1 and no payload (a reliance test: 3 of 3 pass today, 3 of 3 fail with the fix). Hence a MIGRATE note and the next major release.
• Side effect: the pattern is compiled once instead of on every line for delimiters that String.split does not fast-path.

What the checks validated

• Gradle, scratch worktree at 645d4fe14a20, JDK 25, -Ptests.seed=DEADBEEF: the minimal test passes unchanged and TestFileDictionary passes 5 of 5; :lucene:suggest:checkGoogleJavaFormat passes, and the whole :lucene:suggest:test suite passes: 237 tests in 32 suites, 7 skipped, 0 failures.
• Earlier shadow runs of the same diff: the finder's 4 tests and an independent reproducer's 6 tests pass, upstream TestFileDictionary 5 of 5.

State in the class and constructor javadoc that fieldDelimiter is a regular expression as in String.split, and that a literal delimiter such as | must be passed as Pattern.quote("|").

Trade-offs

No behaviour change, safe to backport; SOLR-9368 is a doc-fix precedent for this class. But the | trap and its misleading "More than 3 fields" message stay, only documented, and $/^ still mis-parse silently.

What the checks validated

Not executed (documentation only).

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier), in a hunt unit on whether suggest dictionaries preserve their (suggestion, weight, payload) tuples, run on the stock main jars at ea57360912ed.

• Every file of 1-3 lines over 3 suggestions x 4 weights (including none) x 3 payloads (including none; a payload only after a weight), 27,930 files per delimiter, for 30 delimiter strings (2,954 for the space delimiter: the harness skips files in which a suggestion or payload contains the delimiter, which happens only for the space, in the suggestion a b and the payload q r).
• Reference: a parser that splits each line on the literal delimiter text, following the class javadoc.
• Result: the 17 delimiters without a regex metacharacter (TAB, space, ,, ;, ::, " , ", #, ~, ...) agree on every file; the 13 with one ($ ( ) * + . ? [ \ ^ { | ||) disagree on 27,891 to 27,930 of 27,930 files each.
• Code reading then located line.split(fieldDelimiter), and the finding was reduced to a public-API JUnit test, later to the single test shown here.