SolrSynonymParser rejects a whole synonyms file when one line holds only spaces or tabs

SolrSynonymParser, the default parser of SynonymGraphFilterFactory and SynonymFilterFactory, documents that "Blank lines and lines starting with '#' are comments", but it skips a line only when it has zero length. A line of only spaces or tabs is parsed as a rule whose single term trims to the empty string; the analyzer leaves nothing of it, and the whole file is rejected with ParseException: Invalid synonym rule at line N (the factories wrap it in IOException: Error parsing synonyms file:). Through the factories it fails loudly when the analyzer is built and names the line; nothing goes silently wrong. The pre-3.4 Solr loader of the same format, and Lucene's WordlistLoader today, skip such lines. Every release from 3.4.0 to 10.6.0 and main is affected; the fix is one line.

• SolrSynonymParser: reads the Solr synonyms format into a SynonymMap for SynonymGraphFilter. SynonymGraphFilterFactory and the deprecated SynonymFilterFactory use it when format is missing or solr (the default), and analyze rule terms with a WhitespaceTokenizer unless a tokenizer or analyzer is configured.
• Blank line: the javadoc does not define it. Lucene's WordlistLoader uses the word for lines that are empty after trim(), and so did the Solr loader that read this format before 3.4.

1. addInternal skips a line only if line.length() == 0 || line.charAt(0) == '#' (SolrSynonymParser.java:85-86, comment "ignore empty lines and comments").
2. For the line " ", split(line, "=>") gives one side, so it takes the equivalent-synonyms branch; split(line, ",") gives [" "], and analyze(unescape(...).trim(), ...) analyzes "" (:90-91, :114-117).
3. SynonymMap.Parser.analyze throws IllegalArgumentException("term: was completely eliminated by analyzer") when no characters are left (SynonymMap.java:374-377), or "analyzed to a zero-length token" for a tokenizer that emits one empty token (:349-352). In the hunt's security probe the line was rejected in all 48 combinations of WhitespaceAnalyzer, StandardAnalyzer, KeywordAnalyzer, SimpleAnalyzer, expand on/off and six blank forms. Only an analyzer that emits a non-empty token for empty input accepts such a line (executed with a custom one).
4. parse() turns it into ParseException("Invalid synonym rule at line 2") (:72-76); the factories wrap that in IOException("Error parsing synonyms file:") (SynonymGraphFilterFactory.java:162-163, SynonymFilterFactory.java:165-166). Through the factories no map is built, so every rule in the file is lost. A direct caller that catches the ParseException and calls build() anyway gets only the rules before the line (executed).

• Who is hit. Callers of SolrSynonymParser.parse, and users of SynonymGraphFilterFactory or SynonymFilterFactory with the default format, directly or through CustomAnalyzer and configurations that name the factories. Such a line is easy to produce: an auto-indented separator line, whitespace left on an otherwise empty line, a last line holding a space (trailing whitespace after a rule is harmless: every term is trimmed).
• Default vs opt-in. The synonym filter is in no stock Analyzer; within it, the bug needs only defaults (format solr). SolrSynonymParser and SynonymGraphFilterFactory carry @lucene.experimental.
• Scale. None: one whitespace-only line anywhere in the file.
• Visibility. Loud and early: through the factories the analyzer is not built, the error names the line, and nothing is indexed or searched with a partial map (only a direct caller that ignores the exception and builds anyway could get one). The message is misleading: an "invalid synonym rule" with an empty term: for a line that holds no rule.
• Severity. Low: a wrong error; through the factories no wrong results and no index damage; deleting the whitespace works around it.

parse() alone shows the bug (it throws before build()), so no map or token stream is needed. WhitespaceAnalyzer matches the factories' default tokenizer for rule terms. The try/catch only turns the ParseException into an AssertionError that states expected vs actual.

Also fails with seed 5EED1234 (5 of 5) and on the 10.6.0 release jars with assertions on and off; passes with either fix.

As the regression test: the same test, for example as a new method in TestSolrSynonymParser.

cp TestSolrSynonymParserWhitespaceOnlyLine.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/synonym/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.synonym.TestSolrSynonymParserWhitespaceOnlyLine -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

# gradle, main 645d4fe14a20, seed DEADBEEF (assertions on; the same with -Ptests.asserts=false)
TestSolrSynonymParserWhitespaceOnlyLine > testSpacesOnlyLineIsBlank FAILED
    java.lang.AssertionError: expected: line 2 (three spaces) is skipped as blank; actual: java.text.ParseException: Invalid synonym rule at line 2 <- java.lang.IllegalArgumentException: term:  was completely eliminated by analyzer
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1FA61A41CFDB32CB]:0)
        at org.apache.lucene.analysis.synonym.TestSolrSynonymParserWhitespaceOnlyLine.testSpacesOnlyLineIsBlank(TestSolrSynonymParserWhitespaceOnlyLine.java:34)

        Caused by:
        java.text.ParseException: Invalid synonym rule at line 2
            at org.apache.lucene.analysis.synonym.SolrSynonymParser.parse(SolrSynonymParser.java:74)
            at org.apache.lucene.analysis.synonym.TestSolrSynonymParserWhitespaceOnlyLine.testSpacesOnlyLineIsBlank(TestSolrSynonymParserWhitespaceOnlyLine.java:30)

            Caused by:
            java.lang.IllegalArgumentException: term:  was completely eliminated by analyzer
                at org.apache.lucene.analysis.synonym.SynonymMap$Parser.analyze(SynonymMap.java:375)
                at org.apache.lucene.analysis.synonym.SolrSynonymParser.addInternal(SolrSynonymParser.java:117)
                at org.apache.lucene.analysis.synonym.SolrSynonymParser.parse(SolrSynonymParser.java:71)

• Javadoc (SolrSynonymParser.java:33): "Blank lines and lines starting with '#' are comments."
• Lucene's own word for it. WordlistLoader "adds every non-blank line ... (omitting leading and trailing whitespace)" and does word = word.trim(); // skip blank lines (WordlistLoader.java:43-44, :56-58; also :107, :123-124, :294, :307-308). TestWordlistLoader.testGetLines feeds a line holding one space and expects it skipped. GITHUB#12291, "Skip blank lines from stopwords list", is listed under Bug Fixes in 9.7.0.
• The format's previous loader. Solr's SolrResourceLoader.getLines, which read synonyms files before 3.4, did if (word.startsWith("#")) continue;, then word=word.trim();, then // skip blank lines. LUCENE-3233 created SolrSynonymParser to parse "the existing solr synonyms format"; its thread lists the known differences as "deduping rules, lowercasing" and never mentions blank lines. In the 4.0.0 jar the kept pre-3.4 loader (luceneMatchVersion=LUCENE_33) accepts the file, while LUCENE_34 and LUCENE_40 reject it.
• No test asserts the rejection: no test in the analysis-common module uses a whitespace-only line either way.
• No file that parses today with the analyzers tried changes: with them such a line is rejected in every combination tried. Only an analyzer that emits a token for empty input lets it parse (executed with a custom one); for it, the fix would drop a one-term rule.

The strongest case for "intended" (an intended-behaviour devil's advocate, and a reviewer from another model family, OpenAI GPT-5.6, who rated it "moderate"): the inline comment says "ignore empty lines" (:86); the comment check is literal too, so # x is a rule; the parser is strict on purpose, and upstream tests require a ParseException when the analyzer removes a term (TestSolrSynonymParser.testInvalidAnalyzesToNothingOutput / Input, inputs a => 1 and 1 => a); the class is @lucene.experimental; the behaviour is 15 years old and the error names the line.

Why it fails: those strictness tests use lines that hold a rule whose real term the analyzer removes; a whitespace-only line holds no rule, and the "term" in the error is an empty string the user never wrote. The inline comment is the only support for the narrow reading, while the public javadoc, Lucene's other line loaders and this format's own previous loader all skip trimmed-empty lines, and nothing records a decision to narrow it. These points bear on severity and on the "bug or improvement" label, not on whether the code matches the documented format.

Two one-line fixes in SolrSynonymParser.addInternal (SolrSynonymParser.java:85-86); both were run through gradle in a scratch worktree of 645d4fe14a20.

Matches the pre-3.4 Solr loader and WordlistLoader exactly.

-      if (line.length() == 0 || line.charAt(0) == '#') {
-        continue; // ignore empty lines and comments
+      if (line.trim().isEmpty() || line.charAt(0) == '#') {
+        continue; // ignore blank lines and comments
       }

Trade-offs

Skips exactly the lines whose single term the parser would trim to the empty string (it uses the same trim() on every term, :99, :105, :117); with the analyzers tried every such line fails today, so no file that parses today with them changes meaning (an analyzer that emits a token for empty input is the one exception). No public API signature changes; it is a leniency change: files rejected today are accepted. Indented comment lines ( # x) stay rules, as before, and so do lines of only Unicode spaces that trim() keeps, such as U+3000 (rejected today with WhitespaceAnalyzer, and still rejected). Cost: one trim() per line at load time.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20: without the fix the repro fails with assertions on and off.
• With the fix: it passes with assertions on and off (-Ptests.asserts=false).
• Closest suites: every test class in org.apache.lucene.analysis.synonym (including word2vec) with the repro: 12 classes, 121 tests, 0 failures, 0 skipped.
• ./gradlew :lucene:analysis:common:check -x test (google-java-format, ecj lint, forbidden APIs, generated-file checksums) passes with the fix and the repro in place, once ./gradlew tidy has wrapped the repro's long message line; before that, the format check failed only on that test line. The full test suite was not run.

The same condition with String.isBlank().

-      if (line.length() == 0 || line.charAt(0) == '#') {
+      if (line.isBlank() || line.charAt(0) == '#') {

Trade-offs

A different set of lines, not just a broader one: trim() removes every character up to U+0020, while isBlank() uses Character.isWhitespace. So B also skips lines of Unicode whitespace that trim() keeps (U+2003, U+3000), but still rejects lines of control characters that trim() removes and that are not whitespace (U+0001). Executed on the main jars (couch, sofa / X / big, large): a U+0001 line is rejected today, parses with A and is still rejected with B. A U+3000 or U+2003 line is rejected today with WhitespaceAnalyzer and parses with B; with KeywordAnalyzer it parses today as a one-term rule that adds no mapping with expand on and maps the term to itself with expand off (3 words). B skips it, so with expand off the self-mapping is lost (2 words). B therefore changes how a file that parses today is read, in a small way; A matches the old loader exactly and is the safer choice (the adjudication rated either fix safe before this difference was found).

What the checks validated

Gradle in the same scratch worktree: the repro and TestSolrSynonymParser pass with assertions on (11 tests, 0 failures); the other suites and the static checks were not run through gradle for this option (the fact check later ran the repro with it through gradle with assertions off: pass). Earlier, shadowed into the stock main jars, the four earlier test classes (21 tests) and the 10 upstream TestSolrSynonymParser tests pass, and the minimal repro passes with assertions on and off.

Would also turn indented # x lines into comments.

Trade-offs

A different change: it alters how files that parse today are read (an indented # line is a rule today, and was in the pre-3.4 loader too). Not tried.

Method: semi-formal (bounded-exhaustive translation validation). The harness enumerated synonym files of 1-3 rules, with expand and dedup on and off, parsed and built each with SolrSynonymParser and SynonymMap.Builder, decoded the resulting FST rule set and compared it exactly with a reference implementation of the javadoc's rules: 253,130 files, 1,012,520 checks, 0 mismatches; two deliberately broken reference variants are caught, so the check is not vacuous. The reference had to say what a blank line is, so whitespace-only lines were checked as their own class, and a public-API JUnit test then showed the rejection on the stock main jars (2 of 3 tests fail: spaces-only and tab-only line; the empty-line control passes).