Hunspell: a REP, MAP, ICONV/OCONV, COMPOUNDRULE, BREAK or CHECKCOMPOUNDPATTERN table cut short at the end of the .aff file throws NullPointerException instead of a parse error

Several hunspell .aff directives are counted tables: a header such as REP 2 announces how many entry lines follow. Dictionary.readAffixFile reads those lines with reader.readLine() and passes the result straight to a string method, with no end-of-file check, for REP, MAP, CHECKCOMPOUNDPATTERN, COMPOUNDRULE, BREAK and ICONV/OCONV. When the file ends before the announced lines are all there, readLine() returns null and splitBySpace (or the CheckCompoundPattern constructor) throws NullPointerException: Cannot invoke "String.split(String)" because "line" is null. So the two-line affix file REP 2 / REP a b makes every Dictionary constructor throw an NPE that names no table and no line, and HunspellStemFilterFactory lets it escape unwrapped. The PFX/SFX parser has had the check since #12468 (9.8.0) and throws ParseException: Premature end of rules for ..., and since #14079 (10.2.0) the tolerateRepRuleCountMismatches() javadoc says a wrong REP count gives a ParseException by default. The unchecked reads date from each parser's first commit (ICONV/OCONV in 4.8.0, the other five in 8.9.0 and 9.0.0), and every release since, including 10.6.0, and main are affected. The file is refused either way; only the error is wrong. The same null check that PFX/SFX have, at the six table reads, fixes it.

• Counted table: a header <DIRECTIVE> <count> followed by <count> entry lines, for example REP 2 / REP a b / REP c d (replacement suggestions), MAP (groups of similar characters), ICONV/OCONV (input and output conversions), COMPOUNDRULE, BREAK (word-break characters), CHECKCOMPOUNDPATTERN, and the prefix/suffix tables PFX/SFX.
• Strict mode: the default. tolerateRepRuleCountMismatches() and tolerateAffixRuleCountMismatches() return false unless a subclass overrides them; the other five tables have no tolerant mode.

1. readAffixFile reads the file line by line (Dictionary.java:350 at 645d4fe14a20), trims each line, skips blank ones (:355-356) and dispatches on the first word.
2. Each counted-table branch parses the count and then reads that many lines itself, passing reader.readLine() straight on: REP splitBySpace(reader, reader.readLine(), 3, Integer.MAX_VALUE) (:411), MAP parseMapEntry(reader, reader.readLine()) (:419), CHECKCOMPOUNDPATTERN new CheckCompoundPattern(reader.readLine(), ...) (:462), COMPOUNDRULE and BREAK singleArgument(reader, reader.readLine()) (:621, :633), ICONV/OCONV splitBySpace(reader, reader.readLine(), 3) (:825).
3. At end of file readLine() returns null. splitBySpace starts with line.split("\\s+") (:610) and CheckCompoundPattern's constructor with unparsed.split("\\s+") (CheckCompoundPattern.java:32); neither checks for null, so both throw NullPointerException.
4. If any line follows the short table, even a blank one, the loop takes it as the missing entry (the blank-line skip of step 1 does not apply inside a table) and splitBySpace throws ParseException("Invalid syntax: ..."), or CheckCompoundPattern throws IllegalArgumentException("Invalid pattern: ..."). A following line that happens to fit the entry syntax is silently taken as an entry (read, not executed); that older leniency is not part of this bug.
5. The PFX/SFX parser parseAffix already checks: if (line == null) { ... throw new ParseException("Premature end of rules for " + header, reader.getLineNumber()); } (:703-709, added by #12468).

Gradle on 645d4fe14a20; each .aff announces 2 entries, holds 1 and ends; .dic is 1 / foo:

• Who is hit. Users of the opt-in hunspell analysis component that load an .aff file through any Dictionary constructor (both sorting strategies) or HunspellStemFilterFactory (SPI name hunspellStem, for example in CustomAnalyzer). The failing strict mode is the default.
• What it takes. One of the six tables must be the very last content of the file, with a count higher than the lines present: a download or copy cut off inside the table, or an entry deleted without lowering the count. The layout itself is ordinary: 17 of the 97 .aff files in Lucene's own hunspell test package end with one of these tables. A well-formed file never triggers it, so it is rated rare.
• What happens. The load fails with an exception that does not say what is wrong or where: "line" is null means "your REP table announces more lines than the file has". HunspellStemFilterFactory.inform wraps only ParseException into its Unable to load hunspell data! [dictionary=...,affix=...] IOException (HunspellStemFilterFactory.java:115-118), so factory users do not get the affix file name either.
• What does not happen. No wrong stems, no silent acceptance, no hang, nothing written: the load fails at once even when a table announces 2,000,000,000 lines, and writes no temp files; HunspellStemFilterFactory closes the streams it opened and removes its temp directory (a direct Dictionary caller closes its own streams, as the constructor javadoc asks). Well-formed files are not affected; REP's opt-in tolerant mode loads the truncated REP file, as documented.
• Related, separate fix sites (not part of this bug). In default mode a REP count that is too low gives NumberFormatException from parseNum, which also contradicts the REP javadoc; and the COMPOUNDRULE count sizes an array up front (Dictionary.java:619), so an absurd count gives OutOfMemoryError.

REP is the table with the explicit javadoc contract; the other five tables share the root cause and the fix and were executed with a separate probe (see "The finding in 30 seconds"). The test imports org.apache.lucene.tests.util.LuceneTestCase, a package that exists from 9.1.0 on (9.0.0 still has org.apache.lucene.util.LuceneTestCase), so as written it compiles on 9.1.0 and later; older releases were checked with a release-agnostic main() probe.

As the regression test: the same test, for example as a new method in TestDictionary, after ./gradlew tidy (google-java-format wraps two over-long lines; nothing else changes). Checked with gradle: the formatted test fails on stock, passes with fix B, and :lucene:analysis:common:check -x test is clean with it.

cp TestTruncatedRepTable.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/hunspell/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.hunspell.TestTruncatedRepTable -Ptests.seed=DEADBEEF
# the seed picks the assertion setting (off for DEADBEEF); add -Ptests.asserts=true to force them on: same result

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions off (the seed's default); the same with -Ptests.asserts=true; verbatim excerpt
TestTruncatedRepTable > testRepCountHigherThanLinesAtEndOfFile FAILED
    junit.framework.AssertionFailedError: Unexpected exception type, expected ParseException but got java.lang.NullPointerException: Cannot invoke "String.split(String)" because "line" is null
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:667F3789FE972978]:0)
        at app/org.apache.lucene.test_framework@11.0.0-SNAPSHOT/org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1437)
        at app/org.apache.lucene.test_framework@11.0.0-SNAPSHOT/org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
        at app//org.apache.lucene.analysis.hunspell.TestTruncatedRepTable.testRepCountHigherThanLinesAtEndOfFile(TestTruncatedRepTable.java:34)

        Caused by:
        java.lang.NullPointerException: Cannot invoke "String.split(String)" because "line" is null
            at org.apache.lucene.analysis.hunspell.Dictionary.splitBySpace(Dictionary.java:610)
            at org.apache.lucene.analysis.hunspell.Dictionary.readAffixFile(Dictionary.java:411)
            at org.apache.lucene.analysis.hunspell.Dictionary.<init>(Dictionary.java:264)
            at org.apache.lucene.analysis.hunspell.Dictionary.<init>(Dictionary.java:213)
            at org.apache.lucene.analysis.hunspell.Dictionary.<init>(Dictionary.java:192)

• REP contract (10.2.0 and later). tolerateRepRuleCountMismatches(): "Whether incorrect REP rule counts will be silently ignored. False by default: a {@link ParseException} will happen." (Dictionary.java:1178-1184). #14079 (Robert Muir, 2024-12-18) added it "Similar to support for tolerating PFX/SFX count mismatches", with the PFX/SFX wording (:1170-1176).
• PFX/SFX precedent. #12468 (Peter Gromov, 2023-07-29, "hunspell: check for aff file wellformedness more strictly", 9.8.0) added the readLine() == null check to parseAffix only (:703-709) and extended the existing TestDictionary.testInvalidData (:145-164) with five broken .aff files, each expected to give ParseException; one of them, broken_extraAffRule_last.aff, is an SFX table that announces 4 rules and holds 3 at the end of the file. Up to 9.7.0 that SFX case was an NPE too.
• Constructors. @throws ParseException Can be thrown if the content of the files does not meet expected formats (:187, :204, :224). Nothing documents an NPE for a malformed file.
• Position dependence. The same short table followed by a blank line gives ParseException: Invalid syntax: (executed).
• Maintainer intent. LUCENE-3976 (#5049, 2012) was closed with the preference to find the exceptions malformed affix files cause and "fix them at their source", "on a per-error basis"; #12468 and #16919 (2026-10-28: in AffixCondition an internal AssertionError became an IllegalStateException, and Throwable is no longer caught) continue that work.

The strongest case for "intended" (the study's intended-behaviour checkers, the tie-break and the refute-lens reviewer): the parser was never hardened and ParseException was never exhaustive. Other malformed content already escapes as NumberFormatException (non-numeric count), IllegalArgumentException (bad CHECKCOMPOUNDPATTERN, COMPOUNDRULE or FLAG line), ArrayIndexOutOfBoundsException (short PFX header) or IllegalStateException (duplicate ICONV mapping, documented at :1186-1192); "can be thrown" is permissive; the .aff file is trusted configuration and is refused either way.

Why it fails: it limits how strongly one may say "must be ParseException", not whether this is a defect. Nothing makes an NPE from dereferencing readLine()'s null intended: the REP javadoc names ParseException for this very mistake, the same class already handles the identical case for PFX/SFX, and a blank line after the table already gives ParseException. The REP javadoc is itself loose for a REP count that is too low (NumberFormatException), so it shows the intent, not a complete list of exceptions. Hence "wrong error" at low priority: the case is for a descriptive parse error.

Two options were run through gradle in a scratch worktree of 645d4fe14a20; both cover the same six reads and change nothing else. B gives the better message.

A small helper readTableLine(reader, header) replaces the six reader.readLine() arguments (REP, MAP, CHECKCOMPOUNDPATTERN, COMPOUNDRULE, BREAK, ICONV/OCONV); parseCompoundRules and parseConversions get the header line as a new parameter. 26 lines added, 10 removed, all in Dictionary.java.

+  private static String readTableLine(LineNumberReader reader, String header)
+      throws IOException, ParseException {
+    String line = reader.readLine();
+    if (line == null) {
+      throw new ParseException("Premature end of rules for " + header, reader.getLineNumber());
+    }
+    return line;
+  }

   // REP, and the same at MAP, CHECKCOMPOUNDPATTERN, COMPOUNDRULE, BREAK, ICONV/OCONV:
-  String[] parts = splitBySpace(reader, reader.readLine(), 3, Integer.MAX_VALUE);
+  String[] parts =
+      splitBySpace(reader, readTableLine(reader, line), 3, Integer.MAX_VALUE);

Trade-offs

• The load still fails, now with the same exception and wording as PFX/SFX: ParseException: Premature end of rules for REP 2 (and ... for MAP 2, ... for ICONV 2, ...), which names the table and carries a line number. HunspellStemFilterFactory then wraps it in its IOException("Unable to load hunspell data! [dictionary=...,affix=...]"), which names the affix file (the dictionary= part prints the stream objects, not file names).
• It touches six call sites and two private signatures, so it conflicts textually with the open refactor PR #16422.
• The reported offset is the reader's line number at end of file: 2 for REP 2 / REP a b with and without a final newline on JDK 25. #10121 (LUCENE-9079, 2019) records that an early-access JDK of the time numbered an unterminated last line differently, so do not assert an exact offset.
• No API or format change; the only added work is one null check per table line while the file loads (not measured).

What the checks validated

• Gradle on 645d4fe14a20 (assertions off, the seed's default, and with -Ptests.asserts=true) and standalone (-ea and -da): without a fix the repro fails and the finder's 7-test class fails 6 of 7; with fix B the repro passes and the 7-test class passes 7/7.
• A probe of all seven directives gives ParseException: Premature end of rules for <header> (offset 2) for REP, MAP, ICONV, OCONV, COMPOUNDRULE, BREAK and CHECKCOMPOUNDPATTERN; the SFX control and the blank-line case are unchanged (gradle, and standalone with -ea and -da).
• Closest suites: every test class of the hunspell package (TestDictionary with testInvalidData, TestHunspellStemFilterFactory, TestSpellChecking, TestStemmer, ...): 49 classes including the repro, the probe and the 7-test class, 167 tests, 0 failures, 13 skipped (three classes that need external dictionaries or corpora).
• ./gradlew :lucene:analysis:common:check -x test passes with fix B and the repro in place (google-java-format, ecj lint, forbidden APIs, license and checksum checks) once the repro is reformatted by ./gradlew tidy (two lines wrapped); unformatted, only checkGoogleJavaFormat objects. The full Lucene test suite was not run.

The study's original fix: splitBySpace throws ParseException("Premature end of the affix file", line) for a null line, which covers REP, MAP, COMPOUNDRULE, BREAK and ICONV/OCONV (they all reach it), and the CHECKCOMPOUNDPATTERN loop checks readLine() itself. 9 lines added, 1 removed.

   private String[] splitBySpace(LineNumberReader reader, String line, int minParts, int maxParts)
       throws ParseException {
+    if (line == null) {
+      throw new ParseException("Premature end of the affix file", reader.getLineNumber());
+    }
     String[] parts = line.split("\\s+");

   // CHECKCOMPOUNDPATTERN does not go through splitBySpace:
+  String patternLine = reader.readLine();
+  if (patternLine == null) {
+    throw new ParseException("Premature end of the affix file", reader.getLineNumber());
+  }
-  new CheckCompoundPattern(reader.readLine(), flagParsingStrategy, this)
+  new CheckCompoundPattern(patternLine, flagParsingStrategy, this)

Trade-offs

• Smallest change, and it also guards any future caller of splitBySpace.
• The message does not name the table (Premature end of the affix file, with a line number), unlike PFX/SFX; the six tables give the same message.

What the checks validated

• Gradle, same worktree (assertions off): the repro passes, as it did earlier standalone with -ea and -da; the 7-test class passes 7/7; the probe gives Premature end of the affix file (offset 2) for all seven directives; the hunspell package runs 49 classes, 167 tests, 0 failures, 13 skipped.
• check -x test with the study's diff fails only in checkGoogleJavaFormat (its CHECKCOMPOUNDPATTERN throw fits on one line); the snippet above is the tidied form.
• Earlier, shadowed into the stock jars: 74 upstream hunspell tests at the study's commit and 145 at 645d4fe14a20, 0 failures.

Method: non-formal (static analysis, then executed tests). On 2026-10-26 a hunt unit ran SpotBugs 4.10.5 (-effort:max -low) over the non-sandbox module jars. Its NP_DEREFERENCE_OF_READLINE_VALUE check flagged Dictionary.parseConversions and two reads in readAffixFile (REP and CHECKCOMPOUNDPATTERN). A grep for readLine() passed straight as a call argument, over all non-sandbox modules, added the COMPOUNDRULE, BREAK and MAP reads in the same file. A 7-test JUnit through the public constructor (six truncated tables plus a truncated-SFX control) failed 6 of 7; independent checkers then reproduced it with their own tests, and it was cut to the one-test REP repro.