Hunspell Dictionary cannot load SET ISO8859-10, and loads ISO8859-14 only when spelled exactly so

A Hunspell affix file names its character encoding on a SET line; the Hunspell manual lists "UTF-8, ISO8859-1 - ISO8859-10, ISO8859-13 - ISO8859-15, KOI8-R, KOI8-U, cp1251, ISCII-DEVANAGARI". Lucene turns the value into a decoder in Dictionary.getDecoder (Dictionary.java:899-908 on main 645d4fe14a20): its own ISO8859_14Decoder for the exact string ISO8859-14, two exact aliases, otherwise Charset.forName(value). The JDK has no ISO-8859-10 or ISO-8859-14 charset under any of its 963 names and aliases, so SET ISO8859-10 (every spelling tested), SET ISO-8859-14 and SET iso8859-14 make the public constructor throw java.nio.charset.UnsupportedCharsetException, and the dictionary does not load. C Hunspell lower-cases the value, keeps only letters and digits, and has tables for both. The failure is loud and at load time; only the opt-in Hunspell module is involved, and no surveyed public dictionary uses these values. Every release with Hunspell, 3.5.0 to 10.6.0, and main is affected; since 4.8.0 the exact ISO8859-14 loads.

1. The public constructors (:189, :206, :226) end in the last one, which reads the affix prologue and calls readConfig(...) (:260).
2. readConfig (:837) passes the SET argument on: decoder = getDecoder(singleArgument(reader, line)); (:849). The second pass repeats the lookup in its consistency check (:465-470).
3. CHARSET_ALIASES (:889-890) has two exact keys: microsoft-cp1251 -> windows-1251 and TIS620-2533 -> TIS-620.
4. getDecoder (:899-908): if ("ISO8859-14".equals(encoding)) return new ISO8859_14Decoder(); (exact, case-sensitive), then the alias map, then replacingDecoder(Charset.forName(encoding)) (:907). Its javadoc (:893-894): "Note, This isn't perfect as I think ISCII-DEVANAGARI and MICROSOFT-CP1251 etc are allowed...".
5. ISO8859_14Decoder.java:25: "many hunspell dictionaries use this encoding, yet java does not have it?!?!". There is no ISO-8859-10 counterpart.
6. Charset.forName resolves ISO-8859-2, ISO8859-2 and iso8859-2 through JDK aliases. JDK 25.0.3 has 173 charsets with 963 names and aliases, and none normalises to iso885910 or iso885914; Charset.isSupported is false for ISO8859-10, ISO-8859-10, ISO8859-14 and ISO-8859-14.

In Hunspell's normal non-Mozilla build (the code from #ifndef MOZILLA_CLIENT at :682; a separate Mozilla implementation starts at the #else at :2349), get_current_cs (:2326) runs toAsciiLowerAndRemoveNonAlphanumeric (:2305, called at :2328), then looks the result up in encds[] (:2279), which has iso885910 (:2289) and iso885914 (:2294). An unknown name is not an error in C: it warns and falls back to ISO-8859-1 (:2340-2345).

Over 32 spellings per value (upper or lower case, crossed with no separator, -, _ or . in two places) on the stock main jar: ISO8859-10 loads in 0, ISO8859-14 in 1 (the exact one), ISO8859-1 to 9, 13 and 15 in 8 to 10 each. Every spelling that loads decodes every letter byte as the unicode.org tables say (10,374 checks, 0 mismatches). With the fix all 64 spellings load (17,746 checks, 0 mismatches).

• Who is hit. Applications whose affix file says SET ISO8859-10 (Latin-6, Nordic) in any tested spelling, or writes ISO-8859-14 (Latin-8, Celtic) as anything but exactly ISO8859-14. The exception comes where the affix file is read: new Dictionary(...) or HunspellStemFilterFactory.inform. HunspellStemFilter and the Hunspell spell-checker take a built Dictionary, so they simply never get one. No default analyzer uses Hunspell.
• What happens. The constructor throws java.nio.charset.UnsupportedCharsetException naming the SET value. No wrong results, no index damage, nothing persisted, no scale needed. The constructor javadoc names only IOException and ParseException; through HunspellStemFilterFactory the exception arrives unwrapped. The hunt's impact class "crash" means exactly this, an unchecked exception on input the API accepts, here at load time; the JVM keeps running.
• How likely. Rare. The finder's survey of the two dictionary collections the package javadoc recommends found 0 of 191 dictionaries using these values (recorded only as the finder's summary; not re-checked). Hunspell's own test data writes a hyphenated SET ISO-8859-15 (1.7.2 tests/encoding.aff), which loads in Lucene only because the JDK has that name (the canonical name of its ISO-8859-15 charset).
• Workaround. Convert the dictionary to UTF-8 and write SET UTF-8, or, for ISO-8859-14, write exactly ISO8859-14.

Copy the test into lucene/analysis/common/src/test/org/apache/lucene/analysis/hunspell/ of a Lucene checkout and run the gradle command. With the fix it passes unchanged; its assertTrue after loading only matters on a fixed build, where it checks that byte 0xBF decodes as ISO-8859-10.

The seed only picks the framework's locale, timezone and assertion mode (DEADBEEF: en-BW, Asia/Irkutsk, assertions off); seeds DEADBEEF and 1234ABCD give the same result.

Executed: gradle on a fresh 645d4fe14a20 worktree (fails); the standalone runner on main (-ea, -da, two seeds) and on the 10.6.0, 10.5.2 and 9.1.0 release jars (fails, same message, -ea and -da). The ISO-8859-14 and iso8859-14 spellings are left out on purpose: same path, same exception (see the table under "The finding in 30 seconds").

cp $REPRO/TestSetEncodingIso8859_10.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/hunspell/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.hunspell.TestSetEncodingIso8859_10 -Ptests.seed=DEADBEEF

# gradle, fresh main 645d4fe14a20 worktree + the test, -Ptests.seed=DEADBEEF (2026-10-30)
TestSetEncodingIso8859_10 > testSetIso8859_10Loads FAILED
    java.lang.AssertionError: expected SET ISO8859-10 to load, got java.nio.charset.UnsupportedCharsetException: ISO8859-10
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1C956C176F957FE7]:0)
        at org.apache.lucene.analysis.hunspell.TestSetEncodingIso8859_10.testSetIso8859_10Loads(TestSetEncodingIso8859_10.java:40)

        Caused by:
        java.nio.charset.UnsupportedCharsetException: ISO8859-10
            at java.base/java.nio.charset.Charset.forName(Charset.java:534)
            at org.apache.lucene.analysis.hunspell.Dictionary.getDecoder(Dictionary.java:907)
            at org.apache.lucene.analysis.hunspell.Dictionary.readConfig(Dictionary.java:849)
            at org.apache.lucene.analysis.hunspell.Dictionary.<init>(Dictionary.java:260)
            at org.apache.lucene.analysis.hunspell.Dictionary.<init>(Dictionary.java:213)
            at org.apache.lucene.analysis.hunspell.Dictionary.<init>(Dictionary.java:192)
            at org.apache.lucene.analysis.hunspell.TestSetEncodingIso8859_10.testSetIso8859_10Loads(TestSetEncodingIso8859_10.java:32)
:lucene:analysis:common:test (FAILURE): 1 test, 1  failure

# standalone runner, seed DEADBEEF (the 9.1.0 -da run: seed FC928A08)
main 645d4fe14a20  -ea / -da   tests_run=1 failures=1   same message and cause chain
10.6.0, 9.1.0      -ea / -da   tests_run=1 failures=1   same message
10.5.2             -ea / -da   tests_run=1 failures=1   same message (getDecoder:919 there)

• Contract. The constructors read "hunspell affix and dictionary files" and document only IOException and ParseException (Dictionary.java:178-188, :195-205, :216-225). The package is "A Java implementation of Hunspell stemming and spell-checking algorithms" (package-info.java:19-20, link markup dropped). No javadoc, test or CHANGES entry limits SET to JDK charset names.
• The manual. hunspell(5) lists ISO8859-10 among the possible SET values (1.3.4 and 1.7.2), and C Hunspell resolves it to its own table after normalising the name.
• Lucene adds missing tables on purpose. ISO8859_14Decoder exists because "many hunspell dictionaries use this encoding, yet java does not have it", but only one exact spelling reaches it.
• Precedent. SOLR-2934 (a 2012 comment) reported the same UnsupportedCharsetException for SET TIS620-2533 through the same lookup; LUCENE-5468 (#6531, 2014) fixed that value with the CHARSET_ALIASES entry.
• Tests. No upstream test exercises ISO8859_14Decoder or asserts that a SET name is rejected; the one test aimed at parsing the SET value (TestDictionary.testSetWithCrazyWhitespaceAndBOMs) checks whitespace and BOM handling for UTF-8.

The strongest case for "works as designed" (rated moderate for ISO8859-10, moderate to strong for the ISO8859-14 spellings): the getDecoder javadoc has admitted since 2011 that SET support "isn't perfect"; the supported set is JDK charsets plus a small table built from real dictionaries (LUCENE-5468's CHANGES entry: the filter "loads all known openoffice dictionaries without error"); the manual writes part 14 only as ISO8859-n, which works; LUCENE-9687 scopes C parity to some languages; failing loudly is safer than C's silent ISO-8859-1 fallback; and in SOLR-2934 the maintainer (Robert Muir) replied the same day that there was "no point in worrying about adding aliases/matching up charset naming for Thai", because the filter would be a no-op with that word-list-only dictionary (the alias came two years later).

Why it fails: the javadoc is hedged ("I think ... etc") and names ISCII-DEVANAGARI and MICROSOFT-CP1251, not these values; ISO8859-10 is a documented value that the public constructor accepts; nothing limits SET names; and the one-spelling special case was already narrower than C Hunspell's normalisation when it was written. Concede that the compatibility contract is implied (by the manual and by constructors that read "hunspell affix and dictionary files"), not written down. Honest caveat: C Hunspell's own ISO-8859-10 table has no case data for the high bytes (for 0xBF it stores {0x00, 0xbf, 0xbf}), so C loads such a dictionary but does not case-map its non-ASCII letters. Verdict: a long-standing compatibility gap with a small fix, not a regression.

Three options. The first was validated with gradle in a fresh scratch worktree at 645d4fe14a20; the other two were not executed. C Hunspell's behaviour for unknown names (fall back to ISO-8859-1) is not an option: it would load the file but misread the 46 of the 96 bytes from 0xA0 up where ISO-8859-10 and Latin-1 differ (0xBF would read as ¿, not ŋ).

In Dictionary.getDecoder, compute encoding.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9]", ""); map iso885914 to ISO8859_14Decoder and iso885910 to a new ISO8859_10Decoder (bytes 0xA0-0xFF from the unicode.org 8859-10 mapping, built like ISO8859_14Decoder). The sketch also maps normalised microsoftcp1251 and tis6202533 onto the two existing alias keys, so 12 spellings of those two that fail today (7 for windows-1251, 5 for TIS-620) load too; drop those lines for the narrowest change. Everything else keeps going to the alias map and Charset.forName unchanged. Dictionary.java +13/-1, new ISO8859_10Decoder.java (59 lines).

// C Hunspell matches SET values lower-cased and without non-alphanumerics
// (csutil.cxx get_current_cs)
String normalized = encoding.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9]", "");
if ("iso885914".equals(normalized)) {
  return new ISO8859_14Decoder();
}
if ("iso885910".equals(normalized)) {
  return new ISO8859_10Decoder();
}

Trade-offs

• One more hand-written table to maintain; its 96 entries match an independent ISO-8859-10 codec (96 of 96).
• Like ISO8859_14Decoder, the new decoder reports ISO-8859-1 as its charset(), so it inherits LFV-932 (FLAG UTF-8 with an 8-bit SET merges flags); that bug has its own fix site.
• Over 1,457 names (all JDK names and aliases plus generated spellings), every name that loads today still loads with the same charset and decoder class, and 75 names that fail today load.
• Open PR #16422 moves the SET call sites into a switch but leaves getDecoder alone; the two should not conflict. The 2011 "isn't perfect" javadoc could be updated in the same change.

What the checks validated

• Gradle, fresh scratch worktree at 645d4fe14a20, JDK 25, -Ptests.seed=DEADBEEF (2026-10-30): without the fix the minimal test fails; with fix.diff applied it passes unchanged, and the whole org.apache.lucene.analysis.hunspell test package passes with the seed's assertion mode (off) and with -Ptests.asserts=true: 47 suites, 159 tests, 13 skipped, 0 failures each time. The skips (TestAllDictionaries 2, TestPerformance 10, TestHunspellRepositoryTestCases 1) need external data passed as system properties. :lucene:analysis:common:checkGoogleJavaFormat passes. The rest of the module's tests were not run.
• Standalone runner with the fix shadowed: the minimal test passes (-ea, -da), the supplementary ISO8859-14 spelling test passes 4 of 4 (-ea; no -da run of it with the fix is recorded), and the model's jar check loads all 64 spellings with 0 decode mismatches (17,746 checks).
• An independent reviewer's own 4-test JUnit passes 4 of 4 with it (-ea, -da).

Add if ("ISO8859-10".equals(encoding)) return new ISO8859_10Decoder(); and compare ISO8859-14 case-insensitively. Not executed.

Trade-offs

Fixes the lead case with fewer lines, but ISO-8859-14 and ISO-8859-10 still fail, and the special cases stay inconsistent with the JDK-alias behaviour of the other ISO-8859 parts.

Catch the charset exception in getDecoder and throw a ParseException that names the SET value. Not executed.

Trade-offs

Gives the documented exception type and a clearer message, but the dictionary still does not load: a complement to the first option, not a fix.

Method: a differential model checked on the real jar (the hunt's "semi-formal" tier; the first signal was the probe, the model came after it).

• First signal. A planning probe loaded a one-word-per-letter-byte dictionary through the stock main jar for every SET value hunspell(5) lists plus 8 spelling variants, and checked every letter byte against the standard mapping tables. It reported ISO8859-10, ISO-8859-10, ISO-8859-14 and iso8859-14 as rejected (and the javadoc-acknowledged ISCII-DEVANAGARI and MICROSOFT-CP1251, which this bug leaves out); every other listed value loaded with 0 misdecoded letters.
• Model. The finder wrote a model of SET-name acceptance: C Hunspell 1.7.2's name normalisation and encoding table against Lucene's getDecoder, over the JDK 25 charset registry's 927 case-insensitively distinct names and aliases (963 counted raw). ISO-8859-10 and ISO-8859-14 have no JDK spelling at all.
• Check on the jar. 536 spellings through the public constructor: 113 load, 0 disagreements with the model, 10,374 decode checks with 0 mismatches against the unicode.org tables. The finder reduced it to a public-API JUnit; the hunt later minimised that to the one-test repro.