Hunspell FLAG UTF-8 with an 8-bit SET other than ISO8859-1 merges affix classes, so spell() and the stemmer accept forms the dictionary forbids

After a first pass finds the SET and FLAG lines, Lucene's Hunspell Dictionary reads the whole .aff and the .dic through the SET decoder before it parses flags. For FLAG UTF-8 (flags are UTF-8 characters) it undoes that decoding only when the decoder's charset is ISO-8859-1 (LUCENE-9708); every other 8-bit SET that loads gets one flag per decoded character, except ISO8859-14, whose decoder claims to be ISO-8859-1. A two-byte flag such as á (C3 A1) therefore becomes two characters; an affix rule keeps only the first, the lead byte shared by all of U+00C0-U+00FF, while a word keeps both. Affix classes whose flags share a lead byte merge: spell() accepts and the stemmer stems forms the dictionary forbids, with no error. On main this affects 12 of 18 SET values (ISO8859-2, -4, -5, -7, -9, -13, -14, -15, KOI8-R, KOI8-U, microsoft-cp1251, TIS620-2533); ISO8859-3, -6 and -8 fail to load with UnmappableCharacterException. C Hunspell 1.3.2 and 1.7.2 decode such flags from the raw bytes and are right for all 18. All 13 releases run, from 3.5.0 to 10.5.2, and 10.6.0 (run on branch_10_6 at its release revision) mishandle these files (3.5.0-4.7.2 and 8.9.0-10.6.0 silently, 4.8.0-8.8.2 with a load error), and the source history places the releases in between in the same eras. No public dictionary is known to use the combination. A byte-table decoder fixes all three observed symptoms in the probe and the tests, and the module's 1,917 tests pass with it.

• .aff / .dic: a Hunspell dictionary is an affix file (rules such as SFX á 0 a .: "class á adds the suffix a") and a word list (kot/á: the word kot may take the classes named by flag á).
• SET: the encoding of the files' words, for example ISO8859-2 (Latin-2) or KOI8-R. hunspell(5): "Set character encoding of words and morphemes in affix and dictionary files."
• FLAG UTF-8: each flag is one Unicode character written in UTF-8. hunspell(5): "`UTF-8' parameter sets UTF-8 encoded Unicode character flags." C Hunspell decodes these flags from the raw bytes whatever the SET is.
• Lead byte: the first byte of a multi-byte UTF-8 character. All of U+00C0-U+00FF start with C3, all of U+0400-U+043F with D0.

1. A first pass finds the SET and FLAG lines (Dictionary.java:245-260, :837-868); then Dictionary reads the whole .aff and the .dic through a reader that decodes the SET charset (:348, :1015), so flags arrive as SET-decoded characters.
2. getFlagParsingStrategy (Dictionary.java:922-944) gives FLAG UTF-8 the strategy that turns those characters back into bytes and reads them as UTF-8 (DefaultAsUtf8FlagParsingStrategy) only if the decoder's charset equals ISO-8859-1 (:932). Every other 8-bit SET that loads gets SimpleFlagParsingStrategy (:935): one flag per decoded character, except ISO8859-14 (step 5).
3. An affix rule takes one flag with parseFlag, which returns the first character and ignores the rest because checkFlags = false (:1356, :1364-1370). A word takes all characters (:1130).
4. So with SET ISO8859-2, á (C3 A1) reads as ĂĄ and â (C3 A2) as â: both rules become class Ă, and the words carry [Ă, Ą] and [Ă, ˘]. Each word now takes both suffixes.
5. ISO8859-14 is read by ISO8859_14Decoder, which reports ISO-8859-1 as its charset (ISO8859_14Decoder.java:45), so it takes the ISO-8859-1 branch; re-encoding turns the bytes its table remaps into ?, and both rules get U+FFFD. Under ISO8859-14 flags merge even when their lead bytes differ: á (C3 A1) and ġ (C4 A1) stay apart under ISO8859-2 and -15 but merge under ISO8859-14 (executed, assertions on and off).
6. A flag byte that the SET leaves unassigned (C3 in ISO8859-3 and -8, A1 in ISO8859-6 and -8) makes the reader throw UnmappableCharacterException while the line is read (:350), because the decoder replaces only malformed input (:910-912).

ISO8859-15 decodes C3, A1 and A2 exactly like ISO8859-1 and still merges: only the charset's identity picks the strategy. Under ISO8859-2 and the 10 other SETs that take SimpleFlagParsingStrategy, a merge needs two flags with the same lead byte; under ISO8859-14 it needs flags whose bytes its table remaps, whatever their lead bytes. The 18 values are a selection: hunspell(5) also lists ISO8859-10, which Lucene cannot load at all (sibling bug LFV-928), and ISCII-DEVANAGARI, which was not tested.

• Who is exposed. Any code that loads such a dictionary through the public Dictionary constructors: Hunspell spell checking, HunspellStemFilter, and HunspellStemFilterFactory, which builds the same Dictionary (by reading; not run). Within the module no setting is needed: the two directives alone trigger it.
• What the damage looks like. Silent: affix classes merge, so spell() accepts forbidden forms and the stemmer returns extra stems (for example kote → kot), which then become index and query terms. For ISO8859-3, -6 and -8 (with these flags) the dictionary does not load at all, with UnmappableCharacterException: Input length = 1 and no file or line.
• What it takes. Under ISO8859-2 and the 10 other SETs that read one flag per character: two flags with the same UTF-8 lead byte, which is common for non-ASCII flags (U+00C0-U+00FF share C3, U+0400-U+043F share D0). Under ISO8859-14: flags with bytes its decoder remaps, even with different lead bytes. Deterministic: no scale, randomness or concurrency; assertions on and off behave the same.
• How likely. Rare. The finder found 0 of 191 public dictionaries with this combination (not re-checked); in a 99-file LibreOffice snapshot FLAG UTF-8 appears only with SET UTF-8 (24 files) or with no SET (pt_BR, the case LUCENE-9708 handled). Mixing an 8-bit encoding with UTF-8 flags is a real pattern, though: pt_BR does it with the default encoding.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected kote to be rejected (kot has only flag U+00E1, suffix e needs U+00E2), but spell("kote") returned true
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertFalse(Assert.java:65)
       org.apache.lucene.analysis.hunspell.TestFlagUtf8With8BitSet.testUtf8FlagsSharingLeadByteStayDistinct(TestFlagUtf8With8BitSet.java:41)

• Contract. hunspell(5): SET is the encoding "of words and morphemes in affix and dictionary files"; FLAG UTF-8 "sets UTF-8 encoded Unicode character flags". C Hunspell's decode_flags and decode_flag (case FLAG_UNI) call u8_u16 on the flag bytes with no SET check; C Hunspell 1.3.2 and 1.7.2 give the expected verdicts for all 18 SET values on byte-identical files.
• Lucene's own intent. LUCENE-9708 (7a7949aed2a, PR apache/lucene-solr#2270): title "Hunspell: support FLAG UTF-8 in absence of SET UTF-8", body "This is a valid situation to be supported". (Its first javadoc said "Used for FLAG UTF-8 on non-UTF-8 encoding", but that code was already ISO-8859-1-only and the wording was narrowed in review to match, so it is weak evidence.) The ISO-8859-1 limit follows from the re-encode trick (getBytes(ISO_8859_1) restores the bytes only for ISO-8859-1).
• Javadoc. Hunspell "can be used in place of native Hunspell for many languages" and lists what is unsupported (Hungarian, non-BMP characters, PHONE); this combination is not listed. SimpleFlagParsingStrategy says it "Can be used with both the ASCII and UTF-8 flag types", true only for UTF-8 text. Lucene neither rejects nor warns.
• Tests. TestDictionary.testUtf8Flag covers FLAG UTF-8 only with the default charset, testFlagWithCrazyWhitespace only with UTF-8; flagutf8.aff has no SET line. No test pins the merged behaviour.
• Behaviour varies by accident. Correct for ISO8859-1, silently wrong for ISO8859-15, which decodes these bytes identically, and a load failure for ISO8859-3.
• Precedent. LUCENE-5824 (#6886) fixed FLAG long flags collapsing, with the same overgeneration symptom, as a bug.

The strongest "intended" case (the devil's-advocate verifier; the intended-lens review by another model family rated it moderate): LUCENE-9708 scoped UTF-8 flags to the default encoding on purpose (its private javadoc was narrowed in review to "even if the rest of the file is in the default (8-bit) encoding"), and SET can be read as the encoding of the whole file, flags included.

Why it fails: hunspell(5) restricts SET to words and morphemes and defines UTF-8 flags without a SET condition; C Hunspell decodes them from bytes for every SET; the same PR calls FLAG UTF-8 without SET UTF-8 "a valid situation to be supported"; the ISO-8859-1 limit is a property of the round-trip trick; no javadoc, test, CHANGES or MIGRATE entry excludes other SETs; and the outcome flips between ISO8859-1 and ISO8859-15 for identical bytes. Verdict: a defect (adjudication: confirmed, high). What the counter-case gets right: hunspell(5) and C Hunspell are strong compatibility evidence, not a Lucene API guarantee; the first-commit javadoc ("on non-UTF-8 encoding") sat on code that was already ISO-8859-1-only, so it says little; and no public dictionary is known to use the combination, so this is a low-priority correctness fix.

Option A (smaller) fixes the silent merge and the ISO8859-14 case but not the ISO8859-3/-6/-8 load failure; option B fixes all three in the probe and the tests. Both change the same method as open PR #16422 (coordinate with it), which rewrites getFlagParsingStrategy as a switch with the same ISO-8859-1-only test. Whichever is chosen, add the FLAG UTF-8 + 8-bit SET cases to the hunspell tests.

A new package-private ByteTableDecoder (70 lines) decodes each byte through a 256-entry table built once from the SET decoder (a byte the SET leaves unassigned becomes U+F700 + byte instead of an error) and maps decoded text back to its bytes. getFlagParsingStrategy uses it for FLAG UTF-8 with a single-byte SET other than plain ISO-8859-1 (ISO8859-14 included); readConfig then reads the .aff and .dic through it; the new EightBitAsUtf8FlagParsingStrategy decodes the recovered flag bytes as UTF-8. +107/-3 in two files.

} else if ("UTF-8".equals(flagType)) {
  if (decoder instanceof ByteTableDecoder byteTable) {
    return new EightBitAsUtf8FlagParsingStrategy(byteTable);
  }
  Charset charset = decoder.charset();
  if (DEFAULT_CHARSET.equals(charset) && !(decoder instanceof ISO8859_14Decoder)) {
    return new DefaultAsUtf8FlagParsingStrategy();
  }
  if (isSingleByte(charset)) {
    return new EightBitAsUtf8FlagParsingStrategy(new ByteTableDecoder(decoder));
  }
  return new SimpleFlagParsingStrategy();
}

Trade-offs

• Only dictionaries with FLAG UTF-8 and a single-byte SET other than ISO-8859-1 take the new path; no SET, ISO-8859-1 and UTF-8 dictionaries are untouched. Multi-byte non-UTF-8 charsets (not Hunspell SET values) keep today's behaviour.
• In that configuration the whole .aff and .dic, words included, are decoded through the new table (built once from the SET decoder) instead of that decoder, so it needs compatibility tests on real dictionaries beyond these synthetic flags and words. A word byte that the SET leaves unassigned now loads as a private-use character instead of failing the load (unassigned bytes inside words were not tested). The table would collide only if a SET charset mapped a byte into U+F700-U+F7FF; none of the 17 single-byte charsets checked in Python's codec tables does, and the 15 of them that the JDK provides were also checked in Java 25 (no byte in that range, no duplicate chars); other Java charsets that pass the single-byte gate were not checked.
• One map lookup per flag character, at load time only. Larger than option A (a new class plus a decoder swap in readConfig).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test fails 7 of 8 on stock (assertions on and off) and passes 8 of 8 with the fix; org.apache.lucene.analysis.hunspell.*: 47 suites, 166 tests (13 skipped: they need external dictionaries), 0 failures, assertions on (seed 5EED0932) and off (seed 5EED0933); the whole :lucene:analysis:common:test: 340 suites, 1,917 tests, 18 skipped, 0 failures; :lucene:analysis:common:check -x test passes (ecjLint, forbiddenApis, google-java-format, licenses).
• Standalone runner with the fix shadowed: the new test 8 of 8 (-ea, -da), the finder's test 5 of 5, the independent reproducer's 11 of 11; the fix probe gives the expected flags and verdicts for all 12 SET values it runs (ISO8859-3, -6, -8 and -14 included); not shown for other single-byte Java charsets or for real dictionaries.

For FLAG UTF-8 with any non-UTF-8 SET, a new strategy rebuilds the bytes with rawFlags.getBytes(setCharset) (an inverse table ISO8859_14Decoder.encode for ISO8859-14) and decodes them as UTF-8; the decoder is unchanged. +55/-3 in two files.

if (DEFAULT_CHARSET.equals(charset) && !(decoder instanceof ISO8859_14Decoder)) {
  return new DefaultAsUtf8FlagParsingStrategy();
}
if (!StandardCharsets.UTF_8.equals(charset)) {
  return new EightBitAsUtf8FlagParsingStrategy(decoder); // getBytes(SET charset), then UTF-8
}

Trade-offs

Smaller and does not change how words are decoded. Leaves the ISO8859-3/-6/-8 load failure, because those lines fail to decode before flags are parsed. A one-method variant without the ISO8859_14Decoder change cut the independent reproducer's failures from 7 to 2 (ISO8859-14 and ISO8859-3 remained).

What the checks validated

Gradle, hunspell package: 166 tests, 1 failure (the new testSetIso8859_3); the format check passes after re-wrapping one over-long comment. Standalone: the new test 7 of 8, the independent reproducer's 10 of 11 (its ISO8859-3 test fails); earlier the finder's test 4 of 5 and 146 hunspell tests green.

Throw a ParseException naming the FLAG and SET lines when FLAG UTF-8 meets a SET the code cannot handle. Not run.

Trade-offs

Turns the silent merge into a loud failure, but refuses files that C Hunspell reads correctly, and is no simpler than option A.

Method: differential testing against a reference implementation (the study's "semi-formal" tier), a side leg of a hunt unit that compared Lucene's handling of Hunspell SET encodings with C Hunspell, 2026-10-28.

• A probe built one .aff/.dic pair (FLAG UTF-8, classes U+00E1 and U+00E2) and varied only the SET line over 18 values: none, ISO8859-1 to -9, -13, -14, -15, KOI8-R, KOI8-U, microsoft-cp1251, TIS620-2533 and UTF-8. Lucene was right for 3, merged 12 and failed to load 3; C Hunspell 1.3.2 (the command-line tool) and 1.7.2 (the library, raw-byte driver) were right for all 18.
• Code reading of getFlagParsingStrategy explained the split; an AI finder then wrote a 5-test JUnit (public API plus the package-private Stemmer), and independent AI tools reproduced, reviewed and judged it (see the checks).