Hunspell spell() ignores KEEPCASE, FORCEUCASE, NEEDAFFIX, FORBIDDENWORD and COMPOUNDFORBIDFLAG on COMPOUNDRULE compound parts

A Hunspell dictionary can build compound words with compound flags (COMPOUNDFLAG, COMPOUNDBEGIN, ...) or with COMPOUNDRULE patterns over flags. Its entries can also carry restrictions: KEEPCASE, FORCEUCASE, NEEDAFFIX, FORBIDDENWORD and COMPOUNDFORBIDFLAG. Lucene's Hunspell.spell applies these restrictions to the parts of flag compounds but not to the parts of rule compounds: checkCompoundRules (Hunspell.java:437-464 on main a9779ef3c193) takes every non-final part straight from dictionary.lookupWord (:445), and checkLastCompoundPart (:475-493) never applies the FORCEUCASE check. So spell() accepts compounds that C Hunspell 1.7.2 rejects (1.3.2 too, except the COMPOUNDFORBIDFLAG case): on the public Swedish dictionary sv_SE, lower-case rule compounds such as tolvkronorsgatan; on small dictionaries, a KEEPCASE first part in capitalized or upper case and a NEEDAFFIX, FORBIDDENWORD or COMPOUNDFORBIDFLAG entry as a first part (NEEDAFFIX also as a middle part). suggest() offers such words as corrections (run for the KEEPCASE and FORCEUCASE cases). Nothing throws, assertions on and off give the same result, and it fails on all 7 releases run from 8.9.0, the first with the spell-checking API, to 10.5.2; the main-line history shows only refactors of this code in between. Stemming (HunspellStemFilter, getRoots) is not.

• Flag compound: a compound accepted through COMPOUNDFLAG or COMPOUNDBEGIN / COMPOUNDMIDDLE / COMPOUNDEND: every part carries a compound flag.
• Rule compound: a compound accepted because a COMPOUNDRULE pattern over flags matches its parts, for example COMPOUNDRULE GH (a G word, then an H word) or GH* (then any number of H words).
• Per-entry flags (hunspell(5) of C Hunspell 1.7.2): KEEPCASE "Forbid uppercased and capitalized forms of words signed with KEEPCASE flags"; FORCEUCASE "Last word part of a compound with flag FORCEUCASE forces capitalization of the whole compound word" (the manual's example is the Dutch "straat", street); NEEDAFFIX "virtual stems in the dictionary, words only valid when affixed. Except, if the dictionary word has a homonym or a zero affix"; FORBIDDENWORD "forbidden word form"; COMPOUNDFORBIDFLAG "Dictionary words with this flag are removed from the beginning and middle of compound words".

1. The word as typed matches nothing (Foo is not in the dictionary), so spell retries the lower-case variant foobar and remembers that the original was title case (Hunspell.java:95-110 on main a9779ef3c193).
2. checkCompounds (:150-161) tries the rule path first, without the original case: checkCompoundRules(wordChars, 0, length, new ArrayList<>()) (:152). Only the flag path gets it (:157).
3. checkCompoundRules (:437-464) splits the word and takes the first part's entries straight from the dictionary, IntsRef forms = dictionary.lookupWord(wordChars, offset, breakPos); (:445); middle parts come from the same line in the recursion (:454). Nothing looks at KEEPCASE, NEEDAFFIX, FORBIDDENWORD or COMPOUNDFORBIDFLAG.
4. checkLastCompoundPart (:475-493) looks the last part up through Stemmer.doStem (:490), which skips NEEDAFFIX entries (Stemmer.java:234-236), but its callback (:481-489) only matches the rules and never calls hasForceUCaseProblem. GH matches, so spell returns true.

The FORBIDDENWORD and COMPOUNDFORBIDFLAG middle-part rows were also executed (fact check: foobarbaz with bar/HW or bar/HF is accepted by stock Lucene and rejected by C 1.7.2 and by the fix).

C 1.3.2 agrees except COMPOUNDFORBIDFLAG, which it ignores on dictionary words in both modes. The rule path is also inconsistent with itself: in the GH* dictionary the NEEDAFFIX entry bar is rejected as the last part (foobazbar false) and accepted in the middle (foobarbaz true).

Real dictionary (sv_SE). FORCEUCASE c and the rule 4?5?4?23U*Y; 13 entries carry both c and Y, the dictionary's COMPOUNDEND flag, which its rules reuse as their last flag (afrika, danmark, europa, finland, gatan, gränd, holmen, norge, quist, qvist, själland, sverige, tyskland); gatan/AcYZ also has Z = ONLYINCOMPOUND. Lucene accepts tolvkronorsgatan and treveckorssverige; C 1.3.2 and 1.7.2 reject both and accept the capitalized forms, which Lucene accepts too; both reject the flag compound storgatan. Lower-case compounds ending in three of the 13 were run (afrika, gatan, sverige); the other ten were not. These words are built from the dictionary's rules, not taken from a corpus.

Not part of the claim (C and Lucene agree): KEEPCASE on a middle or last rule part, and FORBIDDENWORD on the last rule part, are accepted by both. The claim is per-flag parity with C Hunspell, not "rule and flag compounds always agree".

• Who is hit. Code that calls Hunspell.spell or Hunspell.suggest (lucene-analysis-common) with a dictionary that uses COMPOUNDRULE and whose rule-flagged entries also carry KEEPCASE, FORCEUCASE, NEEDAFFIX, FORBIDDENWORD or COMPOUNDFORBIDFLAG. The spell checker is opt-in, with a dictionary the application chooses; no default analyzer uses it.
• What they get. spell says "correct" for words the dictionary forbids; suggest can offer them as corrections (Foobarr gives [Foobar], C 1.7.2 suggests foobar; tolvkronorsgaatan gives [tolvkronorsgatan], C suggests nothing). Only these KEEPCASE and FORCEUCASE cases were run for suggest(). Silent, deterministic, the same with assertions on and off; no scale needed.
• Real dictionary. sv_SE has 13 entries that are both FORCEUCASE and rule-final (gatan, gränd, holmen and place names such as sverige). Lower-case numeral rule compounds ending in three of them (gatan, sverige, afrika) were run and are accepted; the other ten were not run. How often such words occur in real text was not measured. Two more public dictionaries declare some of these flags next to COMPOUNDRULE; whether their rule entries carry them was not checked: Galician gl_ES (LibreOffice collection: NEEDAFFIX, KEEPCASE, 3 rules) and Dutch nl (wooorm/dictionaries collection: KEEPCASE, FORCEUCASE, FORBIDDENWORD, 37 numeral rules; the LibreOffice nl_NL copy has no COMPOUNDRULE).
• Not affected. HunspellStemFilter and getRoots: "No roots are returned for compound words" (Hunspell.java:300), so index-time and query-time analysis do not change.
• Severity. A spell-check verdict error in an opt-in component: silent wrong results, likelihood rare.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: spell("foobar") expected false, was true
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertFalse(Assert.java:65)
       org.apache.lucene.analysis.hunspell.TestCompoundRuleForceUCase.testForceUCaseLastRulePart(TestCompoundRuleForceUCase.java:36)

• The format. hunspell(5) of C Hunspell 1.7.2 (man/hunspell.5): KEEPCASE (:561-566) "Forbid uppercased and capitalized forms of words signed with KEEPCASE flags"; FORCEUCASE (:496-500) "Last word part of a compound with flag FORCEUCASE forces capitalization of the whole compound word"; NEEDAFFIX (:582-584) "words only valid when affixed. Except, if the dictionary word has a homonym or a zero affix" (an unflagged homonym is a valid word; C 1.7.2, doStem and the fix skip only the flagged homonym); COMPOUNDFORBIDFLAG (:452-455) "Dictionary words with this flag are removed from the beginning and middle of compound words". None is limited to flag compounds. FORBIDDENWORD (:552-556, "This flag signs forbidden word form. Because affixed forms are also forbidden, we can subtract a subset from set of the accepted affixed and compound words") is the weakest: the text forbids whole word forms, compounds included, but does not name compound parts, so it rests on C's behaviour and Lucene's own flag path.
• Lucene's documentation (Hunspell-rule semantics, broad compatibility). "This class can be used in place of native Hunspell for many languages for spell-checking and suggesting purposes" (Hunspell.java:35-36); its list of unsupported cases (Hungarian, non-BMP characters, PHONE; :39-44) does not cover this. spell returns "whether the given word's spelling is considered correct according to Hunspell rules" (:72).
• C Hunspell. Two 1.7.2 builds reject every claimed word on the same dictionaries, and 1.3.2 every one except the COMPOUNDFORBIDFLAG case, which 1.3.2 ignores on dictionary words in both kinds of compound (executed). In 1.7.2's compound_check, rule mode skips NEEDAFFIX homonyms, breaks on a FORBIDDENWORD or NEEDAFFIX first word, skips COMPOUNDFORBIDFLAG words and checks FORCEUCASE before returning a rule match; spell() applies KEEPCASE to the returned first part (read). Nuspell's check_compound_with_rules has the same checks (read only).
• Lucene's own tests. keepcase.wrong lists Unitvv and UNITVV (a KEEPCASE first part of a flag compound) and forceucase.wrong lists foobaz. No fixture combines COMPOUNDRULE with KEEPCASE, FORCEUCASE or NEEDAFFIX.
• History. LUCENE-9684 (d7968130c3f, apache/lucene-solr#2228) added the raw lookup; its PR says "I've reimplemented Hunspell's logic". LUCENE-9710 (9d45dfe776a) implemented FORCEUCASE on the flag path only; LUCENE-9776 (4b3fb1e0659) moved only the last rule part to doStem, for inflection; LUCENE-9785 (31a64927a46, "Like Hunspell does") kept the case check on the first part of flag compounds. No commit, test or comment says rule compounds skip these checks.

Strongest case, rated weak by every reviewer: hunspell(5) Note III says "COMPOUNDRULE flags work completely separately from the compounding mechanisms using COMPOUNDFLAG, COMPOUNDBEGIN, etc. compound flags. (Use these flags on different entries for words)." Lucene documents compatibility only "for many languages", no fixture pins this behaviour, and sv_SE reuses its COMPOUNDMIDDLE/COMPOUNDEND flags as rule flags against that note.

Why it fails: Note III is about which compound flags a rule matches, not about per-entry flags such as KEEPCASE or NEEDAFFIX, and FORCEUCASE is defined for any compound. C Hunspell 1.7.2 (and 1.3.2 for all but COMPOUNDFORBIDFLAG) and Nuspell apply the checks in rule mode, Lucene's rule path already applies NEEDAFFIX to the last part, and the synthetic repros reuse no flags and still fail, so it is not a misuse. A note in the manual's BREAK entry ("COMPOUNDRULE handles only the suffixation of the last word part") explains why non-final parts are not affix-stripped, not why their entry flags are ignored; the recommended fix keeps the raw lookup. Verdict: a defect.

One fix (A, the finder's) was applied in a scratch worktree at a9779ef3c193 and run through gradle, and shadowed onto the stock jars for every other repro of the hunt. B and C are alternatives that were not run.

Hunspell.java only, +43/-7. checkCompounds passes originalCase to checkCompoundRules (and on through the recursion). A new compoundRulePartForms wraps the raw lookupWord for non-final parts and drops each entry (homonym) with NEEDAFFIX, FORBIDDENWORD or COMPOUNDFORBIDFLAG on its own and, for the first part, entries acceptCase refuses for the word's original case (KEEPCASE). The checkLastCompoundPart callback skips readings for which hasForceUCaseProblem holds (new int entryId overload). The raw lookup stays, so non-final parts are still never affix-stripped.

--- checkCompounds (Hunspell.java:152)
-        && checkCompoundRules(wordChars, 0, length, new ArrayList<>())) {
+        && checkCompoundRules(wordChars, 0, length, new ArrayList<>(), originalCase)) {
--- checkCompoundRules (Hunspell.java:445)
-      IntsRef forms = dictionary.lookupWord(wordChars, offset, breakPos);
+      IntsRef forms =
+          compoundRulePartForms(wordChars, offset, breakPos, words.isEmpty() ? originalCase : null);
--- new: compoundRulePartForms keeps an entry of lookupWord(...) only if
+      !hasFlag(entryId, needaffix) && !hasFlag(entryId, forbiddenword)
+          && !hasFlag(entryId, compoundForbid)
+          && acceptCase(originalCase, entryId, new CharsRef(wordChars, offset, length))
--- checkLastCompoundPart callback (Hunspell.java:481)
+          if (hasForceUCaseProblem(formID, originalCase, wordChars)) {
+            return true;
+          }

Trade-offs

• One extra pass over the homonyms of each candidate part and a small IntsRef per matched part; no benchmark was run.
• Homonyms need a stated rule. The fix filters every entry on its own. C 1.7.2 decides by the homonym listed first in the .dic (affixmgr.cxx:1671-1696, :1747-1756) and Lucene's flag path by the first homonym in Lucene's sorted entry order (Stemmer.java:237-240, Hunspell.java:163-184, :232). In a probe on 32 small dictionaries (foo/G plus a flagged foo), the fix accepts foobar/Foobar with a COMPOUNDFORBIDFLAG, FORBIDDENWORD or KEEPCASE homonym in every order; C rejects them when the flagged homonym is listed first, and the flag path when it sorts first. NEEDAFFIX homonyms are skipped one by one everywhere. Homonym order already separates Lucene from C on plain words (FORBIDDENWORD foo), so the PR should state its rule and add a homonym test; fix.diff's helper javadoc holds for single entries only.
• acceptCase also drops Lucene's hidden case-variant entries (the capitalized copies Lucene adds for mixed-case words and for all-caps words with flags, Dictionary.java:1062-1064, like Hunspell's ONLYUPCASE entries) from middle rule parts, and from the first part unless the word is title or upper case, as the flag path already does (Hunspell.java:166, :186-198); no test covers that case.
• Forbidden compounds stop being accepted and suggested. A dictionary that relied on the old behaviour relied on something C Hunspell does not do.
• For the PR, the repro can become fixture files in Lucene's style (.aff / .dic / .good / .wrong, run by TestSpellChecking), one per flag, plus a .sug case.

What the checks validated

• Stock main a9779ef3c193, gradle: the repro fails 6 of 6 with assertions on and off.
• With the fix, gradle: the repro passes 6 of 6 with assertions on and off; the whole lucene/analysis/common test task with assertions on, repro included, runs 340 classes, 1,915 tests, 0 failures, 18 skipped (the hunspell package: 47 classes, 164 tests, 13 skipped, because TestAllDictionaries, TestHunspellRepositoryTestCases and TestPerformance need external dictionaries); ecjLint, checkGoogleJavaFormat and forbiddenApis pass with the test in place.
• With the fix shadowed onto the stock jars: the finder's test 4 of 4, the independent reproducer's test 5 of 5, the second cross-model reviewer's normative test 4 of 4, the credibility audit's 11-dictionary differential (all 99 words agree with C 1.7.2), the sv_SE words (all 8 agree with C 1.7.2) and the suggest() probe (suggestions now match C 1.7.2's). In the hunt, Lucene's 146 hunspell tests and a sweep over 25 upstream compound fixture families pass with the fix.
• Not run: the full Lucene suite, the external-dictionary suites (TestAllDictionaries, TestHunspellRepositoryTestCases), a benchmark.

Use COMPOUND_BEGIN / COMPOUND_MIDDLE contexts so NEEDAFFIX and COMPOUNDFORBIDFLAG are filtered by the stemmer.

Trade-offs

Not run. doStem also strips affixes, so first and middle parts would start matching inflected forms, which hunspell(5) says COMPOUNDRULE does not handle ("handles only the suffixation of the last word part"); it would need a roots-only mode and still needs the case and FORBIDDENWORD checks.

Only the checkLastCompoundPart hunk of the recommended fix: the one manifestation with real-dictionary evidence (sv_SE).

Trade-offs

Not run. Smallest change, but leaves the KEEPCASE, NEEDAFFIX, FORBIDDENWORD and COMPOUNDFORBIDFLAG manifestations.

Method: cross-implementation differential testing (the hunt's non-formal tier).

• Inputs: seeded, grammar-generated synthetic dictionaries with one or two compounding directives (COMPOUNDFLAG, COMPOUNDBEGIN/END, COMPOUNDRULE) plus modifiers such as KEEPCASE, FORCEUCASE, NEEDAFFIX, FORBIDDENWORD and COMPOUNDFORBIDFLAG, 1-4 short roots, and every short word over the roots' letters plus 2-3-part concatenations.
• Lucene's spell() and getRoots() (public API, stock jar) were compared with C Hunspell 1.7.2 built from the pristine source; every disagreement was re-checked on a Linux distribution build of 1.7.2.
• Greedy one-element-at-a-time minimization of the disagreeing dictionaries gave KEEPCASE (4 dictionaries), FORCEUCASE, NEEDAFFIX, FORBIDDENWORD and COMPOUNDFORBIDFLAG (4) clusters, all on the same code path; the root cause was then read in Hunspell.checkCompoundRules.
• Found 2026-10-27 on main a9779ef3c193; then checked by an independent reproducer, an intended-behaviour devil's advocate (which showed that hunspell(5) documents COMPOUNDFORBIDFLAG for dictionary words, making it the fifth asserted manifestation, and found the sv_SE case), two executed reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a history and prior-report follow-up, a security screen and a credibility audit. This doc added the compact JUnit, the gradle runs, the fix validation, C runs on the repro's dictionaries and the suggest() probe.