Hunspell COMPLEXPREFIXES accepts a second prefix that the inner prefix rule does not allow

Hunspell's COMPLEXPREFIXES option allows two prefixes on one word. As with two suffixes, the outer affix may only follow an inner affix rule that lists the outer affix's flag in its continuation class (the flags after the slash). With PFX A 0 re, PFX C 0 un, PFX C 0 dis/A and the root foo/C, redisfoo is a word and reunfoo is not. Since 9.5.0 Lucene no longer checks the inner rule: after stripping re, Stemmer.isAffixCompatible accepts an inner prefix of a cross-product class (one that passes the compound checks) without reading its continuation class (Stemmer.java:541, 566 on main a9779ef3c193), and what is left of the chain check is dictionary-wide: whether some prefix rule carries A. So Hunspell.spell("reunfoo") is true, getRoots returns [foo] and HunspellStemFilter indexes the token as foo, silently. Native Hunspell 1.3.2 and 1.7.2 reject reunfoo, and so does Lucene from 4.8.0 to 9.4.2 (11 of those releases run, the rest inferred from the unchanged check). The check was lost in the 2022 refactor #975 (LUCENE-10626), which reached 9.x in 9.5.0 and kept the per-rule check for twofold suffixes. Every release from 9.5.0 to 10.6.0 (10.6.0 by reading) and main is affected; reach is narrow, because it needs a custom dictionary with COMPLEXPREFIXES and continuation classes on prefix rules.

• Continuation class: flags after a slash on an affix rule. PFX C 0 dis/A . means: a word that got the prefix dis may take a prefix of class A next. hunspell(4)/(5) calls the suffix version "twofold suffix stripping": "A second s' suffix (affix class Y) will be the continuation class of the suffix able'", and adds that Hunspell "can strip another one yet (or a plus prefix in COMPLEXPREFIXES mode)".
• COMPLEXPREFIXES: "Set twofold prefix stripping (but single suffix stripping) eg. for morphologically complex languages with right-to-left writing system".
• Outer / inner prefix (Lucene's naming): the outer prefix is written first and stripped first (re in re+un+foo); the inner prefix is next to the root (un).

SET UTF-8
COMPLEXPREFIXES

PFX A Y 1
PFX A 0 re .

PFX C Y 2
PFX C 0 un .
PFX C 0 dis/A .

with the entry foo/C. Expected words: foo, unfoo, disfoo, redisfoo. Not words: refoo (the root does not carry A) and reunfoo (the rule un does not carry A).

1. Dictionary collects every flag that appears in the continuation class of any PFX rule (Dictionary.java:738-740, :482). Its javadoc describes it as a pre-filter: "If an outer affix's flag isn't here, there's no need to do 2-level affix stripping with it" (Dictionary.java:138-141). dis/A puts A in it.
2. Stemmer.applyAffix strips re and, under COMPLEXPREFIXES, looks for a second prefix because isSecondStagePrefix(A) is true (Stemmer.java:626).
3. For the candidate un, isAffixCompatible is called with outerPrefix = re (:424):

boolean previousWasPrefix = outerSuffix < 0 && outerPrefix >= 0;      // Stemmer.java:541
...
if (dictionary.isCrossProduct(affix)) {                              // :564
  if (previousWasPrefix) return true;                                // :566, also taken for a 2nd prefix
  if (outerSuffix >= 0) {
    char prevFlag = dictionary.affixData(outerSuffix, Dictionary.AFFIX_FLAG);
    return dictionary.hasFlag(append, prevFlag);                     // :567-569, suffixes only
  }
}

previousWasPrefix is meant for "a suffix after a prefix", which needs no continuation requirement, but it is also true when the affix being checked is itself a prefix, so for a cross-product inner prefix (class C is Y) un's continuation flags are never read.

1. The candidate foo carries un's flag C (:269-270). The root-level check of the outer prefix's flag runs only when there is no inner prefix (:271-276), so nothing rejects reunfoo.
2. Before #975 the prefix loop passed false for the previousWasPrefix parameter and the method ended in return previousWasPrefix || dictionary.hasFlag(append, prevFlag);, so a second prefix had to carry the outer flag (the 9.4.2 code). #975 derived previousWasPrefix from affix ids instead.
3. Native Hunspell stores prefix rules as internal suffixes under COMPLEXPREFIXES, so a twofold prefix gets the same per-rule continuation check as a twofold suffix (Hunspell 1.7.2 affixmgr.cxx, SfxEntry::checkword).

The trigger is wider than one class: un may sit in a class of its own, and a root that itself carries A (foo/AC) does not make reunfoo valid in native Hunspell while Lucene accepts it. It is enough that some prefix rule anywhere carries the outer flag.

• Who. Users of Hunspell.spell, getRoots, analyzeSimpleWord and HunspellStemFilter / HunspellStemFilterFactory (@lucene.experimental) with a dictionary that sets COMPLEXPREFIXES and gives some prefix rules continuation classes, where an inner prefix rule lacks a flag that another prefix rule carries (both prefix classes cross-product). Nothing else to configure; deterministic; assertions on or off.
• What goes wrong. No exception. The spell checker accepts misspellings such as reunfoo; stemming returns a root for a non-word, so the stem filter indexes and queries it as the root. In a probe index with the documents foo, reunfoo and bar, the query foo matched 2 documents on main and 1 on 9.4.2, and the query string reunfoo parsed to f:foo.
• Reach. Narrow. Of 191 surveyed public .aff files (99 LibreOffice, 92 wooorm/dictionaries) only one (Central Kurdish, ckb) sets COMPLEXPREFIXES, and none of its 1,812 prefix rules has a continuation class, so the second-prefix stage is never entered (the finder's survey; the ckb structure was re-checked; the 92 wooorm files were not kept, so the full survey cannot be re-run). Custom dictionaries for languages with complex prefixing are the likely users.
• In the hunt's differential over 4,000 generated micro-dictionaries, words Lucene accepts and native Hunspell 1.7.2 rejects fell from 49 to 16 with only this fix (finder's run).
• Since when. 9.5.0 (January 2023) on 9.x and 10.0.0 on 10.x, through 10.6.0 (10.6.0 by reading) and main.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: spell("reunfoo") expected false, was true
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertFalse(Assert.java:65)
       org.apache.lucene.analysis.hunspell.TestComplexPrefixesSecondPrefix.testSecondPrefixNeedsContinuationClass(TestComplexPrefixesSecondPrefix.java:43)

• The format's contract. hunspell(4) as shipped with Hunspell 1.3.2 (hunspell(5) later): COMPLEXPREFIXES "Set twofold prefix stripping (but single suffix stripping)"; "Twofold suffix stripping": "Hunspell can strip another one yet (or a plus prefix in COMPLEXPREFIXES mode)" and "A second s' suffix (affix class Y) will be the continuation class of the suffix able'". Native Hunspell 1.3.2 (run for this doc) and 1.7.2 accept unfoo, disfoo, redisfoo and reject refoo and reunfoo.
• Lucene's own design record. CHANGES for LUCENE-5483 (4.8.0): "Multi-stage affix-stripping, prefix-suffix dependencies, and COMPLEXPREFIXES now work correctly according to the hunspell algorithm. ... rules for recursive affix application are driven correctly by continuation classes in the affix file." A committer on LUCENE-5484 (#6547): "You should never recurse, unless the affix actually allows for it with a continuation class". The javadoc #975 deleted: "if we are removing a suffix, and it has no continuation requirements, it's ok. but two prefixes (COMPLEXPREFIXES) or two suffixes must have continuation requirements to recurse."
• Lucene claims native behaviour. Hunspell "can be used in place of native Hunspell for many languages" (its listed exceptions are Hungarian, non-BMP characters and PHONE); Stemmer "conforms to the algorithm in the original hunspell algorithm".
• Lucene disagrees with itself. The twofold-suffix path keeps the per-rule check (fooxs is rejected for SFX C 0 x, SFX C 0 y/A, SFX A 0 s), and getAllWordForms, documented as "equivalent to \"unmunch\"", returns [disfoo, foo, redisfoo, unfoo] while spell("reunfoo") is true.
• The change was not meant to change behaviour. #975 ("add tools to aid dictionary editing") has no CHANGES or MIGRATE note about stemming; its test changes add tests and fixtures, none about prefix chains or continuation classes (its one rewritten assertion, in TestDictionary, keeps its value); none of its 3 comments, 12 reviews and 19 inline comments mentions Stemmer, prefixes or continuation classes. 9.4.2 is correct.
• Why tests never caught it. TestComplexPrefix has one rule per prefix class, so it cannot tell "this rule allows it" from "some rule allows it"; no other Lucene fixture sets COMPLEXPREFIXES.

The strongest case for "intended" (the devil's-advocate verifier and the cross-model reviewer's intended lens): the Hunspell javadoc concedes incomplete language support, the class-wide second-stage flag set (LUCENE-9799/9803, #10838/#10842) is deliberate, and over-accepting is low-harm for stemming, so the dictionary-wide check could be a permissive approximation.

Why it fails: Dictionary documents that set as a pre-filter ("no need to do 2-level affix stripping"), and LUCENE-9799/9803 introduced it as pruning while the per-rule check in isAffixCompatible stayed; the per-rule check then survived until a refactor that never mentions it; the twofold-suffix twin keeps it; Lucene's own generator disagrees with its spell checker; and the unsupported-language list is about language-specific logic, while this is generic affix semantics shown by a seven-line dictionary. Every check rejected the argument; the ones that rated its strength (the cross-model reviews and the adjudication) rated it weak. Verdict: a defect, a regression since 9.5.0.

Option A was applied in a scratch worktree at a9779ef3c193 and run through gradle, then shadowed onto the stock jars for the hunt's tests. Option B (A plus LFV-375's fix) was run through gradle and the classpath runner too. Option C is a sketch only.

Stemmer.java +7/-1. A suffix after a prefix keeps return true; a prefix after a prefix must carry the outer prefix's flag in its continuation class, as before 9.5.0 and as a second suffix must. A prefix only reaches this branch with outerSuffix < 0 (after a suffix Lucene never looks for a prefix, Stemmer.java:630-647), so the new branch covers exactly "prefix after prefix". For the PR, extend the existing fixture: PFX B 0 pthree . in complexprefix.aff (class B becomes PFX B Y 2) and assertStemsTo("pthreefoo", "foo"); assertStemsTo("ponepthreefoo"); in TestComplexPrefix, or add the repro.

--- Stemmer.isAffixCompatible (Stemmer.java:564)
     if (dictionary.isCrossProduct(affix)) {
       // cross-check incoming continuation class (flag of previous affix) against this affix's flags
-      if (previousWasPrefix) return true;
+      if (previousWasPrefix) {
+        // a suffix after a prefix needs no continuation requirement, but a second prefix
+        // (COMPLEXPREFIXES) must carry the outer prefix's flag, as a second suffix must
+        if (!isPrefix) return true;
+        char prevFlag = dictionary.affixData(outerPrefix, Dictionary.AFFIX_FLAG);
+        return dictionary.hasFlag(append, prevFlag);
+      }
       if (outerSuffix >= 0) {

Trade-offs

• Speed: not measured (no benchmark run); on the second-prefix path only, an affixData read and a hasFlag lookup replace the immediate return true.
• Behaviour change: dictionaries that relied on the 9.5-10.5 over-acceptance lose those words, which native Hunspell never accepted. HunspellStemFilter stops stemming them, so an index built with an affected version keeps such words as the root term and needs reindexing for exact agreement.
• Leaves LFV-375 in place: with A alone, LFV-375's repro still fails 3 of 4 with StackOverflowError; with only LFV-375's fix, this repro still fails 2 of 2. Also unchanged: a second prefix whose class has cross product N is still rejected (native accepts it; noted with LFV-641's fix).

What the checks validated

• Stock main a9779ef3c193, gradle: the repro fails 2 of 2 and the extended TestComplexPrefix 1 of 1, with assertions on and off.
• With A, gradle: the repro and TestComplexPrefix pass with assertions on and off; the whole hunspell test package (org.apache.lucene.analysis.hunspell.*, assertions on, repro and extended fixture included) runs 47 classes, 160 tests, 0 failures, 13 skipped (TestAllDictionaries, TestHunspellRepositoryTestCases and TestPerformance need external dictionaries); that run used the repro before one statement was rewrapped for google-java-format, same assertions; ecjLint, checkGoogleJavaFormat, forbiddenApisMain and forbiddenApisTest pass with the fix and both tests in place; fix.diff and fix-with-test.diff apply cleanly to a9779ef3c193.
• With A shadowed onto the stock jars (classpath runner): the repro (-ea, -da), the finder's test (1 of 1), the independent reproducer's test (4 of 4, -ea and -da), the cross-model review's variants (2 of 2), the intended-behaviour check's prefix-versus-suffix test (2 of 2) and the cross-model reviewers' tests (4 of 4, 1 of 1) pass.
• Not run: the full Lucene suite, the external-dictionary suites, a benchmark.

A and doPrefix = false at Stemmer.java:640 ("we took away the second prefix: go look for another suffix"), as before 9.5.0. Fits one issue, "the #975 refactor broke COMPLEXPREFIXES twofold-prefix handling", with both repros.

Trade-offs

Same as A, plus LFV-375's own behaviour change (no third prefix under COMPLEXPREFIXES). Reviewers may prefer two small PRs; each fix leaves the other bug in place.

What the checks validated

• Classpath runner: this repro 2 of 2 and LFV-375's repro 4 of 4 pass.
• Gradle: A plus LFV-375's line, with both repros in the hunspell package: 48 classes, 164 tests, 0 failures, 13 skipped.

In processStemCandidate, when innerPrefix >= 0, require isFlagAppendedByAffix(innerPrefix, flag of outerPrefix) next to the existing outer-prefix check (Stemmer.java:271).

Trade-offs

Strips first and rejects later (more work per candidate), and leaves isAffixCompatible treating prefix and suffix chains differently. Not recommended. Not run.

Method: cross-implementation differential testing (the hunt's non-formal tier): Lucene's getAllWordForms, spell and analyzeSimpleWord against native Hunspell 1.7.2 (libhunspell spell and stem) and the hunspell-tools unmunch/munch.

• Inputs: 4,000 seeded, grammar-built micro-dictionaries (per dictionary every candidate word reachable by at most 3 affix rules) and 1,069 per-root cases from the Hunspell 1.7.2 and Lucene test fixtures.
• Root causes were separated with switchable shadow fixes: with only this fix, words Lucene accepts and native rejects fell from 49 to 16.
• First observation: on 2026-10-26 the intended-behaviour verifier of LFV-375 read the #975 change and ran a targeted probe (spell("cax") true on main, 10.5.2 and 9.5.0, false on 9.4.0 and in native Hunspell); it was kept as an unregistered sibling note until this hunt found it independently.
• Native oracle: use native spell(), not stem(), which returns nothing for prefixed words under COMPLEXPREFIXES.