Hunspell rejects and cannot stem twofold-suffix words when a suffix class has cross product N

In a Hunspell .aff file a suffix can carry a continuation class that allows a second suffix after it ("twofold suffix stripping"). Each affix class header also has a cross-product field, Y or N, which hunspell(5) defines as the "permission to combine prefixes and suffixes". Lucene also requires it for suffix + suffix: if either suffix class of a chain says N, Hunspell.spell rejects the word, getRoots, analyzeSimpleWord and Stemmer return no stem, HunspellStemFilter leaves the token unstemmed, and getAllWordForms leaves the form out. Native Hunspell (1.3.2 and 1.7.2, in every Y/N variant tested) accepts these words. Three code sites carry the assumption (Stemmer.java:621, Stemmer.java:564, WordFormGenerator.java:251 on main a9779ef3c193). On the real Luxembourgish dictionary, from the collection Lucene's package docs point to, at least 47,840 word forms of 13,526 distinct roots that native Hunspell 1.3.2 accepts are rejected by Lucene main. All 18 releases tested from 3.5.0 to 10.5.2 fail, and the same gates are in every release since 3.5.0; nothing throws.

• Affix class: a group of PFX or SFX rules under one flag. Its header line is SFX <flag> <cross product> <count>, for example SFX E N 2.
• Cross product (Y or N): hunspell(5) field (2), "Cross product (permission to combine prefixes and suffixes)" (Hunspell 1.7.2 man/hunspell.5:522).
• Continuation class: flags after a slash on a rule, SFX S 0 st/E .: a word that got st may take a suffix of class E next. hunspell(5) calls this "Twofold suffix stripping" (:921); its example is drink/X with SFX X 0 able/Y . and SFX Y 0 s ., giving drink, drinkable, drinkables.
• Outer / inner suffix (Lucene's naming): the outer suffix is the last one applied and the first one stripped (en in raich + st + en), the inner suffix the first one applied (st).

1. Dictionary.java:685 parses the field per class (crossProduct = args[2].equals("Y")), :791 stores it with every rule and isCrossProduct(affix) reads it back (:812-814).
2. Outer suffix N: no search for the inner suffix. Stemmer.applyAffix strips an affix, checks the stem, then decides whether to strip one more. The decision is gated on the affix just stripped: if (dictionary.isCrossProduct(affix) && recursionDepth <= 1) (Stemmer.java:621); for a suffix it then looks for "another suffix" (:630-634). With E = N, after en is stripped from raichsten, the search for st never starts.
3. Inner suffix N: the continuation check is never reached. Stemmer.isAffixCompatible accepts an inner suffix only inside if (dictionary.isCrossProduct(affix)) { ... return dictionary.hasFlag(append, prevFlag); } (:564-571) and otherwise returns false (:573). With S = N: after en is stripped, the intermediate raichst is looked up (:262) and is not an entry; then this check refuses to strip st as the inner suffix, so the root raich is never reached, although st/E carries the continuation flag.
4. The generator copies the rule. WordFormGenerator.expand recurses for a further affix only if (dictionary.isCrossProduct(affix.id)) (WordFormGenerator.java:251-253), so getAllWordForms omits the same forms.
5. Native Hunspell checks cross product only when a prefix is combined: SfxEntry::check_twosfx skips an entry only if ((optflags & aeXPRODUCT) != 0 && (opts & aeXPRODUCT) == 0) (Hunspell 1.7.2 affentry.cxx:712, "if this suffix is being cross checked with a prefix"), and without a prefix it checks the inner suffix with optflags 0 (:753-755). nuspell, an independent C++ implementation, does the same.

With the hunspell(5) example, only X = Y, Y = Y accepts drinkables; the other three Y/N combinations give spell = false, roots = [] and getAllWordForms("drink") = [drink, drinkable], while drinkable stays accepted and drinks rejected in all four. Native Hunspell 1.3.2 accepts drinkables and stems it to drink in all four. For prefix + suffix words Lucene already agrees with native Hunspell 1.3.2 on every Y/N combination of a 4-dictionary matrix with a prefix class (S and E each Y or N, the prefix class tied to S); only the suffix chains differ.

• Who. Users of Hunspell.spell, suggest, getRoots, analyzeSimpleWord, HunspellStemFilter / HunspellStemFilterFactory and getAllWordForms whose dictionary gives cross product N to a class in a twofold-suffix chain. No setting beyond the dictionary; deterministic; assertions on or off.
• What goes wrong. No exception. Spell-checking flags correct words, and suggest may offer a different word (suggest("raichsten") = [raichen] in the hunt's probe). Stemming returns nothing, so the stem filter indexes and queries the surface form: in an 8-document probe index a query for raich found 4 of 6 forms, and a query for raichsten only itself. Every hit set was a subset of the correct one. getAllWordForms omits the forms.
• Scale on a real dictionary (Luxembourgish lb from wooorm/dictionaries at 8cfea406b505, the commit Lucene's regression build pins). generateAllSimpleWords over every entry gives 293,583 distinct forms on stock and 341,573 with the fix, none lost. Of the letters-only added forms, 47,840 of 13,526 distinct roots are accepted by native Hunspell 1.3.2 (the observed hunspell -l result), rejected by stock Lucene and accepted with the fix: 36,137 lowercase (for example superlatives and participles with adjective endings: räichsten, verstümmeltem, standardiséierten) and 11,703 capitalised (for example diminutive and feminine plurals: Hälsercher, Steenemerinne). Native -m output on the named forms shows two suffixes, for example verstümmeltem st:verstümmelen po:verb is:adjective is:papa is:em; morphology was not run on all 47,840.
• Other dictionaries. A static survey of 191 public .aff files found such chains in 8, among them lb, hu_HU, ta_IN, is and fa; only lb was executed. Lucene documents Hungarian as unsupported.
• Downstream. The hunspell token filters of Elasticsearch and OpenSearch return Lucene's HunspellStemFilter (source read, not executed).
• Since. Every release with Lucene's Hunspell code: stemming since 3.5.0, Hunspell.spell since 8.9.0, getAllWordForms since 9.5.0.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected spell("drinkables") to be true, was false
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.hunspell.TestTwofoldSuffixCrossProduct.testTwofoldSuffixWhenOuterClassHasCrossProductN(TestTwofoldSuffixCrossProduct.java:19)

• The format's contract. hunspell(5) (Hunspell 1.7.2 man/hunspell.5:522): "(2) Cross product (permission to combine prefixes and suffixes). Possible values: Y (yes) or N (no)". Its "Twofold suffix stripping" section (:921-962) ties the second suffix to the continuation class only. A native maintainer restated the rule on hunspell/hunspell#177 (2026): "A prefix and a suffix are only combined on the same word when both the prefix rule and the suffix rule are marked with the cross-product flag Y."
• Lucene claims that behaviour. Hunspell javadoc: "can be used in place of native Hunspell for many languages" (Hunspell.java:34-37; the listed exceptions are Hungarian, non-BMP characters and PHONE). Stemmer javadoc: "conforms to the algorithm in the original hunspell algorithm, including recursive suffix stripping" (Stemmer.java:28-32). CHANGES for LUCENE-5483 (4.8.0): "Multi-stage affix-stripping, prefix-suffix dependencies, and COMPLEXPREFIXES now work correctly according to the hunspell algorithm ... rules for recursive affix application are driven correctly by continuation classes in the affix file" (lucene/CHANGES.txt:12545-12549); its issue #6546 "greatly refined this algorithm to match correct hunspell behavior".
• No source we found documents the rule. The searches found no test, comment, CHANGES entry, issue or commit message saying N blocks suffix chains; no commit message in the full history contains "cross product" or "twofold".
• Why tests never caught it. 12 of Lucene's 97 test .aff files chain two suffixes (twofold.aff, the manual's example in TestTwoFold, and dependencies.aff, compress.aff, needaffix5.aff and others), all with Y on both classes; no Lucene fixture and no Hunspell 1.7.2 test fixture puts an N class in a continuation chain.
• Asymmetry. For prefix + suffix words Lucene already applies the cross-product rule exactly as native Hunspell does; only suffix + suffix uses it.

The strongest case for "intended": the gate is deliberate code from the very first Lucene Hunspell port (2011), LUCENE-5483 (2014) added the inner check, LUCENE-10626 (2022) copied it into the generator, every manual example uses Y, the Hunspell javadoc says not all languages are supported, and in 2012 the original author wrote on LUCENE-4311 (#5379) that "our implementation of Hunspell is not a port of the C/C++ app per se". So it could be read as a conservative supported subset.

Why it fails: no source gives a reason for the gate; the later, explicit intent (LUCENE-5483's issue and CHANGES entry, the Hunspell and Stemmer javadocs, LUCENE-9687 "on a par with the native C++ Hunspell") is native compatibility; the unsupported-language list is about language-specific logic, while this is generic affix semantics shown by a six-line dictionary; and keeping drinkable but dropping drinkables only because an unrelated flag says N is not a coherent subset. The cross-model reviewer rated the argument moderate in a lens and weak in the other; the independent reproducer and the devil's-advocate check rated it weak, and the adjudication found that none of the intended-behaviour arguments holds. Verdict: a defect. WordFormGenerator is the same misreading in a second place, not a separate defect; its javadoc promises "all word forms for all dictionary entries with the given root word" (WordFormGenerator.java:117), which the omission also contradicts.

One fix (A) was applied in a scratch worktree at a9779ef3c193 and run through gradle; it was also shadowed onto the stock jars for the hunt's tests and the real-dictionary probe. The finder's broader sketch (B) was run on the same tests for comparison.

Two files, +13/-9. After a suffix Lucene only ever looks for another suffix, never a prefix (Stemmer.java:630-647), so dropping the gate for suffixes changes only suffix chains. A suffix after a prefix still needs Y on both (unchanged), and a prefix + suffix + suffix chain keeps the check on the inner suffix. The generator keeps a form only if Lucene's (fixed) stemmer stems it back to the entry (canStemToOriginal, WordFormGenerator.java:248), a Lucene self-check, not a comparison with native Hunspell.

--- Stemmer.isAffixCompatible (Stemmer.java:564)
-    if (dictionary.isCrossProduct(affix)) {
-      if (previousWasPrefix) return true;
-      if (outerSuffix >= 0) {
-        char prevFlag = dictionary.affixData(outerSuffix, Dictionary.AFFIX_FLAG);
-        return dictionary.hasFlag(append, prevFlag);
-      }
+    if (previousWasPrefix) return dictionary.isCrossProduct(affix);
+    if (outerSuffix >= 0) {
+      char prevFlag = dictionary.affixData(outerSuffix, Dictionary.AFFIX_FLAG);
+      return dictionary.hasFlag(append, prevFlag)
+          && (outerPrefix < 0 || dictionary.isCrossProduct(affix));
     }
--- Stemmer.applyAffix (Stemmer.java:621)
-    if (dictionary.isCrossProduct(affix) && recursionDepth <= 1) {
+    if ((!prefix || dictionary.isCrossProduct(affix)) && recursionDepth <= 1) {
--- WordFormGenerator.expand (WordFormGenerator.java:251)
-            if (dictionary.isCrossProduct(affix.id)) {
+            if (dictionary.isCrossProduct(affix.id) || append.length > 0) {

Trade-offs

• Bounded extra work: suffix recursion still stops at two suffixes, and the generator at two affixes of a kind (WordFormGenerator.java:376).
• It leaves the mirror case unchanged: under COMPLEXPREFIXES, two prefixes whose classes say N (reunfoo) are accepted by native Hunspell and rejected by Lucene with or without this fix. That path is entangled with a separate prefix-recursion bug (option B) and is better fixed on its own.
• Compatibility: the fix changes analysis output. HunspellStemFilter now stems the affected words, so an index built before the fix keeps their surface forms and needs reindexing to match the fixed query-time analysis. A Lucene-only dictionary that relied on N to block a suffix chain (nonportable: native Hunspell ignores it there) would accept more words, and getAllWordForms returns more forms (on Luxembourgish 341,573 instead of 293,583); no benchmark was run.

What the checks validated

• Stock main a9779ef3c193, gradle: the repro fails 2 of 2 with assertions on and off.
• With the fix, gradle: the repro passes with assertions off; the whole lucene/analysis/common test task with assertions on, repro included, runs 340 classes, 1,911 tests, 0 failures, 18 skipped (the hunspell package: 47 classes, 160 tests, 13 skipped, because TestAllDictionaries, TestHunspellRepositoryTestCases and TestPerformance need external dictionaries). ecjLint, checkGoogleJavaFormat and forbiddenApis pass with the test in place; both diffs apply cleanly to a9779ef3c193.
• With the fix shadowed onto the stock jars: the finder's test, the independent reproducer's spell/stem and generator tests (4 of 4 each), the cross-model review's 4-dictionary prefix/suffix matrix (prefix + suffix results unchanged, now equal to native everywhere), the cross-model reviewers' tests (6 of 6, 2 of 2), and the hunt's security probe (no query loses hits; its COMPLEXPREFIXES N dictionary still does not overflow, as on stock, while its Y dictionary still overflows for bx and bbx, as on stock: that is the separate LFV-375).
• Real dictionary: on Luxembourgish the generator adds 47,990 forms and loses none; spell accepts all but 36 of them (hyphenated or special forms).
• Not run: the full Lucene suite, the external-dictionary suites, a benchmark.

Also lets a second prefix under COMPLEXPREFIXES combine without cross product, which fixes the mirror case (reunfoo accepted). It passes the repro and, by the finder's run, the 146 hunspell tests of the time.

Trade-offs

Alone it lets a separate known bug, the COMPLEXPREFIXES third-prefix recursion that ends in StackOverflowError (registry LFV-375), reach dictionaries whose chained prefix classes say N, which today are immune: in the hunt's security probe the COMPLEXPREFIXES dictionary with N overflows for bx with B, and not with A or on stock. In the finder's differential, words Lucene accepts and native rejects went from 49 to 96, all new ones under COMPLEXPREFIXES. Only safe together with or after the LFV-375 fix (doPrefix = false at Stemmer.java:640) and the sibling second-prefix continuation fix.

State in the Hunspell javadoc that suffix chains need cross product Y.

Trade-offs

Contradicts hunspell(5) and the class's "in place of native Hunspell" claim, and leaves real dictionaries such as Luxembourgish broken.

Method: cross-implementation differential testing (the hunt's non-formal tier), in a hunt unit comparing Lucene's word-form generator, spell and analyzeSimpleWord with native Hunspell 1.7.2 (libhunspell spell and stem) and the hunspell-tools unmunch/munch.

• Inputs: 4,000 seeded, grammar-built micro-dictionaries (per dictionary every candidate word reachable by at most 3 affix rules) and 1,069 per-root cases from the Hunspell 1.7.2 and Lucene test fixtures.
• Root causes were separated with switchable shadow fixes: with the finder's fix for this bug (option B) alone, words native accepts and Lucene rejects fell from 202 to 24 (the rest are other causes or native deviations from hunspell(5)).
• The finder then wrote a public-API JUnit (raich/SE) that fails on the stock jar, found the real Luxembourgish case and checked native Hunspell on it.
• Found 2026-10-27 on main a9779ef3c193; then checked by an independent reproducer, an intended-behaviour devil's advocate, two executed reviews by a model from another family (OpenAI GPT-5.6), a re-run of the intended lens, an adjudication, a history and prior-report follow-up, a security screen and a credibility audit. This doc added the gradle runs, the fix validation and the Luxembourgish count.