Hunspell getAllWordForms never generates a suffix that only a prefix allows

Hunspell lets a prefix rule enable a suffix through its continuation class: with PFX U 0 kilo/S ., SFX S 0 s . and the entry ampere/U, the dictionary defines ampere, amperes, kiloampere and kiloamperes, although the entry has no S. Lucene's spell-checker has accepted this since LUCENE-9688 (2021), and C Hunspell 1.7.2 accepts all four words. Lucene's word-form generator (Hunspell.getAllWordForms, WordFormGenerator.getAllWordForms, generateAllSimpleWords, compress, and through them NGramFragmentChecker.fromAllSimpleWords) never produces these forms: getAllWordForms("ampere") yields only the words ampere and kiloampere. After a prefix it recurses with the prefix's continuation flags, but the guard if (!isPrefix && !stem.getPrefixes().isEmpty()) return false; (WordFormGenerator.java:378 on main a9779ef3c193) rejects every suffix once a prefix is applied, so a prefix-plus-suffix form is built only suffix first, from a suffix flag the entry itself carries. No error is raised, and compress proposes needless edits that over-generate. In the current French dictionary getAllWordForms("ampère") returns 29 singular forms and none of the 25 plurals; over the whole dictionary a fix adds 833 forms, all accepted by spell(). Every release from 9.5.0 through 10.6.0 (10.6.0 by reading) and main is affected. Spell-checking, stemming, HunspellStemFilter, indexing and search do not use this code.

• Entry: a line of the .dic file. ampere/U has the stem ampere and the flag U.
• Affix rule: SFX S 0 s . says that entries with flag S take the suffix s (amperes); PFX U 0 kilo . adds the prefix kilo.
• Continuation class: the flags an affix gives to the word it creates. PFX U 0 kilo/S . gives S to kiloampere, so kiloamperes is a word even though ampere/U has no S. hunspell(5): "affix (optionally with flags of continuation classes, separated by a slash)".

1. expand (WordFormGenerator.java:233-259) walks the entry's flags. For each affix it checks isCompatibleWithPreviousAffixes (:240), applies it, keeps the word if stemming it reaches a stem with the entry's stem text and no forbidden word (canStemToOriginal, :184-217; it compares the stem string only, not the entry's flags), and recurses with updateFlags(flags, flag, append) (:251-252): the applied flag removed, the affix's continuation flags added.
2. isCompatibleWithPreviousAffixes (:372-383), line 378: if (!isPrefix && !stem.getPrefixes().isEmpty()) return false;. No suffix after a prefix, which fixes the order: suffixes first, prefixes second.
3. For ampere/U the flags are [U]. Applying kilo gives kiloampere and recurses with [S], the prefix's continuation. S is a suffix, a prefix is present, and line 378 rejects it: kiloamperes is never built. The empty prefix (PFX U 0 0/S) loses amperes the same way.
4. The other cases work. If the entry has S (work/PS), S is applied first and P second, and the guard only prevents building reworks twice. In the mirror case, a suffix whose continuation allows a prefix (SFX S 0 s/P, work/S), a prefix after a suffix is allowed, so reworks is generated.
5. The spell-checker accepts this exact path on purpose: Stemmer.java:269-270, dictionary.hasFlag(entryId, flag) || dictionary.isFlagAppendedByAffix(prefixId, flag) (LUCENE-9688).

The raw lists repeat a word: getAllWordForms("ampere") is [ampere, ampere, kiloampere] on main, because the empty prefix PFX U 0 0/S rebuilds ampere (likewise book and calorie). The table shows distinct words, as the repro compares them.

French dictionary (LibreOffice 4f1df64b2e8e): 29 unit-prefix rules PFX Um, 25 of them with continuation S. (the plural), on 35 entries such as ampère/Um. getAllWordForms("ampère") gives 29 distinct words on main, all singular, and 54 with the fix (plus ampères, kiloampères, d'ampères and 22 more). generateAllSimpleWords: 3,063,236 distinct words on main, 3,064,069 with the fix (+833, none removed); the number spell() rejects is 19,484 in both, so every added word is accepted.

• Who is hit. Callers of Hunspell.getAllWordForms and compress, both WordFormGenerator.getAllWordForms overloads, generateAllSimpleWords and compress, and NGramFragmentChecker.fromAllSimpleWords. The API exists "to aid dictionary editing" (LUCENE-10626): listing a dictionary's words, minimising entries, building fragment checkers. No setting is involved; which forms go missing depends only on the dictionary, deterministically.
• The trigger. A valid dictionary with a PFX rule whose continuation class carries an SFX flag the entry lacks. Lucene's own dependencies.aff has it (PFX p 0 0/s .). The current French dictionary has it on 35 unit entries: 833 missing forms, 25 of them for ampère. The finder's static survey reports the shape in 16 of 191 public .aff files (French, Danish, de_*_frami, Armenian, Bengali, Kinyarwanda); their 3,000-root samples changed only French, not da_DK, de_DE_frami, lb, nb_NO or sv_SE (not re-checked).
• What goes wrong. Incomplete getAllWordForms and generateAllSimpleWords output, with no error. compress proposes edits the dictionary does not need (ampere/SU for words ampere/U already covers; work/PS for work/P), and the work/PS edit adds works, which it lists in getExtraGeneratedWords(). A fragment checker from fromAllSimpleWords lacks n-grams that only the missing forms have: in the hunt's probe it flagged amperes and kiloamperes as containing an impossible fragment. That checker is opt-in (Suggester.withFragmentChecker; Hunspell.suggest does not use one), and no changed suggestion was seen in the probe's 8 misspellings.
• Not affected. spell(), analyzeSimpleWord(), the Stemmer, HunspellStemFilter, indexing and search: they do not call the generator, and the hunt's probe found the same stems and hits with and without a fix.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[ampere, kiloampere, kiloamperes]> but was:<[ampere, kiloampere]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.hunspell.TestWordFormsPrefixEnabledSuffix.testSuffixEnabledByPrefixIsGenerated(TestWordFormsPrefixEnabledSuffix.java:39)

• The contract. getAllWordForms: "Generate all word forms for all dictionary entries with the given root word" (WordFormGenerator.java:117-119, Hunspell.java:334-336). generateAllSimpleWords (:294-298) names only compounds and case variants as excluded. NGramFragmentChecker.fromAllSimpleWords derives "all word forms (using WordFormGenerator) ... to get all words acceptable by the spellchecker" and names only compounds as a limitation (NGramFragmentChecker.java:76-81).
• Lucene's test oracle. TestSpellChecking.checkGoodSugWordsAreGenerated (TestSpellChecking.java:357-376, called at :324-327) asserts, for every fixture whose .dic it can expand whole and whose dictionary needs no input cleaning, that every lowercase .good/.sug word that analyzeSimpleWord accepts is generated; amperes and reworks would meet those conditions if such a fixture listed them. No fixture it runs on has a PFX rule whose continuation carries an SFX flag; dependencies.aff (:24-25), which does, is used only by TestDependencies. The same check runs in TestHunspellRepositoryTestCases on the C Hunspell test fixtures (when configured), and none of the 131 C Hunspell 1.7.2 test .aff files has the shape either.
• Deliberate support elsewhere. LUCENE-9688 (#10727, PR apache/lucene-solr#2229, 0a1a3f4c409, 2021-01-22; backported to 8.x and first released in 8.9.0), "Hunspell: consider prefix's continuation flags when applying suffix", made the Stemmer accept exactly this pattern; TestDependencies.java:42-43 asserts that calories stems to calorie.
• The reference implementation. C Hunspell checks a "conditional suffix (enabled by prefix)" (affentry.cxx:673, 1.7.2) and accepts amperes, kiloamperes, reworks and books, and rejects kilo and works.
• History. The guard came with the generator in #975, without a comment; the PR's 3 conversation and 19 inline review comments and #11904, which moved the line unchanged, never mention prefix continuation classes.

The case for "intended": the javadoc says getAllWordForms "is equivalent to "unmunch" from the "hunspell-tools" package", and C unmunch does not produce amperes either; line 378 looks like a deliberate canonical order (suffix first, prefix second) that avoids building each prefix+suffix form twice; and the same author wrote LUCENE-9688, so they knew the pattern. Read that way, the generator is a bounded enumerator in one canonical order, not a complete one.

Why it fails: C unmunch 1.7.2 has no continuation-class support at all; on this dictionary it prints ampere, 0/S|ampere and kilo/S|ampere, pasting the raw rule text onto the root. Lucene's generator already goes beyond it (kiloampere, twofold suffixes, suffix-enabled prefixes), so "equivalent" names the tool's purpose, not its limits. The canonical-order argument holds only when the entry carries the suffix flag; without it there is no suffix-first path, so the form is not deduplicated but lost. When the guard was written (d537013e708), the generator also collected forms in a LinkedHashSet<AffixedWord>, so a second copy would not have reached the output then; #11904 switched to lists for speed. The "bounded enumerator" reading conflicts with the method contracts ("all word forms"). Two executed reviews by a model from another family rated the intended-behaviour argument weak, as did the cross-model fact check of this doc, and an intended-behaviour devil's advocate and the adjudication rejected it. The argument does support low impact.

One option was built and validated in a scratch worktree of a9779ef3c193; the others are alternatives.

Pass the entry's own sorted flags down expand and replace line 378 of isCompatibleWithPreviousAffixes. The new path applies a suffix only when the single prefix's continuation class carries its flag and the entry does not, the condition under which the Stemmer accepts such a word (Stemmer.java:269-270). The new forms still pass canStemToOriginal, but that is no guarantee: it compares stem text only, not flags (with work/P it is true for works, which spell() rejects), and NGramFragmentChecker.processNGrams overrides it to return true. The evidence is the runs: every added form is accepted by spell() (French: all 833; the repro dictionaries). The rootFlags check avoids a second copy when the entry has the flag: for the tested rules (cross-product, no conditions) the suffix-first path has already built the word, so work/PS still gives 4 forms, not 5.

if (!isPrefix && !stem.getPrefixes().isEmpty()) {
  // Suffixes are applied before prefixes, so a suffix that the root allows has already been
  // combined with this prefix. Only a suffix that the prefix's continuation class allows
  // (and the root does not) still needs to be applied here.
  return size == 0
      && stem.getPrefixes().size() == 1
      && !FlagEnumerator.hasFlagInSortedArray(flag, rootFlags, 0, rootFlags.length)
      && dictionary.isFlagAppendedByAffix(stem.getPrefixes().get(0).affixId, flag);
}

Trade-offs

• Scope. One prefix and no suffix yet. A second prefix under COMPLEXPREFIXES, or a prefix-enabled suffix on top of an entry suffix, stays as today; whether C Hunspell accepts those was not checked. Not tested: affix conditions or strips that let a rule apply in one order only, and the fragment-checker path, where canStemToOriginal is bypassed. With a non-cross-product suffix or prefix (SFX S N, PFX P N), spell("reworks") is false and stock, fix A and patch B generate exactly the accepted forms (executed).
• Cost. One short scan of the entry's flags per suffix flag after a prefix. French generateAllSimpleWords: 24.2 and 23.2 s on main, 27.3 and 22.8 s with the fix (four single runs on a loaded host, not a benchmark): no measurable difference.
• Compatibility. More forms from the dictionary-tooling API and fewer needless compress edits; no index, analysis, stemming or spell-checking change. A fragment checker from NGramFragmentChecker.fromAllSimpleWords gains the n-grams of the added forms, so a Suggester configured with withFragmentChecker may suggest differently (no change in the hunt probe's 8 misspellings). Only private signatures change.

What the checks validated

• stock: the repro fails 3 of 4 with assertions on and off; the hunspell test package (47 classes, 162 tests with the repro) fails only the repro's 3 tests;
• fix A: the repro passes 4 of 4 with assertions on and off; the hunspell package passes (162 tests, 0 failures, 13 skipped: 10 in TestPerformance (no word corpora configured), 2 in TestAllDictionaries (no dictionary directory configured; one test is @Ignored), 1 in TestHunspellRepositoryTestCases (no C Hunspell test repository configured)); the whole lucene/analysis/common test task, repro included, runs 340 classes, 1,913 tests, 0 failures, 18 skipped; ecjLint, checkGoogleJavaFormat, forbiddenApisMain and forbiddenApisTest pass; tidy changes nothing;
• the hunt's repros with fix A shadowed: the finder's (1 of 1), the independent reproducer's (5 of 5), the cross-model reviewers' tests (2 of 2 each; the intended-lens test includes the duplicate-output control that the finder's patch fails) and the hunt's 5-test security probe pass;
• the finder's patch (option B) shadowed: the repro's testControls fails, [work, rework, reworks, works, reworks];
• French whole-dictionary probe: 833 forms added, 0 removed, all accepted by spell(); equal repeated AffixedWords 17,001 on main and with fix A (unchanged).

The finder's first patch: size == 0 && prefixes.size() == 1 && isFlagAppendedByAffix(prefix, flag), without the entry check.

Trade-offs

Builds the same form twice when the entry also has the flag: work/PS gives [work, rework, reworks, works, reworks] (equal AffixedWords), and the repro's testControls fails. Found independently by the reproducer and a cross-model reviewer. On the French dictionary it gives the same word list as option A.

For example a LinkedHashSet per entry. Not validated.

Trade-offs

Still does the duplicate work and keeps a set of all forms of an entry; option A avoids the duplicate at the source. The generator's first version (d537013e708) did collect forms in a LinkedHashSet<AffixedWord>; #11904 ("speed up WordFormGenerator") replaced the sets with lists, so this option partly undoes that change.

State in the getAllWordForms, generateAllSimpleWords and compress javadocs that suffixes enabled only by a prefix's continuation class are not generated. Not validated.

Trade-offs

Leaves the inconsistency with spell(), checkGoodSugWordsAreGenerated and LUCENE-9688.

Method: cross-implementation differential testing (non-formal tier). A hunt unit compared Lucene getAllWordForms, spell and analyzeSimpleWord with C Hunspell 1.7.2 (spell, stem) and the unmunch tool.

• Inputs: 4,000 grammar-built micro-dictionaries (per dictionary, every candidate word reachable by at most 3 affix rules) and 1,069 per-root cases from the Hunspell 1.7.2 and Lucene test fixtures; Lucene's dependencies.aff showed the difference.
• Root causes were separated with switchable shadow fixes: this fix alone adds 21 generated forms that spell() accepts and attributes to the root.
• The finder then wrote a one-test public-API JUnit that fails on the stock jar. Independent checkers and two executed reviews by a model from another family (OpenAI GPT-5.6) reproduced it with their own tests.