Hunspell getAllWordForms returns words that spell() rejects when another affix adds the same text

Hunspell.getAllWordForms(root), and WordFormGenerator.getAllWordForms, generateAllSimpleWords and compress, which share its expansion, keep each affix-derived form only if the check canStemToOriginal passes; its javadoc says the form "is indeed accepted by the spell-checker and analyzed to be the form of the original dictionary entry". The check strips affixes from the form and accepts as soon as any stripping path gives back the stem's text, ignoring which affixes that path removed (WordFormGenerator.java:211 on main a9779ef3c193). When an affix makes a form that is not a word on its own (its continuation carries NEEDAFFIX, or a CIRCUMFIX side is missing), the stemmer skips that path, but any other affix with the same text and strip whose condition matches, even of a class the entry does not have, leads back to the same text and the form is kept. With NEEDAFFIX X, SFX A 0 s/XB, SFX B 0 en, an unrelated SFX C 0 s, foo/A and bar/C, getAllWordForms("foo") returns [foo, foos, foosen] while spell("foos") is false; Lucene's needaffix5 fixture plus two unused same-text lines regenerates exactly its .wrong words; compress suggests foo/AB, which does not accept foos. The public Esperanto dictionary yields 25,554 such words (24,936 of 29,377 roots) and 5,000 Galician roots 1,510. Every release from 9.5.0, the API's first, to 10.6.0 (10.6.0 by reading) and main is affected. Spell-checking, suggestions, stemming, HunspellStemFilter, indexing and search are not.

• Affix class: a flag in the .aff file, for example SFX A 0 s/XB . = class A appends s (strips nothing, any condition) and gives the result the continuation flags X and B. An entry foo/A in the .dic may take class A.
• NEEDAFFIX: a flag (NEEDAFFIX X) that marks a pseudo-form: a form whose flags contain X is not a word until another affix is applied. So foos (foo + A, flags XB) is not a word, but foosen (foos + B) is.
• canStemToOriginal: the generator's filter. It runs the stemmer backwards on each generated form and keeps the form if the stemmer gets back to the entry's stem.

1. WordFormGenerator.expand (:233-259) applies each affix and keeps the result only if canStemToOriginal says so (:248); it has no NEEDAFFIX or CIRCUMFIX check of its own (shouldConsiderAtAll, :247, only drops compound, FORBIDDENWORD and ONLYINCOMPOUND continuations; the bare stem is added without the check unless the entry has NEEDAFFIX, :154-160). For a cross-product affix it recurses with the continuation flags (:251-253), which makes foosen.
2. canStemToOriginal (:184-217) calls Stemmer.removeAffixes (:215) with a processor (:192-214) that receives the ids of the affixes each path removed, but decides on text alone: foundStem |= length == stem.length() && stem.equals(new String(chars, offset, length)); (:211).
3. The stemmer skips the form's own path because A's continuation needs another affix (needsAnotherAffix, Stemmer.java:610, :693-708). It then tries class C, which also strips s (same strip, matching condition), and offers foo with C's id. The text matches, so foos is kept.
4. The spell-checker would reject that candidate because foo has no flag C (Stemmer.java:265-269). The generator's check does not look at entry flags (it cannot rely on them, since compress runs it for hypothetical flag sets), and nothing else ties the accepted path to the form's own affixes, which the generator knows (AffixedWord.getPrefixes() / getSuffixes()).

Real dictionaries (stock main vs the fix, public API): LibreOffice Esperanto, all 29,377 roots: 14,235,519 forms, 25,554 rejected by spell() (24,936 roots), 0 with the fix; Galician gl_ES, 5,000 sampled roots: 2,153,392 forms, 1,510 rejected (243 roots), 0 with the fix. C Hunspell 1.7.2 rejects 397 of 400 and 300 of 300 sampled leaks. Most Esperanto leaks are bare stems such as zink (the word needs an ending, zinko); there the other path is a second entry of the same class (SFX O 0 0/2XN€ needs another affix, SFX O 0 0/543 leads to the same text).

• Who is exposed. Callers of Hunspell.getAllWordForms / compress and WordFormGenerator.getAllWordForms / generateAllSimpleWords / compress: word-list export (the javadoc calls it equivalent to unmunch), dictionary editing (compress, equivalent to munch), auxiliary lists built from a dictionary. The Hunspell module is opt-in and this API is a niche part of it.
• What goes wrong. Silent non-words in the list: 25,554 of 14.2 million Esperanto forms, mostly bare stems. compress can propose an entry that does not accept a word it was asked to cover (foo/AB for foos), so an editing tool that trusts it writes a wrong entry. The javadoc's second promise breaks too: on Esperanto the fix also removes 2,183 forms that spell() accepts only as forms of another entry or with no simple-word analysis.
• What is not affected. spell, suggest, getRoots / analyzeSimpleWord, HunspellStemFilter, indexing and querying do not call this check; leaked words stay rejected and unstemmed. NGramFragmentChecker overrides the check with return true and is unchanged. In the security probe (which runs spell, analyzeSimpleWord, HunspellStemFilter, indexing and search; suggest and getRoots were read, not run): no exception, no hang, and the same number of affix applications with or without the bug; memory was not measured.
• What it takes. A dictionary in which an affix makes a form that is not valid on its own (NEEDAFFIX or CIRCUMFIX in its continuation) and another affix path, at the same position, with the same affix text and strip and a matching condition, that leads back to the same stem text. No other configuration, no scale; deterministic.
• How common. 24 of 191 public .aff files match one conservative syntactic scan (a NEEDAFFIX or CIRCUMFIX entry plus an entry of a different class with the same strip and non-empty text; for example da, de (frami), eo, eu, gl, id, ka, lb, lv, sk). It ignores conditions and the .dic, so it over-counts, and it misses same-class and zero-text collisions such as Esperanto's, so it is not an upper bound. Esperanto and Galician leak at scale. In the sampled Danish and German (frami) runs the fix changed nothing; in Luxembourgish it removed 9 forms that Lucene's spell() rejects but C Hunspell accepts (a separate stemmer difference, so no clean leak). A Basque run did not finish in its time box; the others were not run.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[]> but was:<[foos]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.hunspell.TestWordFormsNeedAffixLeak.testGeneratedFormsPassSpell(TestWordFormsNeedAffixLeak.java:46)

• Contract. canStemToOriginal javadoc (WordFormGenerator.java:178-183, added by #11909): "A sanity-check that the word form generated by affixation in getAllWordForms(String, String, Runnable) is indeed accepted by the spell-checker and analyzed to be the form of the original dictionary entry."
• Test. TestSpellChecking.checkExpansionGeneratesCorrectWords (:378-388), which expandWholeDictionary (:337-355) calls for every .dic line (except empty lines and lines with a backslash) of each fixture that TestSpellChecking runs through doTest, fails with "Misspelled words generated" on any generated word that spell() rejects or analyzeSimpleWord cannot analyse. It came with the generator in #975. Stock fixtures do repeat affix texts (germancompounding.aff, compress.aff), and circumfix.aff has this bug's own shape (CIRCUMFIX X with SFX C 0 obb ., obb/AX and obb/BX): there stock getAllWordForms("nagy") returns nagyobb three times, twice only because the plain obb path strips to the same text. It stays invisible because nagyobb is a real word, and TestCircumfix does not run the expansion check. Among the fixtures that TestSpellChecking expands, the ones that repeat an affix text (base, base_utf, germancompounding, germanManualCase) have neither NEEDAFFIX nor CIRCUMFIX, and needaffix5 repeats no affix text, so the test never fired.
• Fixture. needaffix5.wrong lists pseudoprefoo, foopseudosuf, pseudoprefoopseudosuf: the three words the generator emits once two unused same-text lines are added.
• Independent engine. C Hunspell 1.7.2 rejects foos and nearly all sampled real-dictionary leaks.
• History. The text-only comparison is as old as the generator (#975, d537013e708); none of its 4 conversation and 19 review comments discusses which affix path the check accepts.

The strongest case for "intended": the method is a protected, overridable sanity check; NGramFragmentChecker overrides it to "overgenerate a bit, but avoid very expensive checks"; getAllWordForms is documented as equivalent to C unmunch, which has no NEEDAFFIX support at all; and the check has to ignore entry flags because compress() runs it for hypothetical flag sets.

Why it fails: the default implementation's own javadoc promises spell-checker acceptance; the override is an explicit opt-out by one other caller, and its comment implies the default is the precise, expensive check; Lucene's own test demands more than unmunch; and the fix compares only the affix ids the stemmer already reports with the form's own affixes, so it stays independent of entry flags (compress keeps working and suggests foo/ABC). An independent reproducer, an intended-behaviour devil's advocate and a reviewer from another model family all rated the intended-behaviour argument weak, and so did the cross-model fact check of this doc, which adds that the argument does not settle which fix is right: fix A's exact-affix-id rule is one reading of the contract. Verdict: not intended, a defect.

Option A was applied in a scratch worktree at a9779ef3c193 and run through gradle (and re-run by the fact check), and compared with stock in one JVM on the Esperanto and Galician dictionaries. Options B and C are sketches, not run.

One file, +13/-1. The processor already receives the ids of the affixes each stripping path removed; the fix requires them to equal the derived word's own prefixes and suffixes (AffixedWord.getPrefixes() / getSuffixes(), outermost first) in addition to the stem text. It still ignores entry flags, so compress() keeps working on hypothetical flag sets.

--- WordFormGenerator.canStemToOriginal (WordFormGenerator.java:211)
-            foundStem |= length == stem.length() && stem.equals(new String(chars, offset, length));
+            foundStem |=
+                length == stem.length()
+                    && stem.equals(new String(chars, offset, length))
+                    && outerPrefix == affixAt(derived.getPrefixes(), 0)
+                    && innerPrefix == affixAt(derived.getPrefixes(), 1)
+                    && outerSuffix == affixAt(derived.getSuffixes(), 0)
+                    && innerSuffix == affixAt(derived.getSuffixes(), 1);
+
+  private static int affixAt(List<Affix> affixes, int i) {
+    return affixes.size() > i ? affixes.get(i).affixId : -1;
+  }

Trade-offs

It changes only which forms are kept, never which affixes are applied. The processor now keeps enumerating past text matches from other paths until it reaches the form's own path, so it can do more stemming work per form (not benchmarked), and a FORBIDDENWORD candidate met on the way now rejects a form that stock had already accepted; the check was already order-dependent this way when the forbidden candidate came first (read from the code; no test covers it). A maintainer may prefer another fix site. Fix A requires the exact affix ids, which is stricter than the javadoc's wording: on the stock circumfix.aff fixture it drops the two duplicate nagyobb values from the CIRCUMFIX pseudo-paths obb/AX and obb/BX (the word stays through the plain obb path). It follows the current stemmer: where Lucene's spell() rejects a word that C Hunspell accepts, the fix drops that form too (9 Luxembourgish forms, 3 hyphenated Esperanto E- forms). In the finder's 4,000-dictionary differential this fix alone also changed two other counters (dictionaries the harness could not process 1 to 2, words on which Lucene threw 4 to 3); not investigated. Validated on the listed tests and dictionaries; not proven for every affix graph.

What the checks validated

• Stock main a9779ef3c193, gradle: the repro fails 3 of 4 with assertions on and off.
• With the fix, gradle: the repro passes 4 of 4 with assertions on and off; the hunspell test package (47 classes, 162 tests including the repro, 0 failures, 13 skipped: 10 TestPerformance tests need word corpora, one TestAllDictionaries test needs a dictionary directory, TestHunspellRepositoryTestCases needs the C Hunspell test repository, and TestAllDictionaries.testMaxPrologueNeeded is @Ignored), among them TestSpellChecking 53 of 53, which also checks, for each fixture whose .dic it expands fully and that needs no input cleaning, that every lowercase, analysable .good and .sug word is still generated; the whole lucene/analysis/common test task: 340 classes, 1,913 tests, 0 failures, 18 skipped. ecjLint, checkGoogleJavaFormat, forbiddenApisMain, forbiddenApisTest and checkLicenses pass.
• compress([foo, foos, foosen]) suggests foo/ABC instead of foo/AB; the needaffix5 variant generates exactly the stock fixture's 12 forms.
• Esperanto, all 29,377 roots, stock and fixed generator side by side: the fix removes the 25,554 rejected forms plus 2,183 forms that spell() accepts only as forms of another entry (1,604, for example manĝu, listed under the root manĝu but analysed only as a form of manĝ) or with no simple-word analysis (579, which the upstream test also counts as misspelled); it removes no form that analyses to the same root and adds none. Galician, 5,000 roots: 1,185 rejected forms removed plus 1 accepted only through another entry; again none of the same root.
• Earlier, with the same affix-path comparison logic shadowed onto stock jars (the finder's version of the hunk: a samePath variable, the conditions in another order): the finder's, the reproducer's, the devil's advocate's and the cross-model reviewers' tests pass, and the security probe is unchanged apart from the leaked forms and the compress suggestion.

Before keeping a derived form, check whether its own outermost affix still needs another affix (NEEDAFFIX in the appended flags, or a CIRCUMFIX side missing), mirroring Stemmer.needsAnotherAffix.

Trade-offs

Leaves the check untouched, but duplicates the stemmer's rule, which is subtle: a NEEDAFFIX pseudo-suffix is valid once an outer affix follows (prefoopseudosuf is in needaffix5.good). It also would not cover other forms the stemmer cannot reproduce through their own path but accepts through another one (the finder's cross-product variant, not re-checked). Not run.

Keep a form only if spell(word) is true and analyzeSimpleWord(word) contains an analysis of the same entry, the oracle of checkExpansionGeneratesCorrectWords.

Trade-offs

The most literal statement of the contract for getAllWordForms, but it costs a full spell check per form and does not work for compress(), which generates forms for hypothetical entries that are not in the loaded dictionary. Not run.

Method: cross-implementation differential testing (the hunt's non-formal tier): a hunt unit compared Lucene's getAllWordForms, spell and analyzeSimpleWord with C Hunspell 1.7.2 (libhunspell spell and stem) and the hunspell-tools unmunch.

• Inputs: 4,000 seeded, grammar-built micro-dictionaries (per dictionary, every candidate word reachable by at most 3 affix rules) and 1,069 per-root cases from the Hunspell 1.7.2 and Lucene test fixtures.
• Root causes were separated with switchable shadow fixes: with this fix alone, the 18 self-inconsistent words in 15 dictionaries fell to 0 (the finder's counts; the verifiers did not re-run the 4,000 dictionaries).
• The finder wrote the foo/A + bar/C JUnit; an independent reproducer added the needaffix5 variant and the compress effect; a credibility audit measured the Esperanto and Galician reach.
• Found 2026-10-27 on main a9779ef3c193; then checked by an independent reproducer, an intended-behaviour devil's advocate, two executed reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a history and prior-report follow-up, a security screen and the audit. This doc added the gradle runs, the fix validation and the same-root check on Esperanto and Galician.