Hunspell getAllWordForms returns nothing for entries with a compound-position flag

In a Hunspell dictionary, COMPOUNDBEGIN, COMPOUNDMIDDLE and COMPOUNDEND mark words that may also be the first, middle or last part of a compound; the word stays a normal word on its own, and Lucene's spell() accepts it. ONLYINCOMPOUND is the flag that forbids standalone use. Lucene's word-form generator (Hunspell.getAllWordForms, WordFormGenerator.getAllWordForms, generateAllSimpleWords, and through it NGramFragmentChecker.fromAllSimpleWords) treats the three position flags like ONLYINCOMPOUND and FORBIDDENWORD: an entry that carries one yields no forms, and an affix whose continuation class carries one is dropped with everything derived from it. With COMPOUNDBEGIN B, SFX A 0 s . and foo/AB, getAllWordForms("foo") is [] instead of [foo, foos]. Over the current Swedish dictionary generateAllSimpleWords returns 483,150 distinct forms on main and 958,540 with a one-line fix; for Danish 440,159 and 1,037,591. The cause is WordFormGenerator.shouldConsiderAtAll (WordFormGenerator.java:261-273 on main a9779ef3c193). Every release from 9.5.0 through 10.6.0 (10.6.0 by reading) and main is affected. spell(), stemming, HunspellStemFilter, indexing and search do not use this code; suggestions change only for a suggester with a fragment checker built by NGramFragmentChecker.fromAllSimpleWords.

• Entry: a line of the .dic file. foo/AB has the stem foo and the flags A and B.
• Affix rule: SFX A 0 s . says that entries with flag A take the suffix s (foos).
• Continuation class: the flags an affix gives to the word it creates. SFX C 0 n/W . gives W to Computern.
• Position flags: COMPOUNDBEGIN, COMPOUNDMIDDLE, COMPOUNDEND (the manual's heading says COMPOUNDLAST; dictionaries and both parsers use COMPOUNDEND). hunspell(5): words with them "may be first elements in compound words" (middle, last). They add a permission; they take nothing away.
• ONLYINCOMPOUND: the flag for parts that are "not correct as a word by itself". FORBIDDENWORD marks non-words.

1. WordFormGenerator.shouldConsiderAtAll (WordFormGenerator.java:261-273) returns false if a flag set contains any of five flags: compoundBegin, compoundMiddle, compoundEnd, forbiddenword, onlyincompound.
2. It is applied to the entry's flags (:145, return List.of();), so Hunspell.getAllWordForms and both WordFormGenerator.getAllWordForms overloads return nothing for that entry; to the continuation flags of every affix (:246-254), so the affixed form and everything derived from it are dropped; and to each entry in generateAllSimpleWords (:313-314).
3. Only two of the five belong there. Of the compound-related flags, Lucene's Stemmer checks only ONLYINCOMPOUND for a simple word, on the affix (Stemmer.java:556) and on the root (:676); the position flags are required only for compound parts (WordContext.java:49-60). So spell() and analyzeSimpleWord() accept the words the generator leaves out.
4. The generic COMPOUNDFLAG, which also only permits compounding, is not in the list: foo/AZ with COMPOUNDFLAG Z gives [foo, foos], foo/AB with COMPOUNDBEGIN B gives [].

In Danish most damage is on the affix side: common suffixes such as the definite singular SFX 2 0 en/34 [^e] and the genitive SFX 10 0 s/34 [^sxz] carry COMPOUNDEND 34 in their continuation class. The fix adds 475,390 (sv_SE) and 597,432 (da_DK) forms and removes none: main misses 49.6% and 57.6% of the forms the fix generates.

• Who is hit. Callers of Hunspell.getAllWordForms, both WordFormGenerator.getAllWordForms overloads, generateAllSimpleWords, and through it NGramFragmentChecker.fromAllSimpleWords, with a dictionary that puts a position flag on ordinary entries or on affix continuation classes without ONLYINCOMPOUND. The API exists for dictionary editing and word lists (LUCENE-10626, "add tools to aid dictionary editing"). Deterministic; which words are hit depends only on the dictionary.
• How much. Whole current dictionaries: sv_SE 483,150 distinct forms on main vs 958,540 with the fix; da_DK 440,159 vs 1,037,591, so main misses 49.6% (sv_SE) and 57.6% (da_DK) of the forms. Earlier samples by independent verifiers found about half of the sv_SE entries with no forms (759 of 1,538 entries; 1,029 of 2,000 roots). The finder also reports 102 of 3,000 de_DE_frami roots with missing hyphen forms such as Balkan-, and none in fr, lb and nb_NO (not re-run).
• Fragment checker. A suggester built with withFragmentChecker(NGramFragmentChecker.fromAllSimpleWords(...)) lacks the n-grams of the missing words and can drop valid suggestions: for foo/SX (COMPOUNDBEGIN X) plus qux/S, suggest("foox") gives [foo, foos] without the checker and [foo] with it; the fix restores [foo, foos]. That javadoc already warns that compounds can be missed, so this is secondary.
• Not affected: spell, getRoots, analyzeSimpleWord, suggest without a fragment checker, Stemmer, HunspellStemFilter, indexing and search. compress() was not shown to be affected.
• Since 9.5.0, the API's first release.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[foo, foos]> but was:<[]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.hunspell.TestWordFormsCompoundBeginEntry.testCompoundBeginEntryHasStandaloneForms(TestWordFormsCompoundBeginEntry.java:38)

• The format. hunspell(5) (1.7.2, man/hunspell.5:435-447): "Words signed with COMPOUNDBEGIN (or with a signed affix) may be first elements in compound words"; COMPOUNDLAST (the heading's name for COMPOUNDEND, which the manual's own example and both parsers use) and COMPOUNDMIDDLE are worded the same. ONLYINCOMPOUND: "Suffixes signed with ONLYINCOMPOUND flag may be only inside of compounds ... Note: also valuable to flag compounding parts which are not correct as a word by itself."
• Lucene's fixture. germancompounding.aff:20-21: "Hint: ONLYINCOMPOUND is not required everywhere, but the checking will be a little faster with it." Its .good list contains Computern, built through a COMPOUNDEND continuation (SFX C 0 n/WD .), which the generator does not produce.
• The API. "Generate all word forms for all dictionary entries with the given root word" (Hunspell.java:334-336, WordFormGenerator.java:117-118), with no exclusion stated.
• Lucene's test oracle. TestSpellChecking.checkGoodSugWordsAreGenerated (TestSpellChecking.java:357-376, called for fixtures that expand completely and need no input cleaning, :324-327) requires every lowercase .good word that analyzeSimpleWord analyzes to be generated. On a minimal fixture (foo/AB, COMPOUNDBEGIN B) it fails with "Some *.good/sug words weren't generated: [foo, foos]"; plain and ONLYINCOMPOUND controls pass; all pass with the fix. No existing fixture exposes the gap: in germancompounding the missed Computern is capitalized and the check skips capitalized words (:372); Hunspell's own 2999225 and opentaal_* fixtures list only compounds in .good; other position-flag fixtures escape for fixture-specific reasons.
• Other expansion code. C Hunspell's AffixMgr::expand_rootword and Lucene's GeneratingSuggester.hasCompatibleFlags (GeneratingSuggester.java:256-265) skip only NEEDAFFIX, CIRCUMFIX and ONLYINCOMPOUND, never the position flags.
• History. The five-flag list is in the first version of the class (#975), uncommented; "compound" appears nowhere in its review or issue comments.

The case for "deliberate scope limit": the exclusion was written on purpose in #975 and kept by the same author in #11904; #11909 added generateAllSimpleWords with the javadoc "Only "simple" words are returned, no compounding flags are processed" (WordFormGenerator.java:297); NGramFragmentChecker.fromAllSimpleWords warns that compounding languages can miss n-grams; TestSpellChecking builds n-gram checkers only when compoundRules, compoundBegin and compoundFlag are all unset (TestSpellChecking.java:287-294); AffixedWord is "the analysis result of a simple (non-compound) word".

Why it fails for getAllWordForms: these are simple words (no compound is formed), and spell() and analyzeSimpleWord() accept them as such; "no compounding flags are processed" most naturally means that no compounding is done, and it documents generateAllSimpleWords, not getAllWordForms; the list is inconsistent (COMPOUNDFLAG is expanded); and returning nothing for about half of a language's ordinary nouns is an unlikely goal for a dictionary-editing tool. "Equivalent to unmunch" supports neither side: C unmunch reads only PFX, SFX and FULLSTRIP lines and ignores all compound, ONLYINCOMPOUND and FORBIDDENWORD directives. The independent reproducer and the reviews by a model from another family rated the intended-behaviour argument moderate, the devil's advocate found it does not hold for getAllWordForms, and the adjudication confirmed the bug with this caveat. The exclusion looks deliberate; if maintainers keep it, the fallback is to document it on getAllWordForms.

One option was built and validated in a scratch worktree of a9779ef3c193; the other two are fallbacks.

Remove the three position flags from shouldConsiderAtAll (WordFormGenerator.java:261-273). In a plain WordFormGenerator every affixed form must still pass the existing sanity check canStemToOriginal (:184-217), which runs Lucene's stemmer in the simple-word context; the bare entry is not checked that way but is still skipped for NEEDAFFIX (:156-159); no compounds are generated. NGramFragmentChecker.processNGrams turns the sanity check off on purpose ("overgenerate a bit", NGramFragmentChecker.java:130-135), so on that path the newly reached forms are not checked at all. Optional: reword the generateAllSimpleWords javadoc (:297) to "no compounds are generated".

private boolean shouldConsiderAtAll(char[] flags) {
  for (char flag : flags) {
    // COMPOUNDBEGIN/MIDDLE/END only allow compounding; ONLYINCOMPOUND forbids standalone use
    if (flag == dictionary.forbiddenword || flag == dictionary.onlyincompound) {
      return false;
    }
  }
  return true;
}

Trade-offs

• More output, more time. About 2x (sv_SE) and 2.4x (da_DK) as many forms; single runs of generateAllSimpleWords took 2.0 s vs 3.4 s (sv_SE) and 2.6 s vs 8.1 s (da_DK), not a benchmark. Still bounded by the dictionary and interruptible through checkCanceled.
• It exposes the generator's other soundness gaps on newly reached entries. spell() rejects 885 of the 475,390 added sv_SE forms (0.19%; all capitalized or starting or ending with -, kinds stock already emits: 877 such forms on stock) and 3,336 of the 597,432 added da_DK forms (0.56%; stock emits 4,389). Together with the candidate fix of the sibling bug LFV-638 (the sanity check accepts any affix path back to the stem), 2,709 remain in da_DK; for example abscess (absces plus SFX 7 0 s/11,2,8,35,34 s, whose continuation has NEEDAFFIX 8 and COMPOUNDEND 34) is emitted by this fix alone and removed by the combined fix. The rest were not classified further. A maintainer may prefer to land both together.
• Compatibility. The output of the dictionary-tooling API changes, and so do the suggestions of a suggester with an NGramFragmentChecker.fromAllSimpleWords fragment checker (suggest("foox"): [foo] on main, [foo, foos] with the fix); no index or analysis change.

What the checks validated

• Stock: the repro fails 3 of 4 with assertions on and off; the hunspell test package (47 classes, 162 tests with the repro) fails only those 3.
• Fix: the repro passes 4 of 4 with assertions on and off; the hunspell package passes (162 tests, 0 failures, 13 skipped: three suites need external dictionaries); the whole lucene/analysis/common test task, repro included: 340 classes, 1,913 tests, 0 failures, 18 skipped; ecjLint, checkGoogleJavaFormat, forbiddenApisMain and forbiddenApisTest pass; tidy changes nothing.
• Hunt runs with the same change shadowed: the finder's, the independent reproducer's (9 of 9), the root-cause reviewer's and the other cross-model reviewers' tests pass; Lucene's fixture oracle passes on the devil's advocate's fixtures; 125 Hunspell 1.7.2 test fixtures (131 minus the 6 expected failures), run one by one through that oracle by the devil's advocate's own harness, give identical outcomes with and without the fix (TestHunspellRepositoryTestCases itself reports "No runnable methods" under plain JUnit).
• Whole-dictionary probe (sv_SE, da_DK): forms added, none removed; numbers above.

Say in the getAllWordForms and generateAllSimpleWords javadocs that entries and affixes with COMPOUNDBEGIN, COMPOUNDMIDDLE or COMPOUNDEND produce no forms.

Trade-offs

Leaves the disagreement with spell(), COMPOUNDFLAG and checkGoodSugWordsAreGenerated, and the empty results for Swedish and Danish. Not validated.

For example a WordFormGenerator setting that includes or skips compound-position entries.

Trade-offs

More API for a behaviour for which no explicit rationale was found; the recommended option is simpler. Not validated.

Method: cross-implementation differential testing (non-formal tier). Hunt unit "Hunspell word-form generator vs munch/unmunch" compared Lucene getAllWordForms, spell and analyzeSimpleWord with C Hunspell 1.7.2 (spell, stem) and the unmunch tool.

• Inputs: 4,000 grammar-built micro-dictionaries (3 grammar families, several seeds; per dictionary every candidate word reachable by at most 3 affix rules) and 1,069 per-root cases from the Hunspell 1.7.2 and Lucene test fixtures (germancompounding, 2999225, opentaal_cpdpat, opentaal_cpdpat2 and opentaal_keepcase show it).
• Root causes were separated with switchable shadow fixes: with only this fix, generator misses of words that spell() accepts and attributes to the root fell from 607 to 64.
• The finder then wrote a two-test public-API JUnit that fails on the stock jar; an independent reviewer from another model family (OpenAI GPT-5.6) reviewed it in executed mode before the candidate was filed.