Hunspell spell() and suggest() throw AssertionError when a CHECKCOMPOUNDPATTERN replacement with empty beginchars ends the word

CHECKCOMPOUNDPATTERN endchars[/flag] beginchars[/flag] [replacement] (hunspell(5)) can carry a replacement, a simplified compound form in which the replacement letters stand for the boundary endchars|beginchars. When Lucene's Hunspell checks a compound it tries every break position, and at each one checkCompoundPatternReplacements (Hunspell.java:249-267) rewrites a matching replacement into endchars + beginchars (CheckCompoundPattern.java:94-103) and puts the break right after endchars (Hunspell.java:255). It never checks that anything is left after the break. With empty beginchars (written /X) and a replacement at the end of the word nothing is, so checkCompoundsAfter looks up a second part of length 0 (:272-275) and breaks assert length > 0 in WordStorage.lookupWord (WordStorage.java:106-107). With dictionary foo/C, COMPOUNDFLAG C, COMPOUNDMIN 1 and CHECKCOMPOUNDPATTERN o /X z, spell("foz") throws AssertionError with assertions on; with them off the empty lookup finds nothing and the answer is the correct false, which C Hunspell 1.7.2 also gives. suggest() goes through the same code. The stock LibreOffice Dutch dictionary reaches it too: a trailing # comment on a pattern with empty beginchars is parsed as the replacement #, so spell("huis#") throws with assertions on (with them off the answers match C Hunspell 1.7.2). The missing check came with CHECKCOMPOUNDPATTERN support (LUCENE-9717, 2021) and the assertion with LUCENE-9825 (2021); all 42 releases from 8.9.0 to 10.6.0 and main fail with assertions on. A one-line guard fixes it.

• endchars / beginchars: the letters a CHECKCOMPOUNDPATTERN rule expects at the end of the first part and the start of the second. Each may carry a flag after /; /X alone means no letters, flag X. A flagless empty field cannot be written: the parser splits the line on whitespace (CheckCompoundPattern.java:32), so o z would make z the beginchars.
• Replacement: the optional fourth field (CheckCompoundPattern.java:45), the simplified spelling of the boundary. hunspell(5): "The optional replacement parameter allows simplified compound form."

1. checkCompounds(CharsRef, ...) (Hunspell.java:213-247) tries breakPos from COMPOUNDMIN up to word.length - COMPOUNDMIN (:220-221), so on the regular path both parts have at least COMPOUNDMIN letters. At every position it also calls checkCompoundPatternReplacements (:241).
2. checkCompoundPatternReplacements (:249-267) calls expandReplacement(word, pos) (:252), which, if the replacement matches at pos, returns the word with the replacement rewritten as endChars + beginChars (CheckCompoundPattern.java:94-103). The break is set to pos + pattern.endLength() (Hunspell.java:255), the first part is looked up (:256-257) and, if found, checkCompoundsAfter checks the rest (:259-260).
3. At the first boundary (the call on the whole word, offset 0, as in the repro) the second part has |beginChars| letters plus whatever followed the replacement, so it is empty exactly when beginchars are empty and the replacement ends the word. Nothing checks for that. (At later boundaries of a multi-part compound, stock expandReplacement returns the whole word with offset 0, the sibling bug LFV-377, so the earlier parts count as the remainder and the empty lookup does not happen there.)
4. checkCompoundsAfter (:269-286) sets remainingLength = word.length - breakPos (:272), here 0, and calls findStem with it (:274-275): Stemmer.doStem (Stemmer.java:227-229), Dictionary.lookupWord (Dictionary.java:283-285), then WordStorage.lookupWord, whose first line is assert length > 0; (WordStorage.java:106-107).
5. With assertions off the zero-length lookup finds no entry, the compound attempt fails and spell returns false. The read word[offset + length - 1] (WordStorage.java:118) stays in bounds because the break offset is at least 1 (COMPOUNDMIN is at least 1, Dictionary.java:438).

foz has 3 letters and COMPOUNDMIN is 1, so the loop tries breaks 1 and 2 (limit 3 - 1 + 1 = 3). At 1: f is no word and z does not start at position 1. At 2: fo is no word; then z matches at position 2, the expanded word is fo + o + (nothing) = foo, the break is 2 + 1 = 3, foo (flag C) is accepted as the first part, and the second part has 3 - 3 = 0 letters.

COMPOUNDMIN. A replacement starts at a break position, at most word.length - COMPOUNDMIN, so one that ends the word has at least COMPOUNDMIN letters. With the default 3 the trigger needs a 3-letter replacement: CHECKCOMPOUNDPATTERN d /X xyz with food/C and fooxyz fires it, while o /X z does not.

suggest(). Suggester checks candidates with a Hunspell whose findStem it overrides (Suggester.java:198-208), so the same path runs: suggest("fooxyz") and suggest("fozz") throw the same AssertionError with assertions on; with them off suggest("fozz") returns [foo].

Not reached by analysis. Stemmer and HunspellStemFilter, used inside analyzers, contain no compound-pattern checks; checkCompoundPatternReplacements and checkCompoundsAfter exist only in Hunspell.java.

• Who is hit. Code that calls Hunspell.spell or Hunspell.suggest (module lucene/analysis/common) and runs with assertions enabled and loads a dictionary with a CHECKCOMPOUNDPATTERN replacement whose beginchars are empty (on purpose, or by accident as in the Dutch dictionaries below) and checks a word that ends with that replacement (at least COMPOUNDMIN letters long). Assertions are typically on in test runs; Lucene's own build enables them for about 75% of test seeds.
• What happens. A deterministic AssertionError (an Error, not an Exception) out of spell or suggest instead of an answer. With assertions off the repro, and the Dutch words below, get the same answers as C Hunspell 1.7.2 (the FULLSTRIP side case below is the exception).
• Default vs opt-in. An opt-in API whose behaviour depends on the dictionary; assertions are off by default in the JVM. Stemming and analysis do not reach this code. No scale is needed: a 3-letter word.
• How likely. Rare. No surveyed dictionary uses the replacement field on purpose, but two of the 191 public .aff files surveyed by the hunt (the LibreOffice dictionaries and wooorm/dictionaries at the commits Lucene's regression build pins, lucene/analysis/common/build.gradle:146-153), the two Dutch ones, end some CHECKCOMPOUNDPATTERN lines with a # comment (36 lines in LibreOffice nl_NL, 42 in wooorm nl). Lucene's parser takes the fourth whitespace-separated field as the replacement (CheckCompoundPattern.java:32, :45), as C Hunspell 1.7.2 does, so # becomes the replacement. One such line in nl_NL (line 659, and four in wooorm nl) has empty beginchars: CHECKCOMPOUNDPATTERN /Xf /Xf # moeder-dochter etc, and nl_NL's COMPOUNDMIN 0 is raised to 1 (Dictionary.java:438). With the stock nl_NL dictionary, spell("huis#"), "boek#", "water#" and "moeder#" (a compound-begin word followed by #) throw the same AssertionError with assertions on (main; huis# and boek# also on 10.6.0). With them off Lucene answers huis# false, boek# true, water# true, moeder# false, exactly as C Hunspell 1.7.2 does; with fix A nothing throws and the answers stay the same. It still needs a # at the end of the checked word (# is not in the Dutch WORDCHARS) and assertions on. Lucene's replacement fixtures all have non-empty beginchars, and neither Lucene's nor C Hunspell 1.7.2's fixtures combine a replacement with empty beginchars.
• A side case, not claimed. With FULLSTRIP and a suffix that strips a whole root (for example SFX S y 0 . with y/CSX), the empty second part is read as that root minus the suffix, and Lucene with assertions off returns true for foz (main; a similar variant also on 10.5.2). C Hunspell 1.7.2 says false there but 1.3.2 says true, so no wrong production answer is claimed; with the guard Lucene answers false, like 1.7.2.

The test sits in org.apache.lucene.analysis.hunspell, next to TestSpellChecking, by Lucene test convention; it uses only public API: Dictionary(Directory, String, InputStream, InputStream), Hunspell(Dictionary), Hunspell.spell(String) and ByteBuffersDirectory. It calls no random(), so the outcome does not depend on the seed.

Not shown by this one test, executed by the verifiers from the same root cause: suggest(), the default-COMPOUNDMIN form (CHECKCOMPOUNDPATTERN d /X xyz, fooxyz), and the stock Dutch dictionary case (spell("huis#"), see Who is affected), which needs a dictionary download.

As the regression test: the same test unchanged, for example as a new method of TestSpellChecking. It could also be written as a fixture set for doTest (.aff, .dic, .wrong), the form all 53 test methods of TestSpellChecking use. It fails on stock main with assertions on and passes with fix A with assertions on and off (gradle); :lucene:analysis:common:check -x test passes with fix A and this test in place.

cp TestCompoundPatternEmptyTail.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/hunspell/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.hunspell.TestCompoundPatternEmptyTail -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false instead (the test then passes: spell("foz") is false)

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (with =false the test passes; the same on 10.5.2 and 10.6.0)
TestCompoundPatternEmptyTail > testReplacementAtWordEnd FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1CA89FF644B60EBF]:0)
        at org.apache.lucene.analysis.hunspell.WordStorage.lookupWord(WordStorage.java:107)
        at org.apache.lucene.analysis.hunspell.Dictionary.lookupWord(Dictionary.java:284)
        at org.apache.lucene.analysis.hunspell.Stemmer.doStem(Stemmer.java:229)
        at org.apache.lucene.analysis.hunspell.Hunspell.findStem(Hunspell.java:169)
        at org.apache.lucene.analysis.hunspell.Hunspell.checkCompoundsAfter(Hunspell.java:275)
        at org.apache.lucene.analysis.hunspell.Hunspell.checkCompoundPatternReplacements(Hunspell.java:260)
        at org.apache.lucene.analysis.hunspell.Hunspell.checkCompounds(Hunspell.java:241)
        at org.apache.lucene.analysis.hunspell.Hunspell.checkCompounds(Hunspell.java:157)
        at org.apache.lucene.analysis.hunspell.Hunspell.spellClean(Hunspell.java:100)
        at org.apache.lucene.analysis.hunspell.Hunspell.spell(Hunspell.java:86)
        at org.apache.lucene.analysis.hunspell.TestCompoundPatternEmptyTail.testReplacementAtWordEnd(TestCompoundPatternEmptyTail.java:37)

:lucene:analysis:common:test (FAILURE): 1 test, 1  failure

# C Hunspell 1.7.2 on the same dictionary: foz 0 (rejected), foo 1 (accepted)

• The contract is a boolean. spell promises "whether the given word's spelling is considered correct according to Hunspell rules" (Hunspell.java:71-73) and documents no exception.
• The precondition is internal. LUCENE-9825 (4842e0c9cab, 2021) added assert length > 0 to WordStorage.lookupWord together with zero-length guards in other callers: if (length == 0) return in Stemmer.stem (today in Stemmer.analyze, which stem calls, Stemmer.java:90-92) and if (word.length() == 1) return; in ModifyingSuggester.tryRemovingChar (today ModifyingSuggester.java:296-297). In that PR's review the author wrote "We don't accept empty words for lookup anymore".
• A related precedent. LUCENE-9833 (#10872, "Hunspell: AssertionError in WordStorage.lookupWord", PR #13; released in 8.9.0 and 9.0.0) is the same assertion reached from ModifyingSuggester.tryRep, where a REP candidate with a leading space split into an empty word part. Maintainers guarded that caller (the candidate is trimmed) and kept the assertion. That is an analogy, not proof of what this rule combination should do. The repro fails on every release that contains that fix.
• Every other split keeps both parts non-empty (Hunspell.java:220-221); only the replacement path can make an empty part.
• The input is legal. hunspell(5) gives the syntax CHECKCOMPOUNDPATTERN endchars[/flag] beginchars[/flag] [replacement] and the example CHECKCOMPOUNDPATTERN 0/x /y (empty beginchars with a flag). Lucene's parser accepts the line (Dictionary.java:458-464), Lucene's own fixture onlyincompound2.aff uses empty beginchars (0/B /A), and the class javadoc's list of unsupported features (Hunspell.java:35-44) does not mention it. C Hunspell 1.7.2 parses the test's dictionary and answers false; 1.3.2 also answers false on the hunt's two-root variant (foo/C, bar/C).
• Never tested. No Lucene fixture combines a replacement with empty beginchars (checkcompoundpattern2.aff, checkcompoundpattern3.aff), nor, per the devil's advocate, do Hunspell 1.7.2's own fixtures.

The strongest case for "intended" (the cross-model reviews in the hunt rated it weak, the 2026-11-01 re-review moderate; the cross-model fact check called it strong as a "noise / won't-fix" case and moderate as deliberate design): assertions are debug checks, and production gets C Hunspell's answer for the repro and the Dutch words checked; a 2026-10-29 audit classed it as assertion-only; no dictionary uses the combination on purpose, and it is untested in both Lucene and C Hunspell; the dictionary is trusted configuration; the Hunspell javadoc does not promise full compatibility with C Hunspell. The FULLSTRIP side case gives a different assertions-off answer, but the two C Hunspell versions disagree there.

Why it fails as an "intended" argument: nothing documents the combination as unsupported, and the broken invariant is internal to Lucene; a stock dictionary that Lucene's own regression build pins reaches it with an ordinary word plus #; maintainers guarded the same assertion at another caller rather than accept it; and the guard is one comparison that only drops candidates whose second part would be empty (the hunspell tests pass; in the FULLSTRIP side case it turns true into C Hunspell 1.7.2's false). Verdict: not intended; a missing bound check of low impact.

Fix A was run through gradle in a scratch worktree of 645d4fe14a20; B, C and D are alternatives, not tested.

One guard in checkCompoundPatternReplacements (Hunspell.java:255), right after the break is computed.

         int breakPos = pos + pattern.endLength();
+        if (breakPos >= expanded.length) {
+          continue; // the replacement ends the word: no second part
+        }
         Root<CharsRef> stem =
             findStem(expanded.chars, expanded.offset, breakPos, originalCase, context);

Trade-offs

• It only drops candidates whose second part would be empty. In the recorded FULLSTRIP side case that changes Lucene's assertions-off answer from true to false, like C Hunspell 1.7.2 (1.3.2 says true); with the stock Dutch dictionary it keeps the answers Lucene and C Hunspell 1.7.2 already give.
• Sibling bug LFV-377 (expandReplacement at later compound boundaries) is a separate fix; with its fix alone this failure remains, with both the tests pass. On stock code the guard only matters at the first boundary: at later ones the expanded word still carries the earlier parts, so its second part is never empty and the guard compares in a different frame. After LFV-377's fix the expanded word is relative to the current part, and the guard compares in the same frame.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (2026-11-01): the repro fails on stock main with -Ptests.asserts=true and passes with =false; with A it passes with both.
• With A and the repro in place: TestSpellChecking 53 tests, 0 failures; the whole org.apache.lucene.analysis.hunspell test package with assertions on, 47 classes, 159 tests, 0 failures, 13 skipped (TestPerformance, TestAllDictionaries and TestHunspellRepositoryTestCases need external dictionary checkouts; TestAllDictionaries.testMaxPrologueNeeded is also @Ignored); :lucene:analysis:common:check -x test (google-java-format, ecj lint, forbidden APIs) passes.
• Earlier, the same guard: the finder's one-test class (two control assertions plus foz) and the 146 tests of Lucene's hunspell test classes on a9779ef3c193; an independent 5-test class 5/5; the cross-model reviewer's own tests; the devil's advocate's probes; the guard shadowed onto the 645d4fe14a20 jars.

Use expanded.length - breakPos < dictionary.compoundMin as the skip condition, like the regular loop.

Trade-offs

Changes which simplified forms are accepted when the second part is short; hunspell(5) itself notes that COMPOUNDMIN "doesn't work correctly with the compound word alternation", so the right bound is unclear. Suggested by the devil's advocate; not tested; needs C Hunspell comparisons.

if (remainingLength == 0) return false; in checkCompoundsAfter, right after remainingLength is computed (Hunspell.java:272).

Trade-offs

Protects every caller that builds an empty tail, but hides where it came from. Not tested.

Replace the assertion with an early return null.

Trade-offs

Not recommended: it removes the check LUCENE-9825 added on purpose and LUCENE-9833 kept, so other callers' mistakes would go unnoticed.

Method: non-formal (cross-implementation differential testing). A hunt unit generated small synthetic compounding dictionaries from a grammar (CHECKCOMPOUNDPATTERN with and without replacements, empty endchars or beginchars written with flags), spell-checked every word of up to 6 letters over a 3-letter alphabet plus 2- and 3-part concatenations with both Lucene and C Hunspell 1.7.2, and compared the answers. Lucene ran with the fix sketches of two already-known hunspell bugs applied, so that their differences would not drown new ones; the failure then reproduced on the stock jar.

Lucene threw AssertionError on one generated dictionary and on 23 more from further seeds. A minimiser and hand reduction turned one into a public-API JUnit test. It was then reviewed by a reviewer from another model family (OpenAI GPT-5.6) in the hunt and again on 2026-11-01, reproduced independently, argued against by an intended-behaviour devil's advocate, judged, traced through history, audited and cut to the one-test repro, re-verified on 645d4fe14a20.