All findings · Lucene finding · LFV-375 · hunspell-complexprefixes-third-prefix-recursion

Hunspell COMPLEXPREFIXES keeps stripping prefixes after the second one: three-prefix words pass and an empty prefix recurses until StackOverflowError

With a COMPLEXPREFIXES Hunspell dictionary (at most two prefixes per word), Lucene keeps stripping prefixes after the second: the three-prefix word cbax is accepted, and an empty prefix makes spell("bbx"), or stemming the correct word bx while indexing, recurse until StackOverflowError, which closes the IndexWriter. A one-line regression since 9.5.0.

Summary

Priority
Low, rank 682With a custom COMPLEXPREFIXES dictionary that chains prefixes, ordinary tokens throw StackOverflowError and close the IndexWriter, the same custom-config writer-closing profile as LFV-121 and LFV-281.
Impact
crash
Since
9.5.0 (#975, LUCENE-10626; merged to main 2022-07-05)introduced 2022-07-05 · about 4 years
Affected
9.5.0 to 10.6.0 and main
Status
documented
Found by
coverage-guided fuzzing (Jazzer)

AI-assisted page: this finding was first signalled by code reading or testing and then checked with AI tools; the checks it went through, and what each found, are under Prior reports, history and checks. Not yet triaged by a human. Snapshot of 2026-11-06.

The finding in 30 seconds

Under COMPLEXPREFIXES a word may carry two prefixes. After the Stemmer strips the second one, it is told to look for another prefix instead of a suffix, so it strips a third one, or the same empty prefix again and again.

Hunspell dictionaries in three steps

  1. A Hunspell dictionary is two files: the .dic lists roots with flags (x/A), the .aff lists affix rules per flag (PFX A 0 a . means: a root with flag A may take the prefix a, so x gives ax).
  2. Flags after a slash on a rule are its continuation class: PFX A 0 a/B . lets a prefix of class B go in front of a, so PFX B 0 b . gives bax. A 0 affix is the empty string.
  3. Normally a word may carry two suffixes and one prefix. COMPLEXPREFIXES swaps that: two prefixes and one suffix, never a third prefix.
Summary card of the bug. Left: the dictionary COMPLEXPREFIXES, PFX A 0 0/B (a zero-length prefix that lets a B prefix precede it), PFX B 0 b, root x/A; b + empty prefix + x = bx is a word. spell("bbx"): after b and the empty prefix, Stemmer.java:640 sets doPrefix = true, so the empty prefix comes off again and again: StackOverflowError. Right: expected (Lucene 9.4.2; main with the fix) spell("bbx") = false and getRoots("bx") = [x]; actual (Lucene main; 9.5.0 to 10.5.2): both throw StackOverflowError, so stemming the correctly spelled bx crashes. Notes: with HunspellStemFilter, IndexWriter.addDocument on bx closes the writer and the next call throws AlreadyClosedException; with three chained prefixes (a/B, b/C, c on x/A) spell("cbax") = true and getRoots = [x], while 9.4.2 and the fix give false and [] and C Hunspell rejects cbax; C Hunspell 1.3.2 and 1.7.2 accept bx and reject bbx, and getAllWordForms("x") = [bx, x] stops at two prefixes; with the fix (doPrefix = false at Stemmer.java:640) every call returns the expected value. A box explains why, and the footer gives the affected versions (introduced by #975) and the narrow reach.
The two repro dictionaries: what native Hunspell accepts, and how Lucene's Stemmer strips a third prefix from cbax and the empty prefix from bx again and again. Select the still to open it at full size. Generated with AI, checked against the code.

The finding, step by step

  1. You load a dictionary with COMPLEXPREFIXES and chained prefixes: the root x takes a, a allows b in front of it, b allows c. So ax and bax are words; cbax would need three prefixes.
  2. You expect spell("cbax") to be false, as in native Hunspell 1.7.2 and Lucene 9.4.2. Instead it is true, getRoots("cbax") is [x], and the analysis lists two prefixes, c and a: the b silently disappears.
  3. A second dictionary makes the inner prefix empty: PFX A 0 0/B adds nothing and only lets b (PFX B 0 b) go in front. Now x and bx are words. spell("bbx") throws StackOverflowError, and so does getRoots("bx") of the correctly spelled bx, even on a 512 MB stack.
  4. With HunspellStemFilter in the analyzer, addDocument of the text bx throws the same Error: the IndexWriter closes itself, the next addDocument throws AlreadyClosedException, and only the 1 committed document survives.
  5. Why: after the second prefix the Stemmer sets doPrefix = true, under the comment "go look for another suffix", and looks for yet another prefix. A third prefix overwrites the second; an empty prefix leaves the word unchanged, so the same call repeats forever. Up to 9.4.2 the line was false.

Who is affected

Only applications that load a custom Hunspell dictionary with COMPLEXPREFIXES and chained prefix rules (spell checking, getRoots, analyzeSimpleWord, HunspellStemFilter); no surveyed public dictionary has that shape. Once it is loaded, ordinary short tokens trigger it: silent over-stemming, or a StackOverflowError that closes the IndexWriter or fails the query parse. Affected: 9.5.0 to 10.6.0 and main.

Mechanism, terms and numbers1,011 words

COMPLEXPREFIXES switches Hunspell to "twofold prefix stripping (but single suffix stripping)" (hunspell(5), 1.7.2): a word may carry at most two prefixes. In Stemmer.applyAffix, the branch that runs after the second prefix has been stripped sets doPrefix = true (Stemmer.java:639-641 on main 645d4fe14a20), although the comment beside it says "we took away the second prefix: go look for another suffix". So removeAffixes looks for yet another prefix. A third prefix overwrites the inner-prefix slot, the recursion depth is recomputed from the slots and stays 1, and the same branch runs again. With prefixes a, b, c chained on root x, spell("cbax") is true and getRoots("cbax") is [x]. With a zero-length prefix (PFX A 0 0/B), stripping it returns the same word, so the call repeats with identical arguments until StackOverflowError: spell("bbx"), and getRoots("bx") of the correctly spelled bx, overflow, also on a 512 MB stack. Inside IndexWriter.addDocument the Error closes the writer. Lucene up to 9.4.2 had doPrefix = false here, and native Hunspell rejects both words without error. The change came with #975 (LUCENE-10626, a feature PR, "add tools to aid dictionary editing"); setting the line back to false fixes both symptoms and keeps all existing hunspell tests green.

Terms

  • Outer / inner prefix (Lucene's naming): the outer prefix is written first and stripped first (b in b+a+x); the inner prefix is the one next to the root (a).
  • COMPLEXPREFIXES: hunspell(5) (1.7.2) "Set twofold prefix stripping (but single suffix stripping) eg. for morphologically complex languages with right-to-left writing system"; "Zero stripping or affix are indicated by zero". Lucene's parser maps a 0 affix to the empty string (Dictionary.java:743-746).

The two repro dictionaries (root x/A in both)

COMPLEXPREFIXES             COMPLEXPREFIXES
PFX A Y 1                   PFX A Y 1
PFX A 0 a/B .               PFX A 0 0/B .     (empty prefix, passes on B)
PFX B Y 1                   PFX B Y 1
PFX B 0 b/C .               PFX B 0 b .
PFX C Y 1
PFX C 0 c .
words: x, ax, bax           words: x, bx
not a word: cbax            not a word: bbx

C Hunspell 1.7.2: x 1, ax 1, bax 1, cbax 0 and x 1, bx 1, bbx 0, no error.

Mechanism (main 645d4fe14a20, Stemmer.java)

  1. removeAffixes (:390) tries every prefix that matches the start of the word, skipping only the one equal to outerPrefix (:420), and calls applyAffix (:431). stripAffix returns the same array when the rule strips nothing (:524), so an empty prefix leaves word, offset and length unchanged.
  2. applyAffix (:588) stores the first prefix in outerPrefix and every later one in innerPrefix (:603-604), so a third prefix overwrites the second. It looks the rest up as a root (:612), then decides whether to recurse:
int recursionDepth =
    (outerSuffix >= 0 ? 1 : 0) + (innerPrefix >= 0 ? 2 : outerPrefix >= 0 ? 1 : 0) - 1;   // :619-620
if (dictionary.isCrossProduct(affix) && recursionDepth <= 1) {                            // :621
  ...
  } else {
    if (prefix && dictionary.complexPrefixes) {
      doPrefix = true;                                   // :640  the bug
      // we took away the second prefix: go look for another suffix
    } ...
  return removeAffixes(word, offset, length, doPrefix, outerPrefix, innerPrefix, outerSuffix, processor); // :650
  1. After two prefixes the depth is 0 + 2 - 1 = 1; after a third it is still 1, because it is computed from the two slots, not counted. doPrefix = true makes removeAffixes look for another prefix, so the step repeats while some prefix other than outerPrefix matches.
  2. cbax: c becomes the outer prefix, b the inner one, then a overwrites b, leaving x. The root carries the last affix's flag A (:269), and the outer-prefix check (:271-277) is skipped because there is an inner prefix: accepted, reported as prefixes [C, A].
  3. bbx: b is the outer prefix, leaving bx; the empty A is the inner one, word still bx; depth 1, doPrefix = true; the empty A matches again (it is not outerPrefix), strips nothing and calls applyAffix with exactly the same arguments, forever. For bx, the root x is found on the way: spell() stops at the first root (Hunspell.java:179-181), so it returns true, but getRoots and analyzeSimpleWord collect every analysis and overflow.
  4. Before #975 (9.4.2 Stemmer.java:601-623) this branch set doPrefix = false and recursed with an explicit recursionDepth + 1, so after the second prefix only a suffix was tried and depth 2 never recursed again. #975 replaced the explicit depth with the slot formula and flipped the line to true.

A third shape

A prefix rule that strips more than it adds (PFX B ab 0: removing it puts ab back) makes the candidate word grow level after level only if it can be stripped again and again, which after the second prefix means as the inner prefix. In the study's simple dictionary of this kind (PFX A 0 0/B, PFX B ab 0) it is not: B becomes the outer prefix, which is never stripped again (:420), so the word stays abx and the stack overflows through the same empty-prefix loop as bbx, at every stack size up to 64 MB (an instrumented copy of the Stemmer measured a longest candidate of 3 characters). The fuzzer-found dictionary has several prefix rules of one class (W), among them PFX W uob 0, which strips letters and adds none; there the word does grow: to 5,042 characters before StackOverflowError on a 1 MB stack, to 55,745 before OutOfMemoryError on an 8 MB stack. On stacks of 8 MB or more it ended in OutOfMemoryError: Java heap space (1 GB heap) in 21 of 22 calls; the 22nd (10.5.2, 8 MB stack) ended in StackOverflowError after 7 s. With the fix every probe returns normally.

Exposure and likelihood381 words
  • Who. Users of Hunspell.spell, suggest, getRoots, analyzeSimpleWord and HunspellStemFilter / HunspellStemFilterFactory (@lucene.experimental) with a dictionary that sets COMPLEXPREFIXES and gives prefix rules continuation classes. Hunspell support is opt-in (module analysis-common) and the dictionary is supplied by the application. Deterministic; assertions on or off.
  • Three prefixes deep: silent. cbax passes the spell checker and is indexed and searched as x; in the study's probe a query for x matched 2 documents instead of 1, and a deny clause on x also excluded the cbax document.
  • A zero-length prefix that passes on another prefix's flag: loud. Ordinary tokens throw StackOverflowError, including correctly spelled words once they are stemmed (bx); by code, in spell() every word of two or more letters that starts with b, except the correctly spelled bx itself (the study ran bz and bhello).
  • What the Error does. Inside IndexWriter.addDocument it closes the writer, as documented for any Error (IndexWriter.java:162-165): uncommitted documents are lost, the committed index stays intact (this doc's probe: 1 committed document, then AlreadyClosedException). In the study, with 4 threads sharing one writer, one such document got 273-435 of the other threads' documents rejected; in a separate single-writer run (with the wider trigger below), CheckIndex was clean after the close, with the 1 committed document, and a new writer worked. QueryParser.parse("bx") with the stemming analyzer and suggest("bbx") (both timeout policies) throw it too.
  • Not a deep-but-finite recursion. The arguments repeat exactly; it still overflows on a 512 MB and on a 256 MiB thread stack.
  • Reach. Narrow. Of 191 surveyed public .aff files (LibreOffice dictionaries and wooorm/dictionaries) only ckb sets COMPLEXPREFIXES, and none of its 1,812 prefix rules is zero-length or has a continuation class, so it cannot recurse. Zero-length prefixes with continuation classes do occur in public dictionaries (fr_FR, de, bn_BD, ka), only not with COMPLEXPREFIXES.
  • Wider with the sibling LFV-642 (same commit; isAffixCompatible accepts any cross-product prefix after a prefix): the empty prefix need not be chained. An ordinary two-level chain plus an unrelated PFX Z 0 0 . overflows on spell("bax"), a correctly spelled two-prefix word; this fix removes it.
  • Released: 9.5.0 (January 2023) to 9.12.3 and 10.0.0 to 10.6.0, and main.

Minimal repro

One JUnit class in lucene/analysis/common, public API only, two tiny dictionaries built in memory, one method per symptom: spell("cbax") must be false, and spell("bbx") must return false instead of overflowing. Both fail on main with assertions on and off and pass with the fix.

TestComplexPrefixesAtMostTwo.java (analysis-common, 2 methods)36 lines
package org.apache.lucene.analysis.hunspell;

import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.util.List;
import org.apache.lucene.tests.util.LuceneTestCase;

public class TestComplexPrefixesAtMostTwo extends LuceneTestCase {

  private static Hunspell hunspell(String aff) throws Exception {
    return new Hunspell(
        new Dictionary(
            new ByteArrayInputStream(aff.getBytes(StandardCharsets.UTF_8)),
            List.of(new ByteArrayInputStream("1\nx/A\n".getBytes(StandardCharsets.UTF_8))),
            false,
            SortingStrategy.inMemory()));
  }

  public void testThirdPrefixRejected() throws Exception {
    Hunspell h =
        hunspell(
            "COMPLEXPREFIXES\nPFX A Y 1\nPFX A 0 a/B .\nPFX B Y 1\nPFX B 0 b/C .\n"
                + "PFX C Y 1\nPFX C 0 c .\n");
    // COMPLEXPREFIXES allows two prefixes, so cbax (c + b + a + x) must be rejected
    assertFalse("expected spell(\"cbax\") == false, got true", h.spell("cbax"));
  }

  public void testZeroLengthPrefixTerminates() throws Exception {
    Hunspell h = hunspell("COMPLEXPREFIXES\nPFX A Y 1\nPFX A 0 0/B .\nPFX B Y 1\nPFX B 0 b .\n");
    try {
      assertFalse("expected spell(\"bbx\") == false, got true", h.spell("bbx"));
    } catch (StackOverflowError e) {
      throw new AssertionError("expected spell(\"bbx\") == false, got StackOverflowError", e);
    }
  }
}
Command
./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.hunspell.TestComplexPrefixesAtMostTwo -Ptests.seed=DEADBEEF
Expected
spell("cbax") and spell("bbx") both return false, as in native Hunspell and Lucene 9.4.2.
Observed
spell("cbax") returns true, and spell("bbx") throws StackOverflowError (assertions on and off).

All files of this repro: repro/LFV-375/

More files, environment, full output369 words

TestComplexPrefixesAtMostTwo (package org.apache.lucene.analysis.hunspell; public API only: the Dictionary(InputStream, List<InputStream>, boolean, SortingStrategy) constructor and Hunspell.spell). No files, no randomness, one thread. Two methods because the one line has two symptoms of different kinds: a wrong true (three prefixes) and a StackOverflowError (zero-length prefix). The catch only turns the Error into an assertion message that states expected versus actual; the Error stays chained as the cause.

It passes :lucene:analysis:common:checkGoogleJavaFormat. Add the standard ASF license header before committing it to the Lucene tree. With the fix it passes unchanged, so it doubles as the regression test.

The test uses the SortingStrategy constructor, which exists from 9.9.0 on. Through the standalone runner the unmodified file fails 2 of 2 on the 10.5.2 and 10.6.0 release jars. A portable copy that builds the Dictionary through the older constructor and is otherwise unchanged fails 2 of 2 on 9.5.0, 9.12.3, 10.0.0, 10.5.2 and 10.6.0 and passes on 9.3.0 and 9.4.2.

Environment

lucenemain 645d4fe14a20 (11.0.0-SNAPSHOT): fails; release jars 10.5.2 and 10.6.0: fail (the test needs 9.9.0 or later; a portable copy with the older Dictionary constructor fails on 9.5.0, 9.12.3 and 10.0.0 and passes on 9.3.0 and 9.4.2)
jdkOpenJDK 25.0.3
osLinux aarch64
Command (environment)
cp $REPRO/TestComplexPrefixesAtMostTwo.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/hunspell/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.hunspell.TestComplexPrefixesAtMostTwo -Ptests.seed=DEADBEEF
# add -Ptests.asserts=true or -Ptests.asserts=false: it fails both ways

Observed output

# gradle, fresh worktree at main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (the same with false)
TestComplexPrefixesAtMostTwo > testZeroLengthPrefixTerminates FAILED
    java.lang.AssertionError: expected spell("bbx") == false, got StackOverflowError
        Caused by:
        java.lang.StackOverflowError
            at org.apache.lucene.analysis.hunspell.WordStorage.lookupWord(WordStorage.java:109)
            at org.apache.lucene.analysis.hunspell.Dictionary.lookupWord(Dictionary.java:284)
            at org.apache.lucene.analysis.hunspell.Stemmer$2.processStemCandidate(Stemmer.java:262)
            at org.apache.lucene.analysis.hunspell.Stemmer.applyAffix(Stemmer.java:612)
            at org.apache.lucene.analysis.hunspell.Stemmer.removeAffixes(Stemmer.java:431)
            at org.apache.lucene.analysis.hunspell.Stemmer.applyAffix(Stemmer.java:650)
            at org.apache.lucene.analysis.hunspell.Stemmer.removeAffixes(Stemmer.java:431)
            ... (applyAffix:650 / removeAffixes:431 repeating; the innermost 1-4 frames differ from run to run)
TestComplexPrefixesAtMostTwo > testThirdPrefixRejected FAILED
    java.lang.AssertionError: expected spell("cbax") == false, got true
2 tests completed, 2 failed

# public-API probe on the main 645d4fe14a20 jars, plain java (-ea identical)
cbax spell=true getRoots=[x] analyzeSimpleWord=[... prefixes=[C(id=2), A(id=0)] ...]
bx   spell=true getRoots=StackOverflowError analyzeSimpleWord=StackOverflowError
bbx  spell=StackOverflowError getRoots=StackOverflowError
addDocument("x") -> ok ; addDocument("bx") -> StackOverflowError, writer.isOpen=false ; addDocument("x") -> AlreadyClosedException
committed docs after reopen: 1
# the same probe with the fixed Stemmer.java first on the classpath
cbax spell=false getRoots=[] ; bx spell=true getRoots=[x] ; bbx spell=false getRoots=[] ; all three addDocument calls ok

Why it looks unintended

  • The changed line sits under the comment "go look for another suffix"; up to 9.4.2 it was false.
  • hunspell(5) defines COMPLEXPREFIXES as twofold prefix stripping; native Hunspell rejects cbax and bbx without error.
  • Lucene's own API says two: AffixedWord.getPrefixes() returns "at most two", and the word-form generator from the same PR never produces cbax.
The full argument262 words
  • The comment beside the line. Stemmer.java:639-641: doPrefix = true; // we took away the second prefix: go look for another suffix. git blame gives the comment to LUCENE-5483 (2014) and the true to #975 (2022).
  • The format's contract. hunspell(5) (the 1.7.2 manual page): "COMPLEXPREFIXES Set twofold prefix stripping (but single suffix stripping)". C Hunspell 1.7.2 and 1.3.2 reject cbax and bbx and do not crash.
  • Lucene's own statements of the limit. AffixedWord.getPrefixes(): "@return the list of prefixes applied to the stem, at most two, outermost first" (AffixedWord.java:54, added by the same #975 commit). The generator from #975 refuses a third affix of a kind (if (size == 2) return false;, now WordFormGenerator.java:376): getAllWordForms("x") is [ax, bax, x]. The Dictionary comment: "2-stage prefix+1-stage suffix instead of 2-stage suffix+1-stage prefix" (Dictionary.java:368-370). CHANGES for LUCENE-5483: "COMPLEXPREFIXES now work correctly according to the hunspell algorithm". Robert Muir on #6547 (2014): with COMPLEXPREFIXES "max 2 prefixes, max 1 suffixes".
  • Nothing in #975 points to an intended change in stemming. It is listed under 9.5.0 "New Features" as "Hunspell: add tools to aid dictionary editing: analysis introspection, stem expansion and stem/flag suggestion"; it adds no COMPLEXPREFIXES test, and its review never mentions the Stemmer or doPrefix. The only upstream COMPLEXPREFIXES test (TestComplexPrefix) covers one and two prefixes and passes with and without the fix.
  • Lucene claims native behaviour. Hunspell "can be used in place of native Hunspell for many languages"; its listed exceptions (Hungarian, characters outside the BMP, PHONE) do not include COMPLEXPREFIXES.
The devil's-advocate check117 words

The strongest case for "intended": doPrefix = true was a deliberate step towards longer prefix chains, and the Java API itself never promises a two-prefix limit.

Why it fails: the comment was kept unchanged; the same commit documents "at most two" prefixes in AffixedWord and caps its generator at two; the two slots (outerPrefix, innerPrefix) cannot even represent a third prefix, which is why cbax is reported as [C, A]; native Hunspell rejects these words; and a recursion that never ends on a five-line dictionary cannot be a design choice. An independent reviewer from another model family (OpenAI GPT-5.6) and an intended-behaviour devil's advocate both rated the "intended" case weak. Verdict: a regression.

Suggested fix

Recommended Restore doPrefix = false after the second prefix (one line)

Set doPrefix = false in that branch of Stemmer.applyAffix, as its comment says and as 9.4.2 did: after two prefixes only a suffix is looked for. Validated: the repro passes with assertions on and off; the hunspell test package runs 160 tests, 0 failures. The sibling LFV-642 needs its own fix; one PR can carry both.

--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Stemmer.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Stemmer.java
@@ -637,7 +637,7 @@
         }
       } else {
         if (prefix && dictionary.complexPrefixes) {
-          doPrefix = true;
+          doPrefix = false;
           // we took away the second prefix: go look for another suffix
         } else if (prefix || dictionary.complexPrefixes || !dictionary.isSecondStageSuffix(flag)) {
           return true;
All options and trade-offs3 options · 527 words

Option A was applied in a fresh scratch worktree at 645d4fe14a20 and run through gradle. Option B was validated in the sibling LFV-642's brief on main a9779ef3c193. Option C is a sketch only.

Restore doPrefix = false after the second prefix (one line) (validated by the checks) (recommended)

After two prefixes removeAffixes then looks only for a suffix, as the comment says and as 9.4.2 did. COMPLEXPREFIXES allows one suffix, so two prefixes plus a suffix still work (TestComplexPrefix asserts poneptwofoosuf -> foo and passes). No prefix is stripped after the second one, so the depth-1 branch cannot repeat: at most two prefixes (empty or not) and one suffix are stripped on any path.

--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Stemmer.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Stemmer.java
@@ -637,7 +637,7 @@
         }
       } else {
         if (prefix && dictionary.complexPrefixes) {
-          doPrefix = true;
+          doPrefix = false;
           // we took away the second prefix: go look for another suffix
         } else if (prefix || dictionary.complexPrefixes || !dictionary.isSecondStageSuffix(flag)) {
           return true;

Trade-offs

  • None known for correct dictionaries; it only removes recursion. No benchmark was run.
  • Dictionaries that relied on three-prefix words since 9.5.0 lose them (native Hunspell never accepted them); an index built with HunspellStemFilter on an affected version keeps such words stemmed until reindexed.
  • It does not fix the sibling LFV-642 (isAffixCompatible, Stemmer.java:566), and LFV-642's fix does not fix this one.

What the checks validated

  • Fresh scratch worktree at 645d4fe14a20, OpenJDK 25.0.3, ./gradlew --no-daemon -Ptests.seed=DEADBEEF (2026-10-30): without the fix the minimal test fails 2 of 2 with -Ptests.asserts=true and false; git apply succeeds; with the fix :lucene:analysis:common:checkGoogleJavaFormat and the minimal test pass, assertions on and off.
  • --tests 'org.apache.lucene.analysis.hunspell.*' with the fix: 160 tests, 0 failures, 13 skipped; the stock package without the new test: 46 suites, 158 tests, 0 failures, the same 13 skipped (TestAllDictionaries 2, TestHunspellRepositoryTestCases 1, TestPerformance 10 need external dictionaries).
  • The public-API probe with the fixed Stemmer.java: cbax and bbx rejected, bx stemmed to x, all three addDocument calls succeed.
  • Earlier, the study ran the same line against Lucene's checker over the 131 C Hunspell 1.7.2 fixtures (unchanged: 122 pass, the same 9 known failures) and the 11 fuzzer crash inputs (0 of 11 crash).

Option A plus the sibling LFV-642's fix in one PR (validated by the checks)

LFV-642 restores the per-rule continuation check for a second prefix in isAffixCompatible (Stemmer.java:566, +7/-1). Both come from #975's rewrite of the same method family.

Trade-offs

Good if you file "#975 broke COMPLEXPREFIXES twofold-prefix handling" as one issue; reviewers may prefer two small PRs, and each fix leaves the other bug in place.

What the checks validated

In LFV-642's brief, through gradle on main a9779ef3c193: the hunspell package with both repros ran 48 classes, 164 tests, 0 failures.

Also guard the prefix loop

In removeAffixes, strip prefixes only while innerPrefix < 0 (if (doPrefix && innerPrefix < 0 && dictionary.prefixes != null), Stemmer.java:400).

Trade-offs

Would stop a third prefix even if another path set doPrefix = true again; with A it is redundant, and alone it leaves the misleading true under the "another suffix" comment. Not run.

Downstream note

A product that lets less-trusted users install their own Hunspell dictionaries and then analyzes other users' text with them would inherit the crash; Lucene treats the dictionary as trusted configuration.

How it was found

Coverage-guided fuzzing (Jazzer) of Lucene's Hunspell code with .aff/.dic files built from a grammar: 941,807 runs in about 21 minutes gave 11 crash reports, all in the Stemmer's affix recursion. They were minimised by hand, compared with native Hunspell and turned into a public-API JUnit on the stock jars.

The method in full133 words

Method: coverage-guided fuzzing (the hunt's non-formal tier; the secondary method of the hunspell hunt unit). A Jazzer target built .aff/.dic pairs from a grammar and called spell() and suggest(); a sibling compound-pattern fix was shadowed so that bug would not mask others.

  • 941,807 runs in 1,235 s gave 11 crash reports (10 StackOverflowError, 1 OutOfMemoryError), all in Stemmer.removeAffixes / applyAffix. Replay: 11 of 11 crash without the fix, 0 of 11 with it.
  • Minimised by hand to the two dictionaries, triaged against hunspell(5) and C Hunspell 1.7.2, then a JUnit on the stock jars; the credibility audit's no-noise gate reduced it to the two-method test.
  • While verifying it, the intended-behaviour devil's advocate noticed a second regression from the same commit (spell("cax")), now the separate LFV-642.

Prior reports, history and checks

Prior reports

No prior report found (refreshed 2026-10-30): 98 tool queries over GitHub, the mailing lists, JIRA, downstream projects and the web, 44 manual GitHub searches, and the file lists of all 408 open PRs. Not fixed upstream.

Searches4 searches

Verdict new: no report by anyone. Keyword search cannot prove absence, comment text was searched only through the search index (plus direct reads of the key threads), and private channels cannot be searched. Closest items, none this bug: #975 / #11662 (introduced it; not discussed there), #6546 / #6547 (2014, the two-prefix contract), #10842, #6902, #6879, #6880, #10710 (older hunspell work), #16919 (AffixCondition.java only), hunspell/hunspell#667 and #497 (the reference engine stops at two prefix levels).

What was searched, when, and what turned up
WhereWhenHowFound
apache/lucene issues, PRs and commits; Elasticsearch, OpenSearch, Solr; dev@, java-user@, issues@, commits@; JIRA; web2026-10-30a scripted prior-report search, 98 queries (40 GitHub issue/PR, 4 commit, 4 downstream, 32 mailing-list, 8 JIRA, 10 web), 233 distinct hitsno report; the medium-scoring hits are history (#10710, #6546, #6547, #10791, #10731, #10736) or unrelated
GitHub, apache/lucene and downstream2026-10-3044 manual queries (COMPLEXPREFIXES, hunspell StackOverflowError, second/third prefix, zero affix, doPrefix, removeAffixes, applyAffix, isAffixCompatible, regression 9.5, ...), recent-activity searches, GitHub advisoriesno report
open apache/lucene PRs2026-10-30file lists of all 408 open PRs (GraphQL, every file page of the 3 PRs with more than 100 files)only #16422 touches hunspell (Dictionary.java only)
apache/lucene issues and PRs since 2022-07-01; lists.apache.org2026-10-26title/body scan of 10,433 items (158 mention hunspell, all titles read); hunspell mails on issues@, dev@ and java-user@ readno report

History

Introduced by #975 (LUCENE-10626, d537013e708, merged to main 2022-07-05), which reached 9.x as a cherry-pick first released in 9.5.0. 8.9.0 to 9.4.2 pass; every release from 9.5.0 to 10.6.0 and main fails. Not fixed upstream.

Versions tested6 rows

Introduced by d537013e70872015364c745e5f320727efc034b7, "LUCENE-10626 Hunspell: add tools to aid dictionary editing" (#975, donnerpeter, one approving review), merged to main 2022-07-05. In the Stemmer.applyAffix rewrite the "we took away the second prefix" branch went from doPrefix = false to doPrefix = true, and the explicit recursion depth was replaced by one recomputed from the affix slots. branch_9x got it only as the cherry-pick 3b763af12f5 (committed 2023-01-13), so 9.3.0 and 9.4.x, released after the main commit, are not affected. Swapping only the hunspell package source between each commit and its parent flips the result, and so does flipping only this line. Upstream main 2f424fd428da (2026-10-30) still has doPrefix = true. The unmodified minimal test needs 9.9.0 or later (SortingStrategy); older releases ran a portable copy that differs only in the Dictionary constructor. Of the 8.x releases with the Hunspell API, the study's driver ran only 8.9.0 and 8.11.4.

Versions and branches executed, with results
Version or commitVerdictResult
main 645d4fe14a20 affectedthe minimal test fails 2 of 2 through gradle, assertions on and off
10.6.0 newest releasefails 2 of 2, -ea and -da (portable test; the unmodified test with -ea too)
10.0.0 to 10.5.2 affectedevery release fails the study's driver (9 of 9 bug checks, controls OK); the portable minimal test fails on 10.0.0 and 10.5.2, the unmodified one on 10.5.2
9.5.0 to 9.12.3 first affected 9.5.0every release fails the study's driver; the portable minimal test fails on 9.5.0 (-ea, -da) and 9.12.3
8.9.0, 8.11.4, 9.0.0 to 9.4.2 last unaffected 9.4.2pass (the study's driver, controls OK; the portable minimal test on 9.3.0 and 9.4.2)
#975 d537013e708 and backport 3b763af12f5 vs their parents source swapthe commit's hunspell package fails, its parent's passes; 9.5.0's Stemmer.java with only this line set back passes

Independent checks

An independent reproducer, an intended-behaviour devil's advocate, an adjudication, an audit and a tie-break each re-ran or rewrote the test; reviews by a model from another family (OpenAI GPT-5.6, reading only) agreed on the root cause; this doc's gradle runs reproduce it on 645d4fe14a20 and validate the fix.

Checks and verdicts413 words

Every executing check wrote or re-ran its own JUnit on stock jars; every control passed, and native Hunspell rejected the over-accepted words in every check that ran it. The reviews by the other model family could only read code; their verdicts rest on the code and separate re-runs. Correction (fact check, 2026-10-30): the registry card in the Sources block still has two older statements. Its verification text says the fuzzer dictionary ended in OutOfMemoryError on 8 MB and 32 MB stacks in the second probe; one of those six calls (10.5.2, 8 MB) ended in StackOverflowError. It also ends with the security verdict of the 2026-10-26 assessment, which was re-evaluated on 2026-10-27: not security-sensitive. This doc's text is the corrected version.

Each independent check and its verdict
CheckVerdictWhat it found
finder candidate4-test JUnit (incl. HunspellStemFilter inside an IndexWriter) fails 4 of 4 on main (-ea, -da), 10.5.2 and 9.12.3; passes with the fix.
cross-model review, intended lens (OpenAI GPT-5.6) real, medium; intended argument weakRoot cause agreed; a separate run re-ran the repro: 4 of 4 fail.
Independent reproducer realOwn 10-test JUnit from another package: 9 of 10 fail (control passes) on main, 10.5.2, 9.12.3, 9.5.0; 9.4.0 passes; overflows with -Xss512m; native Hunspell 1.3.2 rejects cbax and bbx; 10 of 10 pass with the fix.
Root-cause lens (OpenAI GPT-5.6) real, highRoot cause agreed; separate runs: analyzeSimpleWord("bbx") overflows on main, not with the fix; 9.4.0 pass, 9.5.0 fail.
Intended-behaviour devil's advocate realC Hunspell 1.7.2 rejects cbax and bbx; the generator never produces cbax; found the cax sibling (LFV-642).
adjudication confirmed, highRe-ran the independent reproduction's test (-ea, -da), the finder's (-ea) and the independent reproduction's with the fix.
credibility audit keepRe-ran the test; a public-API probe from another package; found the wider zero-prefix trigger with LFV-642.
current-main reproducer and tie-break real, highThe adversary on current main could not run anything (procedural "not real"; root cause agreed, intended case weak). The tie-break wrote its own test with other dictionaries: 5 of 8 fail with 3 controls passing on main, 10.6.0 and 9.5.0; 9.4.2 passes; overflows on a 256 MiB stack; 8 of 8 pass with the fix.
this doc reproduced, fix validatedGradle in a fresh worktree without and with the fix, format check, the whole hunspell package; public-API probe including the IndexWriter.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,162 words

Finding record

LFV-375, snapshot of 2026-11-06

Registry title
Hunspell COMPLEXPREFIXES: after removing the second prefix the Stemmer keeps removing prefixes, so words with three prefixes are accepted and a zero-length prefix recurses until StackOverflowError (also in HunspellStemFilter, where it closes the IndexWriter)
Area
analysis / hunspell (Stemmer affix removal)
Priority
Low, rank 682: With a custom COMPLEXPREFIXES dictionary that chains prefixes, ordinary tokens throw StackOverflowError and close the IndexWriter, the same custom-config writer-closing profile as LFV-121 and LFV-281.
Impact class
crash
Impact note
Only applications whose hunspell dictionary combines COMPLEXPREFIXES with chained prefixes (prefix continuation classes) are hit: with three chained prefixes, spell(), getRoots() and HunspellStemFilter silently accept and stem words that need three prefixes, and with a zero-length chained prefix ordinary tokens (even a correctly spelled word, once it is stemmed) make spell(), suggest(), getRoots(), analyzeSimpleWord() and HunspellStemFilter throw StackOverflowError, which inside IndexWriter.addDocument closes the writer (committed data stays intact, uncommitted documents are lost) and in a query analyzer fails the query parse; no surveyed public dictionary has this shape.
Summary
You load a hunspell dictionary that sets COMPLEXPREFIXES (twofold prefix stripping, meant for morphologically complex right-to-left languages) and chains prefixes through continuation classes, and expect at most two prefixes per word, as hunspell(5), C Hunspell and Lucene up to 9.4.2 do. Instead, after removing the second prefix the Stemmer goes on removing prefixes: with prefixes a, b, c chained on root x, the three-prefix word cbax is accepted by spell() and stemmed to x, and with a zero-length prefix that only passes on a flag (PFX A 0 0/B), the same prefix is removed again and again without shortening the word, so spell("bbx"), getRoots("bx") and HunspellStemFilter throw StackOverflowError. Inside IndexWriter.addDocument that Error closes the writer, and every later addDocument throws AlreadyClosedException.
Introduced
d537013e70872015364c745e5f320727efc034b7 (apache/lucene#975, LUCENE-10626 'Hunspell: add tools to aid dictionary editing', a feature PR by donnerpeter with one approving review; it reached branch_9x as the backport 3b763af12f576b8b8120e98ae38b7a67461a99e8) [read: git blame and git log -G on Stemmer.java show it changed 'doPrefix = false' to 'doPrefix = true' in the second-prefix branch and kept the comment; exec: release bracket 9.4.2 pass / 9.5.0 fail, and swapping only the hunspell package source between each of the two commits and its parent flips the result, as does flipping only that line] (2022-07-05 (merged to main); 2023-01-13 (branch_9x backport, author date kept as 2022-07-05))
First affected version
9.5.0 [exec] (10.0.0 on the 10.x line); 9.3.0 and 9.4.0-9.4.2 were released after the main commit but before the backport and are not affected [exec]
Affected versions
9.5.0 - 9.12.3, 10.0.0 - 10.5.2, 10.6.0 (by reading) and main ea57360912ed [exec: all 26 releases listed in the follow-up (9.5.0, 9.6.0, 9.7.0, 9.8.0, 9.9.0-9.9.2, 9.10.0, 9.11.0, 9.11.1, 9.12.0-9.12.3, 10.0.0, 10.1.0, 10.2.0-10.2.2, 10.3.0-10.3.2, 10.4.0, 10.5.0-10.5.2) and main fail 9 of 9 bug checks with 8 of 8 controls OK; also with -da on 9.5.0, 9.12.3, 10.5.2 and main]; not affected: 8.9.0, 8.11.4 and 9.0.0-9.4.2 [exec, pass with controls OK]; 8.8.2 and older: no Hunspell spell API, and by code the second-prefix step recursed with doPrefix = false from LUCENE-5483 (4.8.0) on [read; the 8.8.2 stem-filter run was not informative because a control failed]; [read] upstream main, branch_10x and branch_10_6 still set doPrefix = true (2026-10-26)
Status
documented (brief, checks and a per-finding doc exist)
Verification
Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its “not real” is procedural), an adjudication that ruled it real, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by
non-formal
Upstream and prior reports
none found (2026-10-26, read-only, the prior-report record verdict new: a scripted prior-report search over GitHub issues, PRs and commits, downstream repositories (Elasticsearch, OpenSearch, Solr), the dev@, java-user@, issues@ and commits@ lists, LUCENE JIRA and the web; manual GitHub searches over apache/lucene, elastic/elasticsearch, opensearch-project/OpenSearch, apache/solr and hunspell/hunspell; the changed files of all 411 open apache/lucene PRs; a title/body scan of the 10,433 apache/lucene issues and PRs updated since 2022-07-01 (158 mention hunspell, all titles read); lists.apache.org issues@, dev@ and java-user@ hunspell mails read; SOLR JIRA; git history and CHANGES.txt; none the same bug). Origin, not a report: #975 (LUCENE-10626, issue #11662), whose review never mentions Stemmer or doPrefix. Contract context: #6546 (LUCENE-5483, 2014, added the doPrefix = false second-prefix step), #6547 (LUCENE-5484, a committer comment: with COMPLEXPREFIXES 'max 2 prefixes, max 1 suffixes'), hunspell/hunspell#667 and #497 (the reference implementation supports at most two prefix levels). The only open PR touching hunspell, #16422, changes Dictionary.java only. Not fixed upstream: upstream main, branch_10x and branch_10_6 still set doPrefix = true (2026-10-26).
Lucene commit tested
main 645d4fe14a20 (2026-10-28): the minimal JUnit through gradle in a fresh scratch worktree, stock (fails 2 of 2, assertions on and off) and with the one-line fix (passes; checkGoogleJavaFormat and the whole hunspell test package green), plus a public-API probe including HunspellStemFilter inside an IndexWriter, stock and with the fix; the minimal test on the 10.5.2 and 10.6.0 release jars (fail; it needs 9.9.0 or later), and a portable copy that differs only in the Dictionary constructor on 9.3.0 and 9.4.2 (pass) and 9.5.0, 9.12.3, 10.0.0, 10.5.2 and 10.6.0 (fail). The study's main() driver ran 8.9.0, 8.11.4 and every release from 9.0.0 to 10.5.2, and main ea57360912ed, whose hunspell package differs from 645d4fe14a20 only in AffixCondition.java.

Lucene and upstream sources

Finding record
  • lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Stemmer.java at ea57360912ed: :390-431 removeAffixes (:420 skips only outerPrefix, :431 calls applyAffix), :504 stripAffix, :588-655 applyAffix (:603-604 'else innerPrefix = affix', :619-620 recursionDepth recomputed from the affix slots, :638-641 the branch 'doPrefix = true; // we took away the second prefix: go look for another suffix', :650 recursive removeAffixes call)
  • lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/WordFormGenerator.java:376 ('if (size == 2) return false;', at most two same-kind affixes); lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Dictionary.java:368-370 (COMPLEXPREFIXES: '2-stage prefix+1-stage suffix instead of 2-stage suffix+1-stage prefix'); lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Hunspell.java:71-74 (spell javadoc), :305 getRoots, :314 analyzeSimpleWord, :574 and :586 suggest
  • lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:162-165 (javadoc: 'if you hit an Error ... IndexWriter will close itself ... Any subsequent calls will throw an AlreadyClosedException')
The finding in 30 seconds
  • lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Stemmer.java:298, :390-431, :504-524, :588-655 (read at 645d4fe14a20)
  • lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Hunspell.java:163-182 (findStem stops at the first root), :305-306 (getRoots), :314 (analyzeSimpleWord)
  • lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Dictionary.java:743-746 (zero affix -> empty string)
  • a Lucene clone releases/lucene/9.4.2:lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Stemmer.java:601-623 (doPrefix = false, recursionDepth + 1)
Who is affected
  • lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/HunspellStemFilter.java:39 (@lucene.experimental, read at 645d4fe14a20)
  • lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:162-165
Why it looks unintended
  • lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Stemmer.java:639-641; lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/AffixedWord.java:54; lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/WordFormGenerator.java:376; lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Dictionary.java:368-370; lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Hunspell.java:35-43
  • lucene/CHANGES.txt:3801 (9.5.0 New Features, LUCENE-10626), :2308 (10.0.0), :12562-12566 (LUCENE-5483)

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

About the diagrams

Values on screen come from the diagram probe on main 645d4fe14a20 (stock and with the fix shadowed in), on 10.5.2, 9.5.0 and 9.4.2, and from C Hunspell 1.3.2 and 1.7.2 on the same two dictionaries.

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.