Hunspell reads a dash inside [ ] as a range where an affix condition overlaps the strip

hunspell(5) says that inside the braces of an affix condition "Dash hasn't got special meaning", so [a-c] is the set {a, -, c}. Lucene's AffixCondition.compile splits a bracket condition into the part that overlaps the rule's strip and the part checked against the stem. It decides the strip part once, at dictionary load, with String.matches (java.util.regex) on the condition text without dash escaping (AffixCondition.java:108, :113, :120), while only the stem part goes through escapeDash. Java reads [a-c] as the range a..c, so such a rule is applied where Hunspell skips it, or dropped where Hunspell applies it; a reversed pseudo-range such as [c-a] makes Java throw, and the rule is silently turned into "never applies". With SFX A b y [^a-c] on the root ab, Hunspell.spell("ay") is false and HunspellStemFilter leaves ay unstemmed; C Hunspell accepts ay with the stem ab. It came with LUCENE-9806, a speed-only change, and fails on every release run from 8.9.0 to 10.5.2 and on main (the 9.x and 10.x releases in between have the same code); no surveyed public dictionary has an affected rule. The fix is three escapeDash(...) calls, evaluated only at load.

• Affix rule: a .aff line such as SFX A b y [^a-c]: for roots with flag A, strip b from the end, add y, if the condition [^a-c] holds. A prefix rule (PFX) works at the start of the word.
• Condition: a simplified pattern matched against the end (SFX) or start (PFX) of the root. hunspell(5): "Dot signs an arbitrary character. Characters in braces sign an arbitrary character from the character subset. Dash hasn't got special meaning, but circumflex (^) next the first brace sets the complementer character set." So [a-c] is {a, -, c} and [^a-c] is any other character.
• Strip part and stem part: the condition's positions that fall on the stripped letters are fixed for the rule, so Lucene decides them once at load; the other positions are checked against each candidate stem.

1. Dictionary calls AffixCondition.compile(kind, strip, condition, line) once per distinct condition/strip/kind at load (Dictionary.java:749, :760).
2. For a condition with [, . or - (AffixCondition.java:153-155), compile counts its character patterns (:105) and checks the strip part with String.matches on the text without dash escaping (only an unclosed [ is closed first, :98-102): the whole condition when it fits in the strip (:107-108), the leading patterns of a prefix condition (:113), the trailing patterns of a suffix condition (:120). A PatternSyntaxException becomes ALWAYS_FALSE (:124-125).
3. The stem part goes to regexpCondition, which compiles new RegExp(escapeDash(condition), ...) (:179); the comment above escapeDash reads "\"dash hasn't got special meaning\" (we must escape it)" (:186).
4. So within one rule, the strip part reads a-c as the range a..c and the stem part as three characters. Only a dash between two characters is affected: a dash first, last or after ^ is literal in Java too.

Worked example, SFX A b y [^a-c]: the condition has 1 pattern and the strip b has 1 letter, so :107 builds [^a-c].* and :108 evaluates "b".matches("[^a-c].*"), false because b is in a..c: the rule becomes ALWAYS_FALSE. With escapeDash the regex is [^a\-c].*, b matches, and the rule becomes ALWAYS_TRUE.

English-looking rules from the independent reproducer (C Hunspell 1.3.2 agrees with every expected value): SFX B y ied [^x-z] on happy makes Lucene reject happied; SFX E a o [z-a] on pizza makes it reject pizzo (Java throws on the reversed range); SFX A y ies [x-z] makes it accept happies and stem it to happy; PFX C y z [x-z] on yes accepts zes; SFX D ey ow [d-f]y on key accepts kow.

Each of the three calls is reached by an executed case: :108 by [a-c], [^a-c] and the reversed [c-a] (SFX A c y [c-a] on ac: ay rejected); :113 by PFX B b y [^a-c]a on baa (yaa rejected); :120 by SFX A b y a[^a-c] on aab (aay rejected); the no-dash control a[^ac] passes.

• Who. Spell checking (Hunspell.spell) and stemming (Stemmer, HunspellStemFilter) are affected by execution. Suggestions and word-form generation use the same condition objects, by code reading only.
• Opt-in. Hunspell is a separate analysis component with an application-supplied dictionary; no Lucene analyzer uses it by default.
• Effect. Silent and deterministic: each rule is decided once at load, so a word whose only derivation goes through that rule gets the wrong verdict and the wrong stem, with no error (a dictionary entry or another rule for the same word can mask it; the repro dictionaries have none). On a field analysed with HunspellStemFilter and the repro's dictionary, the query body:ab returns the documents ab and ax on main and ab and ay with the fix.
• Real dictionaries. None of 191 public .aff files (LibreOffice dictionaries and wooorm/dictionaries, about 2.31 million affix rules) has a rule whose decision changes. An independent survey of the same collections plus the Hunspell 1.7.2 tests, system dictionaries and Lucene's test files found 67 rules with a dash in the strip part, all outside brackets, and 0 changed decisions. None of Lucene's 97 test .aff files has an unescaped dash between two characters in a bracket condition.
• Likelihood: contrived. It takes an unusual hand-written rule; the over-accepting forms ([a-c] over a strip b) are pairs C Hunspell itself treats as dead rules. No scale is needed: one dictionary line triggers it.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: spell("ay") expected:<true> but was:<false>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.apache.lucene.analysis.hunspell.TestAffixConditionDashInStrip.testDashInConditionOverStrip(TestAffixConditionDashInStrip.java:36)

• Contract. hunspell(5): "Characters in braces sign an arbitrary character from the character subset. Dash hasn't got special meaning, but circumflex (^) next the first brace sets the complementer character set." Hunspell.spell returns "whether the given word's spelling is considered correct according to Hunspell rules" (Hunspell.java:71-72). The class "can be used in place of native Hunspell for many languages"; its javadoc gives examples of what is not supported yet ("For example:" Hungarian, characters outside the BMP, the PHONE option, :35-44); none is about affix conditions.
• Lucene's own intent. The comment above escapeDash (AffixCondition.java:186) and its use for the stem part (:179). LUCENE-5777 (2014): "Basically '-' doesnt mean anything special (e.g. what it normally means for a regex), so we usually have to escape it." CHANGES 4.9: "LUCENE-5768, LUCENE-5777: Hunspell condition checks containing character classes were buggy."
• Tests. No test asserts range semantics. testConditionHasBracketsIntersectingWithStrip uses [io], no dash; the only dash fixture (double-escaped.aff, [^\-]ar with strip ar) has the class on the stem side; testImpossibleCondition shows Lucene folds a condition that cannot hold over the strip to ALWAYS_FALSE.
• History. Before LUCENE-9806, Dictionary escaped the dash in the whole condition (since 2014). The commit that split the condition removed that escaping; its PR describes a speed-up only and its review never mentions dashes or ranges. 8.8.2 gives the hunspell(5) answers.
• Reference. C Hunspell 1.3.2 (every word the adjudication and the independent reproducer checked) and 1.7.2 (the primary test's dictionary in UTF-8 and ISO8859-1, and the three variants that reach each strip check) give the expected spell verdicts; both give the expected stems (ay to ab, none for ax).

The strongest case (from the works-as-designed review, the devil's advocate and the adjudication): Lucene promises to replace native Hunspell only for many languages, HunspellStemFilter is @lucene.experimental, the strip/stem split is a deliberate optimization, the over-accepting forms are rules C Hunspell calls "incompatible stripping characters and condition", and no public dictionary is affected.

Why it fails: the javadoc's examples of unsupported features say nothing about affix conditions, and spell promises a verdict "according to Hunspell rules"; the same class escapes the dash for the stem part, so one condition text is read two ways in one rule; the change was speed-only and 8.8.2 was correct; Hunspell.spell is not experimental, and experimental does not license wrong results; the under-accepting forms ([^a-c] over the strip, [c-a]) are rules C Hunspell applies with no warning, and Lucene silently drops them; for the over-accepting forms C Hunspell keeps a dead rule, which Lucene itself means to fold to ALWAYS_FALSE. "No public dictionary is affected" is reach, not intent. Every reviewer rated the intended-behaviour case weak. Verdict: not intended, a regression.

One fix was built and run in a scratch worktree at a9779ef3c193, with a one-method regression test in TestAffixCondition. The sibling bug in the same lines (a condition shorter than the strip checked against the wrong end of the strip) needs its own one-line fix at :107; the two were also run together, and the patches and their tests sit next to each other, so one PR for both is simpler.

Wrap the arguments of the three strip.matches calls in escapeDash(...), so the strip part and the stem part both treat the dash as a plain character (they still use different regex engines: java.util.regex for the strip, Lucene RegExp for the stem). escapeDash keeps an already escaped \- (pt_PT style) and returns its input unchanged when there is no dash.

-        return strip.matches(regex) ? ALWAYS_TRUE : ALWAYS_FALSE;
+        return strip.matches(escapeDash(regex)) ? ALWAYS_TRUE : ALWAYS_FALSE;
...
-        if (!strip.matches(condition.substring(0, split))) {
+        if (!strip.matches(escapeDash(condition.substring(0, split)))) {
...
-      if (!strip.matches(condition.substring(split))) {
+      if (!strip.matches(escapeDash(condition.substring(split)))) {

Trade-offs

Only dictionaries with a dash between two characters inside [ ] over the strip change verdicts, toward C Hunspell's; escaped dashes, a dash at a bracket's edge and dash-free conditions give the same answers. No per-word cost: the strip part is still folded to a constant at load; the fix only adds a little string work while the dictionary loads. A reversed pseudo-range such as [c-a] no longer throws, so such a rule becomes live as hunspell(5) defines it.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: stock plus the regression test and the three repro classes fails 13 of 24 (13 of 25 with the line probe, which passes; controls pass), with assertions on and off.
• With the fix: --tests 'org.apache.lucene.analysis.hunspell.*' plus the independent reproducer's class: 50 classes (the package's 46 test classes plus the four added ones), 176 tests, 0 failures, 13 skipped (tests that need external data: dictionaries, word corpora or a Hunspell checkout; one is @Ignore), with assertions on and off. So this run did not cover real dictionaries or corpora.
• :lucene:analysis:common:check -x test (google-java-format, ecj lint, forbidden APIs) passes with the fix and the regression test.
• Together with the sibling short-condition fix and both bugs' tests: 52 classes, 195 tests, 0 failures, 13 skipped.
• Earlier, on ea57360912ed with the fix shadowed: Lucene's hunspell test classes 146 of 146; the exhaustive harness 0 mismatches over 144,482,592 checks with both fixes; commit isolation: the introducing commit plus only this change is correct.

Build the strip check with new RegExp(escapeDash(...)) and run it on the strip, so one engine and one escaping rule handle the whole condition and java.util.regex goes away.

Trade-offs

A larger change; Lucene RegExp and java.util.regex differ on other characters (testNonHunspellPatternCharacters pins some of that), so it needs its own review. Not run.

Call escapeDash on the condition at the top of compile.

Trade-offs

The pattern counter (:131-151) treats a \ outside brackets as its own pattern, so a condition with a dash outside brackets (such as a-) would be miscounted. Read only; not run.

Method: bounded-exhaustive translation validation (the hunt's "semi-formal" tier).

• A harness drove the real AffixCondition (the uniqueKey shortcut, compile and acceptsStem, as Dictionary and Stemmer use them) against a separate reference of the hunspell(5) condition semantics, over every condition of 1-3 patterns from 34 element kinds over {a, b, -} (dot, literals, every bracket set and its negation), strips of up to 3 and stems of up to 2 characters over {a, b, c, -}: 144,482,592 checks.
• On the stock jar it found 4,740,482 mismatches in two classes: this one (a dash set over the strip) and a sibling (a condition shorter than the strip checked against the wrong end). With both fixes: 0 mismatches; with this fix alone only the sibling class remains.
• The finder reduced it to a public-API JUnit and surveyed 191 public dictionaries (0 affected rules). An independent reproducer, cross-model reviews, a devil's advocate and an adjudication then checked it (see the checks).