ArabicStemmer strips only one prefix: after the conjunction و ("and") the article stays, unlike the light10 algorithm ArabicAnalyzer cites

ArabicAnalyzer's javadoc says it "implements light-stemming as specified by: Light Stemming for Arabic Information Retrieval" (ArabicAnalyzer.java:38-39). Section 2.2 of that paper runs ordered steps: 1) remove a leading و ("and") if 3 or more characters remain, 2) remove one definite article (ال, وال, بال, كال, فال, لل) if 2 or more remain, 3) go through the suffixes once. ArabicStemmer.stemPrefix instead scans one array, the six articles followed by و (ArabicStemmer.java:46-54), and returns after the first match (:90-94); stem() calls it once (:76-81). So once it removes the و it never tries the articles on the rest: وبالكتاب ("and with the book") is indexed as بالكتاب instead of كتاب, the term of الكتاب and بالكتاب, and the same for و + لل, كال and فال (وللطلاب becomes للطلاب, وبالتالي becomes بالتال). A TermQuery for the analyzed الكتاب then finds 0 of 1 documents that contain only وبالكتاب, with no error. و + ال is unaffected because وال is in the array itself. The single-return loop came with the class (LUCENE-1406, 2008) and was kept when it moved to light10 (LUCENE-1758, 2009), so every release from 2.9.0 to 10.5.2 and main has it. The fix tries the articles again after removing the one-letter و; it changes indexed terms (reindex note needed) and also brings light10's own overstemming for a few words, so no retrieval gain is claimed.

• Light stemming: removing a few frequent prefixes and suffixes without a dictionary, so that الكتاب ("the book"), بالكتاب ("with the book") and وبالكتاب ("and with the book") all get the index term كتاب ("book").
• The prefixes: the conjunction و (wa-, "and") and the definite articles ال (al-, "the"), وال (wa + al-), بال (bi-l-, "with the"), كال (ka-l-, "like the"), فال (fa-l-, "so the"), لل (li-l-, "for the"). They are written joined to the word, and و can come before one of the others.
• light10: the stemmer of Larkey, Ballesteros and Connell, "Light Stemming for Arabic Information Retrieval", which ArabicAnalyzer cites. Section 2.2 (page 13 of the paper): "1. Remove و (\"and\") ... if the remainder of the word is 3 or more characters long. ... 2. Remove any of the definite articles if this leaves 2 or more characters. 3. Go through the list of suffixes once ...".

1. ArabicStemmer.prefixes is one array: ال, وال, بال, كال, فال, لل, و, with و last (ArabicStemmer.java:46-54).
2. stemPrefix tries the entries in order and returns deleteN(...) on the first that matches (:90-94). stem() calls it once, then the suffixes (:76-81).
3. For وبالكتاب none of the six articles matches (the word starts with و and then ب, so even وال fails at its second letter); و matches and is removed, and the method returns بالكتاب. بال would match now (7 characters, it needs 5), but nothing tries it again.
4. The length rules (:118-132: و needs 4 or more characters, the others their length + 2) are exactly the paper's 3 and 2 remaining characters; only the order of steps 1 and 2 was lost when they were folded into one list.
5. ArabicStemFilter applies the stemmer to every non-keyword token (ArabicStemFilter.java:44-55); ArabicAnalyzer ends its chain with it (ArabicAnalyzer.java:131-142); the SPI factory arabicStem creates the same filter.
6. A double و before وال (ووال...) is hit the same way, because وال no longer matches at position 0. Words too short for light10's length rules (for example و + بال + one letter) come out the same either way; in the follow-up's probe, 70 of 148,795 waw+article-shaped words are unchanged by the fix.

The rows for وبالتالي, وبالنسبة and وللمرة come from a probe through the default ArabicAnalyzer, and the fact check's probe through ArabicStemFilter alone gives the same terms; the other word rows ran through ArabicStemFilter.

When the query text goes through the same analyzer (for example a query parser over ArabicAnalyzer), the miss goes both ways: the query وبالكتاب gives بالكتاب and misses documents that say الكتاب or بالكتاب. A TermQuery built from a raw string is not analyzed, and ArabicAnalyzer.normalize() does not stem.

• Who is exposed. Applications that analyze Arabic text with the stock ArabicAnalyzer (the default constructor included, ArabicAnalyzer.java:96-98), ArabicStemFilter directly, or the arabicStem factory (for example in a CustomAnalyzer). No option turns the behaviour on or off. Tokens marked as keywords (for example by the analyzer's stem exclusion set) are not stemmed, and stopwords are removed before the stemmer.
• Not affected. و + ال (والكتاب), any single prefix alone, words without a leading و, words too short for light10's length rules, and the separate Snowball Arabic stemmer.
• What triggers it. Real words of the shape و + بال, كال, فال or لل ("and with the", "and like the", "and so the", "and for the"), for example وبالتالي ("and therefore"), وبالنسبة, وللمرة, and a double و before وال. It is deterministic, at index time and, when the query is analyzed the same way, at query time, and needs no size or setting. How often these words occur in real text was not measured.
• What the damage looks like. Mostly missed matches between a waw+article form and the other forms of the same word, inside each document's own field: the document وبالكتاب is not found by الكتاب or بالكتاب, and the query وبالكتاب misses them; much less often, extra conflations that light10 would not make (see the next point). No exception and no term longer than its input: a probe over 411,089 words on main (assertions on and off) found neither. The work per token stays a fixed handful of comparisons (from the code), and the index is not damaged.
• Scale, honestly. Over that bounded, mostly constructed word set the stock stemmer loses 19,859,718 same-stem pairs that light10 would make and keeps 240,799 that light10 would not; over 29 real words it loses 18 pairs and keeps 2 (one right, وكالة/وكالات; one spurious, وبالون/بال). These are counts over a constructed set, not corpus frequencies, and the net retrieval effect of a fix was not measured.
• Existing indexes keep the old terms until reindexed; after a fix, new query terms and old documents disagree for these words until the field is reindexed.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<[]كتاب> but was:<[بال]كتاب>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.ar.TestArabicWawThenArticle.testWawThenArticle(TestArabicWawThenArticle.java:14)

• The analyzer's contract. ArabicAnalyzer.java:38-39: "This analyzer implements light-stemming as specified by: <i> Light Stemming for Arabic Information Retrieval </i>" (the PDF URL after it is dead).
• CHANGES of the light10 change (LUCENE-1758, 820620f3a72, 2009, the whole entry): "ArabicAnalyzer now uses the light10 algorithm, has a refined default stopword list, and lowercases non-Arabic text. You'll need to fully re-index any previously created indexes. This isn't a break in back-compatibility because ArabicAnalyzer has not yet been released. (Robert Muir)"
• The stemmer's javadoc defines stemming as "Removal of attached definite article, conjunction, and prepositions" (ArabicStemmer.java:26-29).
• The paper, section 2.2 (page 13): "We tried several versions of light stemming, all of which followed the same steps: 1. Remove و (\"and\") for light2, light3, and light8, and light10 if the remainder of the word is 3 or more characters long. ... 2. Remove any of the definite articles if this leaves 2 or more characters. 3. Go through the list of suffixes once ...". Lucene's length rules are exactly these 3 and 2 characters. Its Table 1 (page 14) lists the light10 "prefixes" as ال, وال, بال, كال, فال, لل, و, the same strings in the same order as ArabicStemmer.java:46-54.
• The lab's distributed code. The Lemur Toolkit's arabic_light10 (University of Massachusetts) does "Remove waw (advance pointer) if remainder >= 3 chars", then "Remove definite article if remainder >=2 chars", then the suffixes; its article and suffix tables equal Lucene's. A diff of Lucene against a Java port of its stemming steps (Lemur's normalization and stopword steps left out, as ArabicStemFilter does neither), every word up to 6 letters over 13 letters plus 5 million random longer words (10,229,043 words), finds 6,072 differences, every one و followed by an article, and 0 with the fix. It is the light10 code the lab distributed, not necessarily the code of the paper's experiments (which ran on an in-house engine).
• The original author's aim (LUCENE-1406, #2480): "the major motivation for implementing it exactly the way it is presented in the paper is that the algorithm is TREC-tested".
• Tests. TestArabicStemFilter checks each prefix alone (والحسن, بالحسن, كالحسن, فالحسن, للاخر, وحسن), و plus suffixes and two suffixes; none has و followed by an article other than وال, and none asserts that such an article is kept.
• History. Neither LUCENE-1406 nor LUCENE-1758 discusses stopping after one prefix; no CHANGES or MIGRATE entry mentions it. (The 2008 light8 code cited an earlier paper by the same authors, "Improving Stemming for Arabic Information Retrieval: Light Stemming and Co-occurrence Analysis" (2002), which was not checked here; the light10 paper says light8 followed the same steps.)

The intended-behaviour checks built the strongest case for "works as designed" (rated moderate by the cross-model reviewer): the helper's javadoc says "Stem a prefix off" (singular) while the suffix helper says "Stem suffix(es) off"; the array copies the paper's Table 1 column, which lists و next to the articles and also وال, so "remove one Table 1 prefix" is a possible reading; the output has been stable since 2009 with no report; a fix forces a reindex; on LUCENE-8028 a committer asked to be conservative about changing the default; ArabicStemmer is package-private and ArabicStemFilter only promises to "apply" it; and the fix brings light10's own overstemming (وكالات, "agencies", becomes ات instead of كال).

Why it fails: the public contract is the analyzer's "as specified by" and the CHANGES "now uses the light10 algorithm", and the paper's steps are explicitly separate and ordered. The lab's own light10 code keeps وال in its article table and still removes و first, so the Table 1 reading does not hold. The original author said they followed the paper "exactly", and the length rules do. Nothing records stopping after و as a decision. The reindex cost and the overstemming are real, but they are a compatibility and quality decision for maintainers, not evidence that the current output is the documented one; the overstemming is the paper's own behaviour, and stock already overstems the same way without the و (بالون gives ون).

Two equivalent code fixes were validated (A through gradle and the module's check, B with the standalone runner); C and D are the alternatives a maintainer may prefer.

In ArabicStemmer.stemPrefix, when the matched prefix is the one-letter و, try the six articles on the rest before returning (+15/-2 as formatted by tidy; the same logic as the finder's +9/-1 sketch). A regression test: the repro's class, or its cases added to TestArabicStemFilter.

int stemPrefix(char[] s, int len) {
  for (int i = 0; i < prefixes.length; i++) {
    if (startsWithCheckLength(s, len, prefixes[i])) {
      len = deleteN(s, 0, len, prefixes[i].length);
      // light10 removes the conjunction waw (step 1) before a definite article (step 2):
      // after removing waw, try the articles once more
      if (prefixes[i].length == 1) {
        for (int j = 0; j < prefixes.length - 1; j++) {
          if (startsWithCheckLength(s, len, prefixes[j])) {
            return deleteN(s, 0, len, prefixes[j].length);
          }
        }
      }
      return len;
    }
  }
  return len;
}

Trade-offs

• Changes indexed terms for waw+article words long enough for light10's length rules: the CHANGES entry should advise reindexing fields analyzed with ArabicAnalyzer, ArabicStemFilter or arabicStem; it suits main (11.0). (LUCENE-1758's reindex note of 2009 is no precedent: it came before the analyzer's first release.)
• Brings light10's own overstemming where the و is a root letter followed by letters that look like an article: وكالات ("agencies") becomes ات instead of كال, so it no longer conflates with وكالة ("agency", كال on both); وبالون becomes ون, as بالون already does on stock, instead of بال. That is the paper's behaviour; no retrieval gain is claimed.
• Relies on و being the only one-letter entry and the last one in prefixes, and on every entry before it being a light10 article (true today, ArabicStemmer.java:46-53).
• Cost: at most 6 more 2-3-character comparisons and one more in-place shift for a token that starts with و.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test fails 5 of 6 on stock and passes 6 of 6 with the fix; the 5 Arabic suites pass 56 of 56 with assertions on and off (TestArabicStemFilter 23, TestArabicNormalizationFilter 15, TestArabicAnalyzer 8, TestArabicFilters 4, the new 6); :lucene:analysis:common:check passes: checkGoogleJavaFormat, ecjLint, forbiddenApis and the whole module's tests (340 suites, 1,915 tests, 0 failed, 18 skipped).
• An exhaustive diff against a light10 reference written from the paper (every word up to 6 letters over 13 letters, 5,229,042 words): 0 mismatches with the fix, 3,042 on stock. The diff against the lab's code port (10.2 million words) gave 0 with the finder's equivalent sketch.

Step 1: if the word starts with و and has 4 or more characters, remove it. Step 2: try the six articles once (+10/-2).

int stemPrefix(char[] s, int len) {
  // light10 step 1: remove the conjunction waw if 3 or more characters remain
  if (len >= 4 && s[0] == WAW) {
    len = deleteN(s, 0, len, 1);
  }
  // light10 step 2: remove one definite article if 2 or more characters remain
  for (int i = 0; i < prefixes.length - 1; i++) {
    if (startsWithCheckLength(s, len, prefixes[i])) {
      return deleteN(s, 0, len, prefixes[i].length);
    }
  }
  return len;
}

Trade-offs

Reads like the paper and gives the same output as A on the tested space (for a word that starts with وال, removing و and then ال removes the same 3 characters as وال alone, under the same length rules). It still loops to prefixes.length - 1, so it relies on و staying the last entry; moving و out of the array would be cleaner but touches more lines. Same reindex and overstemming notes as A.

What the checks validated

The doc's test 6 of 6 and the upstream TestArabicStemFilter, TestArabicAnalyzer and TestArabicFilters 35 of 35 with it on the classpath; the exhaustive diff (5,229,042 words) finds 0 mismatches. Not run through gradle tidy or check.

Say in the ArabicAnalyzer and ArabicStemmer javadocs that at most one prefix is removed, so و + بال, كال, فال, لل keep the article.

Trade-offs

No reindex, but the analyzer would no longer implement the algorithm it cites, and the missed matches stay.

Keep the default and add a switch (a constructor flag on ArabicStemFilter, a factory parameter). Not built.

Trade-offs

Avoids silent term changes for existing indexes, but adds API surface for what the docs already promise, and the default stays different from light10.

Method: bounded-exhaustive differential checking against a reference written from the published rules (the study's "semi-formal" tier). Hunt unit "light stemmers against their published rules", 2026-10-27, main a9779ef3c193.

• The reference implements light10 steps 1-3 from section 2.2 of the paper that ArabicAnalyzer cites, independently of Lucene's code.
• Word space: 14 prefix chains x 2,380 stems (length 0 to 3 over 13 letters: the 11 letters of the affix tables plus 2 neutral letters, د and س) x 13 suffix chains = 433,160 combinations, which the harness deduplicates to 408,800 distinct words, each run through the stock ArabicStemFilter and the reference.
• Result: 119,138 mismatches, every one with a و before an article (و + بال, كال, فال, لل: 4 x 29,744; a double و before وال: 162); 0 mismatches with the fix sketch on the classpath.
• An AI finder then wrote the public-API JUnit and traced the cause in stemPrefix and the git history; independent AI tools reproduced it, built their own exhaustive diffs (one against the light10 code the authors' lab distributed) and judged it (see the checks).