Sorani normalizer: an invisible direction mark after a word-final HEH (ه) keeps it from becoming AE (ە)

Lucene's Sorani normalizer (SoraniNormalizer, used by SoraniNormalizationFilter, SoraniAnalyzer and the soraniNormalization factory) documents two rules that meet here: a word-final HEH (U+0647) is converted to AE (U+06D5), and "formatting characters such as directional controls are removed". It applies them in one left-to-right pass and tests i == len - 1 when it reaches the HEH, while a trailing RLM, LRM or ALM is still in the buffer; the mark is deleted one step later and the HEH, now last, is never converted. StandardTokenizer keeps such a mark inside the token, so the stock analyzer indexes خانه + RLM as خانه and خانه as خانە: the two look the same and do not match each other, at index and query time, with no error. Every release from 4.7.0 (2014) to 10.5.2 and main is affected. A 15-line fix that looks past trailing formatting characters the class deletes (any Character.FORMAT character except ZWNJ and ZWJ) passes the new test and all 30 upstream Sorani tests.

• Sorani normalization: SoraniNormalizationFilter, and SoraniAnalyzer which uses it, runs SoraniNormalizer.normalize on every token. The class javadoc lists its rules, among them "Alternate forms of vowel 'e' (0647+200C, word-final 0647, 0629) are converted to 06D5 (AE)" and "Harakat, tatweel, and formatting characters such as directional controls are removed." (SoraniNormalizer.java:29, :33).
• HEH and AE: AE ە (U+06D5) is Sorani's letter for the vowel e. At the end of a word the normalizer treats HEH ه (U+0647) as the same vowel; the upstream test gives the reason: "always (and in final form by def), so frequently omitted" (TestSoraniNormalizationFilter.java:70-73).
• Direction marks: invisible characters such as RLM (U+200F), LRM (U+200E) and ALM (U+061C) that steer the display of mixed left-to-right and right-to-left text. Java classifies them as Character.FORMAT. They do not change how the HEH joins (Unicode joining type Transparent), so خانه + RLM looks exactly like خانه.

1. normalize walks the buffer once, left to right, with one switch per character (SoraniNormalizer.java:72-128).
2. case HEH converts the HEH only if (i == len - 1): if it is the last character of the buffer at that moment (:89-93).
3. A trailing mark is still in the buffer then. The default branch deletes it later, like every Character.FORMAT character (:120-124).
4. For خانه + RLM (len = 5): at i = 3 the test is 3 == 4, false, so the HEH stays; at i = 4 the RLM is deleted (len = 4); the result خانه ends in an unconverted HEH.
5. The class's other position rule is safe: a leading mark is deleted before the REH is visited, so a word-initial REH still becomes RREH (:100-104). The ZWNJ rule rewrites the HEH before it itself (:82-88).
6. SoraniAnalyzer tokenizes with StandardTokenizer, which keeps a trailing format character inside the token (UAX#29 rule WB4, StandardTokenizerImpl.jflex:83-85), and its normalize (the query side) applies the same filter (SoraniAnalyzer.java:113-130).

• Filter output for خانه + one tail, stock and with the fix: no tail ..06D5 both; RLM, LRM, ALM, PDF (U+202C), RLO (U+202E), PDI (U+2069): ..0647, with the fix ..06D5; ZWNJ ..06D5 both (its own rule); ZWJ, TATWEEL and FATHA ..0647 both (not claimed: they force a joining form or mark a consonant).
• SoraniAnalyzer, three documents خانە گەورەیە (AE spelling), خانه گەورەیە (HEH spelling) and the HEH spelling with an RLM after خانه: a search for خانە or for خانه finds 2 of 3; the query copied with the mark finds 1 of 3; with the fix every query finds 3 of 3.
• Stemming: کوردییه ("Kurdish") is stemmed to [..06CC 06CC], with a trailing RLM to [..06CC 06CC 0647].
• Exhaustive, all 1,118,481 strings of length 0 to 5 over a 16-character alphabet: within that alphabet the fix changes exactly the 31,420 inputs whose last HEH is followed only by RLM, LRM, ALM, PDF or soft hyphen, and in each only that HEH.
• Every BMP character after خانه: the fix changes the output (HEH to AE) for exactly 41 characters, the Character.FORMAT characters other than ZWNJ and ZWJ on JDK 25. Besides the direction marks they include U+FEFF, U+200B, U+2060-U+2064, U+206A-U+206F, U+180E, U+FFF9-U+FFFB and the prepended concatenation marks (U+0600-U+0605, U+06DD, U+070F, U+0890, U+0891, U+08E2). None of them is join-causing (ICU: 26 Transparent, 15 Non_Joining).

• Who is exposed. Applications that index Sorani Kurdish text with new SoraniAnalyzer() or any other constructor (no option avoids it), with SoraniNormalizationFilter in a custom chain, or with the SPI factory soraniNormalization (the factory javadoc shows the Solr form solr.SoraniNormalizationFilterFactory). Downstream servers that ship a built-in Sorani analyzer on this filter, such as Elasticsearch's sorani analyzer with its sorani_normalization filter, inherit it (read in their documentation; not executed).
• Default or opt-in. You opt in by choosing the Sorani analyzer or filter for a field; inside it, this is the default behaviour.
• What triggers it. A word whose final vowel is typed as HEH and whose final HEH is followed only by one or more invisible formatting characters that the class deletes, such as RLM, LRM, ALM, PDF, RLO, PDI or the soft hyphen, with no ZWNJ or ZWJ among them (HEH + RLM + ZWNJ already becomes AE through the ZWNJ rule). Deterministic: every such input, every time; no size or scale is needed.
• Effect. The marked word is indexed and queried as ..0647, the unmarked one as ..06D5, so they do not match each other in either direction (a search finds 2 of 3 spellings, a query copied with the mark 1 of 3). The stemmer then sees a different word, so the stemmed terms differ too (کوردییه: [..06CC 06CC] without the mark, [..06CC 06CC 0647] with it). The marked term is one that HEH + TATWEEL or HEH + FATHA already produce, so a query typed with the mark also matches a document whose word ends in HEH + TATWEEL (executed in the follow-up's probe).
• Not affected. Words spelled with AE, words without a trailing mark, marks anywhere but directly after a final HEH. No exception, no hang, no extra cost; offsets and positions are consistent and CheckIndex is clean (the follow-up's probe: 3,000 trigger-dense strings, checkRandomData(200), CheckIndex after forceMerge(1)).
• How likely. Rare. The upstream Sorani vocabulary test file (8,272 rows of frequent corpus words) has 11 words with a trailing direction control (U+202E), none of them after a HEH, and the prior-report searches found no report in the 12 years since 4.7.0.
• Existing indexes and rollout. A fix changes indexed terms of affected words, so those fields need a reindex, and index-time and query-time analysis must change together: with the fixed analyzer at query time only, a query typed with the mark no longer finds the old marked documents, so neither spelling finds them (reasoning).
• Workaround. Until a fix ships, a patternReplace char filter that deletes the direction marks in front of the tokenizer gives ..06D5 and 3 of 3 hits (executed with SPI names through CustomAnalyzer); it must be in both the index-time and the query-time analyzer. Its pattern leaves out the soft hyphen and the other format characters the fix covers, and the executed CustomAnalyzer has no stop filter or keyword marker, so it is not the same as SoraniAnalyzer; to keep SoraniAnalyzer (a final class), wrap it in an AnalyzerWrapper that adds the char filter in wrapReader and wrapReaderForNormalization (not executed).
• Unicode versions and older JDKs. JDK 25's Character data is Unicode 16.0, StandardTokenizer's tables are Unicode 12.1, and the joining types here come from ICU (Unicode 17). ALM (U+061C) and the isolates LRI, RLI, FSI and PDI (U+2066-U+2069) were added in Unicode 6.3, after Java 8's Unicode 6.2, so on Java 8 they are not format characters and would stay in the term (not executed); RLM and LRM behave the same on every supported JDK.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<خان[ە]> but was:<خان[ه]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.ckb.TestSoraniFinalHehBeforeRlm.testFinalHehFollowedByRlm(TestSoraniFinalHehBeforeRlm.java:25)

• The contract. The class javadoc (SoraniNormalizer.java:21-35): "Alternate forms of vowel 'e' (0647+200C, word-final 0647, 0629) are converted to 06D5 (AE)" (:29) and "Harakat, tatweel, and formatting characters such as directional controls are removed." (:33). The public SoraniNormalizationFilter is "A TokenFilter that applies SoraniNormalizer to normalize the orthography" (SoraniNormalizationFilter.java:24). SoraniNormalizer was a public class, with this rule list in its published javadoc, from 4.7.0 to 9.12.x (javap of the 4.7.0 and 9.12.3 jars; Lucene renders javadoc with -protected); it is package-private only since 10.0.0 (LUCENE-10561). The javadoc states no order of operations and no general "removed characters never change the result" rule, so the claim is that this order leaves two documented rules in tension, not that a stated invariant is broken.
• The rule's reason. testFinalH: "always (and in final form by def), so frequently omitted" (TestSoraniNormalizationFilter.java:70-73). The rule is about the final letter form; RLM, LRM, ALM, PDF and RLO have Unicode joining type Transparent (PDI: Non_Joining), so a HEH before them is still drawn in final form.
• The class's own symmetry. A leading RLM, LRM, ALM, RLE or TATWEEL before REH still gives word-initial RREH, and the ZWNJ rule looks past a mark (HEH + RLM + ZWNJ gives AE). Only the trailing check runs before the deletion.
• Upstream data. The vocabulary of TestSoraniStemFilter.testVocabulary (8,272 rows) has 11 words ending in U+202E whose expected output drops the mark and normalizes and stems as if it were absent (none of them ends in a HEH). Of its words ending in a HEH before characters the class removes, 320 end in HEH + ZWNJ (made AE by the ZWNJ rule) and one is HEH + TATWEEL -> HEH, which is not claimed (TATWEEL forces a joining form) and which the fix keeps; none has a HEH before a direction mark.
• History. Both rules came in the first commit, in today's order (LUCENE-5379, 2014); the four later commits on the file change only the license header, formatting, array syntax and visibility (package-private from 10.0.0). No test, comment, commit message, CHANGES or MIGRATE entry mentions trailing marks.

The intended-behaviour checker and the cross-model intended lens (which rated it moderate) built the strongest case: the javadoc is an unordered list of independent rules, and "word-final" can be read literally as "the last character of the token as given"; the upstream vocabulary pins HEH + TATWEEL -> HEH, so a trailing removable character can block the rule on purpose; the class is package-private, so its javadoc is no published contract; the behaviour has been stable for 12 years; and a fix forces a reindex.

The answer: the same class treats removed characters as absent for its word-initial rule and its ZWNJ rule; the stated reason is the final letter form, which the marks do not change, while TATWEEL and ZWJ do, so the pinned row agrees with the narrow claim and the narrow fix keeps it; the class promises to remove direction controls itself and the stock tokenizer hands them over inside the token; the project's own vocabulary expects trailing marks to be ignored; no record supports the literal reading. The package-private point does not hold for most of the bug's life: the class was public, with this list in its published javadoc, from 4.7.0 to 9.12.x; and a reindex is the usual cost of an analyzer fix.

Limits of the answer: the javadoc states no order of operations; the leading-mark case shows what the loop order allows, not what the authors meant for trailing marks; the 11 vocabulary words with a trailing mark do not end in a HEH; no Sorani reference is cited. Verdict: a real, narrow order-of-operations defect, best presented as an inconsistency between two documented rules that appears unintended, with the compatibility cost acknowledged. Do not argue idempotence: it is not documented, and HEH + TATWEEL stays non-idempotent with the fix.

Two equivalent fixes were built and validated with gradle in a scratch worktree at a9779ef3c193; A has the same condition as the finder's fix, which every verifier also ran. A wider fix fails the upstream vocabulary. Until a fix ships, a patternReplace char filter that deletes the direction marks in front of the tokenizer works around it (executed: 3 of 3 hits).

case HEH calls a helper instead of i == len - 1: the HEH is word-final if every character after it is a Character.FORMAT character other than ZWNJ and ZWJ (+15/-1 in SoraniNormalizer.java, with a new ZWJ constant). It tests no Unicode joining property: on JDK 25 it admits 41 BMP characters, every format character the class deletes except ZWNJ and ZWJ (see the validation). The helper's javadoc in the diff ("does not change joining") is a draft; more exact: "a formatting character that normalize deletes, other than ZWNJ (handled by its own case) and ZWJ (which joins the HEH to what follows)". The code below is an excerpt; the full diff:

--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/ckb/SoraniNormalizer.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/ckb/SoraniNormalizer.java
@@ -45,6 +45,7 @@
   static final char HEH = '\u0647';
   static final char AE = '\u06D5';
   static final char ZWNJ = '\u200C';
+  static final char ZWJ = '\u200D';
   static final char HEH_DOACHASHMEE = '\u06BE';
   static final char TEH_MARBUTA = '\u0629';
 
@@ -87,7 +88,7 @@
           i--;
           break;
         case HEH:
-          if (i == len - 1) {
+          if (isWordFinal(s, i, len)) {
             s[i] = AE;
           }
           break;
@@ -126,4 +127,17 @@
     }
     return len;
   }
+
+  /**
+   * Returns true if every char after {@code pos} is a formatting character that {@link #normalize}
+   * removes and that does not change joining (not ZWNJ or ZWJ), such as a directional mark.
+   */
+  private static boolean isWordFinal(char[] s, int pos, int len) {
+    for (int i = pos + 1; i < len; i++) {
+      if (s[i] == ZWNJ || s[i] == ZWJ || Character.getType(s[i]) != Character.FORMAT) {
+        return false;
+      }
+    }
+    return true;
+  }
 }

static final char ZWJ = '\u200D';                      // new constant
case HEH:
  if (isWordFinal(s, i, len)) {                       // was: if (i == len - 1)
    s[i] = AE;
  }
  break;
private static boolean isWordFinal(char[] s, int pos, int len) {
  for (int i = pos + 1; i < len; i++) {
    if (s[i] == ZWNJ || s[i] == ZWJ || Character.getType(s[i]) != Character.FORMAT) {
      return false;
    }
  }
  return true;
}

Trade-offs

• Existing indexes keep the old term for affected words until reindexed; worth a CHANGES line.
• Cost: a forward scan from a HEH that stops at the first other character. The extra scan reads each character at most once per token, so the fix adds only linear work. (normalize() itself is not linear today: every deletion shifts the rest of the buffer, StemmerUtil.delete.)
• Breadth: it admits every format character the class deletes, not only the direction marks. If maintainers want only the direction marks and the soft hyphen, the helper can test an explicit list instead (not built or tested).
• Index-time and query-time analysis must change together during the rollout.
• Unchanged by design: HEH + TATWEEL, ZWJ or a vowel mark stays HEH; HEH + ZWNJ was already AE.
• Only main was built and tested with gradle; normalize() is the same, ignoring whitespace, from 4.7.0 on, so a backport should need at most context changes.

What the checks validated

• Gradle, JDK 25, scratch worktree at a9779ef3c193: the new test fails 5 of 7 on stock (assertions on and off) and passes 7 of 7 with the fix; every test in org.apache.lucene.analysis.ckb passes: 6 suites, 37 tests (30 upstream, including checkRandomData over SoraniAnalyzer and the 8,272-row vocabulary, plus the 7 new), 0 skipped, assertions on and off.
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest of lucene/analysis/common pass.
• Exhaustive, all 1,118,481 strings of length 0 to 5 over a 16-character alphabet: within that alphabet it changes exactly the 31,420 inputs whose last HEH is followed only by RLM, LRM, ALM, PDF or soft hyphen, and only that HEH, from U+0647 to U+06D5.
• Every BMP character after خانه (surrogates skipped): the output changes for exactly 41 characters, each from ..0647 to ..06D5: the direction marks and controls (U+200E, U+200F, U+061C, U+202A-U+202E, U+2066-U+2069), U+00AD, U+200B, U+2060-U+2064, U+FEFF, U+206A-U+206F (deprecated format controls, including INHIBIT and ACTIVATE ARABIC FORM SHAPING), U+180E, U+FFF9-U+FFFB and the prepended concatenation marks (U+0600-U+0605, U+06DD, U+070F, U+0890, U+0891, U+08E2). ICU 78.3: 26 Transparent, 15 Non_Joining, none join-causing. Through StandardTokenizer 38 of them stay in the token; U+200B, U+0890 and U+0891 do not. Fix B gives the same output.
• The finder's fix has the same condition; with it every verifier's claim tests passed (finder 5 of 5, the independent reproduction 12 of 12, adjudication 10 of 10), as did the 28 (finder) and 30 (refute reviewer) upstream tests they ran.

Before the loop, drop trailing Character.FORMAT characters other than ZWNJ and ZWJ (+10):

--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/ckb/SoraniNormalizer.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/ckb/SoraniNormalizer.java
@@ -45,6 +45,7 @@
   static final char HEH = '\u0647';
   static final char AE = '\u06D5';
   static final char ZWNJ = '\u200C';
+  static final char ZWJ = '\u200D';
   static final char HEH_DOACHASHMEE = '\u06BE';
   static final char TEH_MARBUTA = '\u0629';
 
@@ -70,6 +71,11 @@
    * @return length of input buffer after normalization
    */
   int normalize(char[] s, int len) {
+    // remove trailing formatting characters that do not change joining (such as directional marks)
+    // first, so that they do not hide a word-final HEH
+    while (len > 0 && isTrailingRemovable(s[len - 1])) {
+      len--;
+    }
     for (int i = 0; i < len; i++) {
       switch (s[i]) {
         case YEH:
@@ -126,4 +132,8 @@
     }
     return len;
   }
+
+  private static boolean isTrailingRemovable(char ch) {
+    return ch != ZWNJ && ch != ZWJ && Character.getType(ch) == Character.FORMAT;
+  }
 }

Trade-offs

Reads as "strip trailing invisible marks first" and touches no case of the switch; A keeps the change inside the HEH rule. Choose by taste.

What the checks validated

• The same output as A on all 1,118,481 inputs of the exhaustive run (identical output digests).
• Gradle: the new test and the 30 upstream ckb tests pass (assertions on), and checkGoogleJavaFormat passes; standalone: the new test 7 of 7.
• The same output as A for every BMP character after خانه (the 41-character scan).

The finder's first fix also skipped harakat and TATWEEL after the HEH.

Trade-offs

Fails the upstream vocabulary test (هـ expected ه, got ە; 1 of 28 fails): TATWEEL forces a joining form, and a vowel mark can mark the HEH as the consonant h. A post-pass "if the output ends in HEH, make it AE" would change the same HEH + tail inputs and fail the same row, and more: a word ending in HEH DOACHASHMEE (U+06BE), which the class rewrites to HEH, would also become AE (reasoning).

Change the javadoc instead of the code.

Trade-offs

Keeps the two documented rules in tension (a removed mark would still change the result), unlike the word-initial rule, and leaves two spellings of the same word unmatched.

Method: bounded-exhaustive translation validation of the script normalizers against their documented rules (the hunt's "semi-formal" tier), 2026-10-27.

• The harness (NormalizerTV.java) ran the stock SoraniNormalizationFilter on every string of length up to 4 over a 29-character alphabet (letters including ZWNJ and TATWEEL, the harakat and 6 format characters), 732,541 inputs, and compared it with oracles derived from the class javadoc.
• The deletion-invariance oracle (deleting a character that the javadoc says is removed must not change the result) failed on 23,190 inputs, all a HEH followed only by removable characters at the end. A residual run that excluded that shape found 0 other violations.
• The upstream vocabulary pins HEH + TATWEEL -> HEH, and a wider fix failed it, so the finder narrowed the claim to trailing format characters other than ZWNJ and ZWJ and wrote the public-API JUnit on the stock jars.
• Independent verifiers then reproduced and judged it (see the checks).