SoraniNormalizer keeps invisible format characters from outside the BMP, so a Sorani word with one no longer matches

SoraniAnalyzer, SoraniNormalizationFilter and the soraniNormalization factory delete invisible formatting characters (Unicode general category Cf), as SoraniNormalizer's javadoc says: "Harakat, tatweel, and formatting characters such as directional controls are removed." The test reads one UTF-16 char at a time, Character.getType(s[i]) == Character.FORMAT (SoraniNormalizer.java:121). A format character outside the Basic Multilingual Plane (BMP), such as U+E0001 LANGUAGE TAG or a tag character U+E0020..U+E007F, is stored as a surrogate pair, and each half has type SURROGATE, so it is never deleted: all 43 BMP Cf code points are removed and all 127 supplementary ones (JDK 25) are kept. With the default SoraniAnalyzer, the word کوردستان with U+E0001 inside or at the end is not found by the plain word (1 of 3 documents found; the right-to-left-mark spelling is), and a trailing such character also blocks stemming. With these inputs nothing throws. Sorani letters and marks are all BMP characters, and emoji flags are tokenized separately, so only stray or constructed input hits it. The code is as old as the analyzer (4.7.0, 2014); testing the code point and deleting its chars fixes it.

• UTF-16 char: Java stores text as 16-bit char values. Every Sorani letter and mark fits in one char: they are in the Basic Multilingual Plane (BMP).
• Supplementary character: a code point above U+FFFF, stored as two chars, a surrogate pair. U+E0001 LANGUAGE TAG is the pair U+DB40 U+DC01. A surrogate alone is not a character; its general category is SURROGATE.
• Format character (Cf): an invisible character that affects layout or processing, such as the right-to-left mark U+200F, the soft hyphen U+00AD or the tag characters U+E0020..U+E007F. Character.getType(int) returns FORMAT for them; the char overload cannot, and its javadoc says it "cannot handle supplementary characters".

1. SoraniNormalizer.normalize walks the term one char at a time and switches on the char: explicit cases fold letters and delete harakat, tatweel and ZWNJ (SoraniNormalizer.java:72-119).
2. Every other char reaches the default branch: if (Character.getType(s[i]) == Character.FORMAT) { len = delete(s, i, len); i--; } (:120-124).
3. For U+E0001 the two halves U+DB40 and U+DC01 arrive one by one; getType returns SURROGATE (19) for each, not FORMAT (16), so neither is deleted and the character stays in the term.
4. SoraniNormalizationFilter "applies {@link SoraniNormalizer}" (SoraniNormalizationFilter.java:24). SoraniAnalyzer runs it right after StandardTokenizer, which keeps U+E0001 and U+E0067 inside the word token (word-break rule WB4), and again in normalize (SoraniAnalyzer.java:113-130).

With JDK 25 there are 43 Cf code points in the BMP, all removed (including non-directional ones such as U+00AD, U+0600..U+0605, U+200B, U+200D and U+FEFF), and 127 outside it, all kept: U+110BD, U+110CD, U+13430..U+1343F, U+1BCA0..U+1BCA3, U+1D173..U+1D17A, U+E0001 and U+E0020..U+E007F.

With the default SoraniAnalyzer the plain word and the right-to-left-mark spelling give the term [U+06A9 U+0648 U+0631 U+062F U+0633] (کوردس, after stemming); U+E0001 inside gives [U+06A9 U+0648 U+0631 U+062F U+E0001 U+0633], and U+E0001 at the end gives the unstemmed [U+06A9 U+0648 U+0631 U+062F U+0633 U+062A U+0627 U+0646 U+E0001].

• Who is exposed. The stock SoraniAnalyzer with any constructor (indexing and normalize), any chain with SoraniNormalizationFilter, and the soraniNormalization factory (SoraniNormalizationFilterFactory.java:40; its normalize is the same filter, :60-63).
• What it takes. One of the 127 supplementary format characters inside or at the end of a word in a Sorani-analysed field. Sorani letters and marks are all BMP characters, so Sorani text has no need of them. The best-known real source, emoji subdivision flags (a black flag followed by tag characters), does not reach a Sorani word: StandardTokenizer emits the flag as its own <EMOJI> token, with or without a space, and the word token stays clean and is stemmed. So it takes a bare stray tag character, a deprecated language tag or another exotic format control glued to a word: garbled or deliberately constructed input.
• What changes for the user. The word with the character and the plain word become different terms, so a search for the plain word misses the document (1 of 3 found in the repro); a trailing such character also blocks stemming. Analyzer.normalize keeps the character too, so wildcard, prefix and fuzzy terms built from such text keep it (read from the query parser; not run end to end).
• What does not happen. With the inputs run, no exception; the surrogate pair stays intact, so the term is valid UTF-16; other tokens, fields and documents are unaffected; the run time stays small (about 1,200,000 chars of such words in 67-137 ms; one input size, no scaling measured).
• Scale. Deterministic, with assertions on or off, at any index size.
• JDK dependence. The affected set follows the JDK's Unicode tables (127 with JDK 25); U+E0001 and the tag characters have been Cf since Unicode 3.1, so every JDK Lucene supports shows it. Only JDK 25 was run.
• How likely. Contrived for accidental hits; the documented searches found no report in the 12 years since 4.7.0.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<کورد[]ستان> but was:<کورد[󠀁]ستان>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.ckb.TestSoraniNormalizationFilterSupplementaryFormatChar.testRemovesFormatCharOutsideBmp(TestSoraniNormalizationFilterSupplementaryFormatChar.java:31)

• Contract. "Harakat, tatweel, and formatting characters such as directional controls are removed." (SoraniNormalizer.java:33, unchanged since 4.7.0). Directional controls are an example; the code's criterion is Character.FORMAT, and every BMP Cf character is removed (ZWNJ by its own case, the other 42 by this branch), including the soft hyphen, ZWJ, the BOM and U+0600..U+0605. Only the UTF-16 encoding decides which Cf characters survive.
• The contract is public. SoraniNormalizationFilter: "A {@link TokenFilter} that applies {@link SoraniNormalizer} to normalize the orthography." SoraniNormalizer was a public class from 4.7.0 through 9.12.3, and Lucene's javadoc builds document protected members (-protected; javadoc.access=protected in the 4.x ant build), so the release javadoc builds included the sentence (build settings read, not the published pages); 10.0.0 made the class package-private (LUCENE-10561), changing only modifiers.
• Nothing presents it as a choice. The only upstream test that reaches the FORMAT branch is TestSoraniStemFilter.testVocabulary: 11 words of its Sorani vocabulary end in U+202E RIGHT-TO-LEFT OVERRIDE, a BMP format character no switch case matches, and their expected stems lack it, so upstream relies on this branch to remove a BMP directional control (with the branch disabled, that test fails; executed by the fact check). No upstream test uses a supplementary format character. TestSoraniNormalizationFilter.testRemove covers tatweel, harakat and ZWNJ, all explicit cases. The issue that added the analyzer (#6442, LUCENE-5379) never mentions format or supplementary characters, and no comment, CHANGES or MIGRATE entry limits the removal to the BMP.
• Precedent. lucene/CHANGES.txt:8852: "LUCENE-6914: Fixed DecimalDigitFilter in case of supplementary code points." DecimalDigitFilter runs two filters after this one in SoraniAnalyzer. #2763 (LUCENE-1689, 2009) set the rule that "supplementary character support should be fixed for code that works with char/char[]", and TurkishLowerCaseFilter makes the same kind of category test by code point.

The strongest case for "intended" (the intended-behaviour check and the cross-model reviewers): this is a Sorani orthographic normalizer, Sorani text only needs BMP marks, the removal sentence sits on a package-private class, and keeping tag characters preserves emoji flags.

Why it fails: the code tests the general Cf category, not a list of Sorani marks, and already removes non-directional BMP Cf characters; the public filter links the class as what it applies, and the class was public with this javadoc through 9.x; there is no emoji-preservation intent, because the same branch already strips the BMP ZWJ from ZWJ emoji sequences ([U+1F468 U+200D U+1F469 U+200D U+1F467] becomes [U+1F468 U+1F469 U+1F467]), and flags survive only because their tags are supplementary; and the JDK documents that getType(char) cannot classify these characters. Every verifier rejected the argument, and the cross-model reviews rated it weak. What survives is plausibility, not intent: such input is rare, which lowers the priority but does not make the behaviour intended. No record states the 2014 authors' intent either way; changing indexed terms is the maintainers' compatibility call.

One fix was built and validated in a scratch worktree at a9779ef3c193; the hunt had already checked an equivalent version. Two alternatives are listed for the maintainers' discussion.

In the default branch, read the code point with Character.codePointAt(s, i, len), test Character.getType(int), and delete Character.charCount(cp) chars with the existing StemmerUtil.deleteN. BMP chars take the same path as before; a low surrogate of a kept supplementary character is looked at again on the next step, but alone it is SURROGATE, so nothing else changes.

         default:
-          if (Character.getType(s[i]) == Character.FORMAT) {
-            len = delete(s, i, len);
+          final int cp = Character.codePointAt(s, i, len);
+          if (Character.getType(cp) == Character.FORMAT) {
+            len = deleteN(s, i, len, Character.charCount(cp));
             i--;
           }

+import static org.apache.lucene.analysis.util.StemmerUtil.deleteN;

Trade-offs

• Changes output only for the 127 supplementary Cf code points.
• Emoji flags fold. Emoji subdivision-flag tokens (U+1F3F4, tag characters, U+E007F) lose their tags, so the Scotland and England flags both become the term U+1F3F4. The stock branch already folds ZWJ emoji sequences the same way (the family emoji loses its ZWJs), so the two become consistent, but a reviewer may raise it.
• Indexed terms change for text with these characters, so affected fields need reindexing; a CHANGES entry should say so.
• If the fix for LFV-668 (word-final HEH before a direction mark, same method) goes in too, its isWordFinal helper should read code points as well, or HEH followed by a supplementary format character keeps this bug there.

What the checks validated

• Stock main: the paste-ready test fails 3 of 4, assertions on and off; the 30 upstream Sorani tests pass.
• With fix A, org.apache.lucene.analysis.ckb.*: 6 classes, 34 tests, 0 failures, assertions on and off.
• With fix A, the whole :lucene:analysis:common:test task: 340 classes, 1,913 tests, 0 failures, 18 skipped (skips not examined).
• ecjLintMain, ecjLintTest and checkGoogleJavaFormat of the module pass.
• The standalone runner with the fixed class first on the classpath: the test passes on main with -ea and -da, and so do the independent reproduction's test over all 127 supplementary Cf code points (4 of 4) and the finder's test (3 of 3).
• The hunt's two equivalent sketches passed the finder's, the independent reproduction's and separate tests; 28 upstream Sorani tests pass with and without the finder's.

Exempt U+E0020..U+E007F (and perhaps U+E0001) from removal, so emoji flags keep their identity.

Trade-offs

Not built or run. It leaves up to 97 of the 127 supplementary Cf code points in Sorani words, contradicts the javadoc's general rule and is inconsistent with the ZWJ removal; only worth it if maintainers object to the flag folding.

A javadoc note that only BMP formatting characters are removed.

Trade-offs

Cheap, but contradicts the documented rule and the project's own fixes of char-wise analysis code (LUCENE-6914, LUCENE-1689) for no benefit.

Method: a bytecode call-site inventory (the hunt's semi-formal tier), in a hunt unit about supplementary-plane characters in parsers and char-wise filters. A scanner built on the JDK 25 ClassFile API read the 27 non-sandbox, non-test module jars of main a9779ef3c193 and listed every call of the char overloads of Character category tests (getType, isLetter, ...), which cannot classify a supplementary character. One row was SoraniNormalizer.normalize:121 Character.getType(C).

Code reading found it to be the only Character.FORMAT removal site in the analysis modules. A probe put 7 BMP Cf characters (U+200F, U+200E, U+202B, U+2066, U+FEFF, U+00AD, U+061C) and 7 supplementary ones (U+E0001, U+E0067, U+E007F, U+1BCA0, U+1D173, U+110BD, U+13430) into a Sorani word: all BMP ones were removed, none of the supplementary ones. A public-API JUnit on the stock main jars then confirmed it through SoraniNormalizationFilter and SoraniAnalyzer (found 2026-10-27).