ICUFoldingFilter folds ҧ to п but not ԥ, its modern Abkhaz counterpart: none of the 24 Cyrillic hook, stroke and descender letters added in Unicode 4.1-7.0 is folded

ICUFoldingFilter (module analysis-icu) folds text for search; its javadoc lists "Diacritic removal (including stroke, hook, descender)". Cyrillic letters such as ҧ PE WITH MIDDLE HOOK have no Unicode decomposition, so the only thing that folds them is a hand-written block of DiacriticFolding.txt, compiled into utr30.nrm. That block was copied in 2010 from the 2004 draft data of UTR#30, which was based on Unicode 4.0, and has never been extended: 26 lines, none above U+04CF. So 46 of the 48 Cyrillic letters of this kind from Unicode 1.1-3.2 fold to their base letter (ҷ/Ҷ fold to a wrong letter, a separate bug), while none of the 24 added in Unicode 4.1-7.0 loses its mark: ӷ ӻ ӽ ӿ ԓ ԡ ԣ ꚋ ԥ ԧ ԩ ԯ and their capitals are only lower-cased. In Abkhaz, Unicode's character list marks ҧ as the letter of "older orthographies" and ԥ as "used in modern Abkhaz orthography": the older spelling "аҧсуа" is indexed as апсуа, the modern spelling "аԥсуа" stays аԥсуа, and it finds 0 of 1 documents; a document in the modern spelling is missed by the older and the plain spelling as well. No error is raised, and each exact spelling still matches itself. The file header says "Created from Unicode 5.2 UCD", yet 18 of the 24 code points were already in Unicode 5.2. Every release from 3.1.0 to 10.5.2, 10.6.0 (by reading) and main is affected: the 138 releases up to 10.5.2 ship 13 distinct compiled tables, and a standalone check fails on at least one release of each. The fix is 12 added data lines and a regenerated utr30.nrm; it changes indexed terms for these letters, so affected fields need reindexing.

• ICUFoldingFilter (module analysis-icu): a token filter for "search term folding". Its normalizer NORMALIZER loads the compiled data file utr30.nrm (ICUFoldingFilter.java:69-77); the one-argument constructor uses it, and so does ICUFoldingFilterFactory (SPI name icuFolding), directly or, with a filter argument, wrapped in a FilteredNormalizer2, which can only exclude characters.
• utr30.nrm: compiled by the gradle task regenerateUtr30DataFiles from src/data/utr30/*.txt with ICU's gennorm2 and icupkg. The task first regenerates the rule-based parts of the tables from Unicode properties and keeps the data lines after a # Rule: verbatim marker as written.
• Why only a table line helps: none of the 72 Cyrillic letters of this kind (48 older, 24 newer) has an NFKD decomposition or the Unicode Diacritic property, so neither NFKC nor the [:Diacritic:] rule touches them. A Cyrillic letter with a hook, stroke or descender is folded only if the block has a line for it.

DiacriticFolding.txt:597-626, headed "Cyrillic script \"composed\" that do not further decompose, so decompose here", "These are from UTR#30 DiacriticFolding.txt" and "# Rule: verbatim": 26 lines from 047D>0461 to 04CE>043C, for example 0493>0433 (ғ GHE WITH STROKE to г), 04A7>043F (ҧ PE WITH MIDDLE HOOK to п) and 04C6>043B (ӆ EL WITH TAIL to л). The UTR#30 draft data it names is "DRAFT 1 - 2004-07-14", "Base version of Unicode: 4.0.0", and has no line for any of the 24 newer letters. Capitals need no line: nfkc_cf.txt case-folds them to the small letter first (for example 0524>0525). The file's header (:24) says "Created from Unicode 5.2 UCD".

Over every Cyrillic letter named "<base> WITH <hook, stroke, descender, tail, tick, upturn or bar>" whose base letter exists: Unicode 1.1, 36 fold to the base letter and 2 (ҷ/Ҷ, CHE WITH DESCENDER) to a wrong letter, a separate bug; 3.0, 2 of 2; 3.2, 8 of 8; then 4.1, 0 of 2; 5.0, 0 of 8; 5.1, 0 of 6; 5.2, 0 of 2; 6.0, 0 of 2; 7.0, 0 of 4. The same check also lists three Unicode 15.0 subscript and modifier letters of this kind (U+1E067, U+1E06B, U+1E06D); they do fold, through their compatibility decomposition to the older ґ, ҫ and ұ, which have block lines.

89 Latin letters with a stroke, bar, hook, descender, tail, curl or similar mark are unfolded too: 76 added in Unicode 6.0-16.0, and 13 older ones (Unicode 1.1-5.1) that the ASCIIFoldingFilter-derived Latin block lists only as commented-out placeholders (such as #019B>, DiacriticFolding.txt:283). That block deliberately leaves many letters out, as ASCIIFoldingFilter does, so they are not part of this report.

• Who is exposed. Analysis chains with ICUFoldingFilter: built by hand, with CustomAnalyzer or the icuFolding SPI name, or through a downstream product that wraps the filter. No Lucene analyzer adds it by default; once chosen, its default data is enough.
• Which text. Cyrillic text in an orthography that uses one of the 24 letters. Unicode's NamesList names modern Abkhaz for ԥ (cross-referenced to ҧ, "older orthographies for Abkhasian"), Yupik for ӷ, Itelmen for ԓ and Nenets for ԯ.
• Effect. Deterministic, for every token with one of these letters: it keeps its hook, stroke or descender, so the same word written with the older letter form or with the plain base letter (for example on a keyboard without the letter) gives a different term, and the query misses. Either direction misses: a modern-spelling document is missed by the older and the plain spelling too. No exception, no index damage, nothing in the logs; each exact spelling still matches itself.
• How likely. Rare for most indexes: the text must use one of these minority-language letters and the spellings must differ. Nobody has reported it since 3.1.0 (2011).
• Fixing it changes indexed terms: fields that contain these letters need a reindex after the fix.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<[п]> but was:<[ԥ]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.icu.TestICUFoldingPeWithDescender.testPeWithDescenderFoldsToPe(TestICUFoldingPeWithDescender.java:25)

• Contract. ICUFoldingFilter.java:33: "Diacritic removal (including stroke, hook, descender)". The module overview: search term folding "removes distinctions (such as accent marks) between similar characters" and is "a more thorough replacement for ASCIIFoldingFilter and LowerCaseFilter that applies the same ideas to many more languages". Unlike ASCIIFoldingFilter ("only those characters with reasonable ASCII alternatives are converted"), it has no coverage hedge.
• The block's stated job. "Cyrillic script \"composed\" that do not further decompose, so decompose here" (DiacriticFolding.txt:597). The 24 letters are exactly such characters.
• The file claims Unicode 5.2. "Created from Unicode 5.2 UCD" (:24), yet 18 of the 24 code points (Unicode 4.1-5.2) existed when the block was copied in 2010. In that 2010 file the three large sections already had post-4.0 entries (157 in the [:Diacritic:] list, 117 in the Latin block, 113 in "Additional signs and diacritic"); the Cyrillic block had none (as did the small "Additional Arabic/Hebrew decompositions" block).
• Old and new siblings disagree. ҧ, ғ, ҕ, ҳ, ӆ, ң, ӈ, ҭ and 38 more older letters fold to their base letter; ԥ, the modern counterpart of ҧ, and the 23 other newer letters do not. Only the Unicode version decides.
• The source expects growth. UTR#30 section 5.3: "As characters are added to [Unicode] many of them need to be added to the definition of foldings described here." Its data file: "Additional instances of such characters are expected to be added over time, so there will be future additions to this table."
• No decision recorded. Where the maintainers chose not to fold a Latin letter, the file keeps a commented-out placeholder (67 lines such as #019B>); the Cyrillic block has none. No test, CHANGES entry or issue mentions these letters, no upstream test asserts any Cyrillic folding of the filter, and no commit ever had a line for any of them.

The strongest case for "intended": the javadoc says the filter applies "foldings from UTR#30", and the module overview says it "implements many of the foldings specified in" that report. UTR#30 says its foldings "cover Unicode, Version 4.0" and lists diacritic removal under "Provisional foldings". In its table the source set of "Diacritic removal (includes stroke, hook, descender)" is "Latin/Greek/Cyrillic characters with diacritics <>", where <> means the characters listed in its data file, which covers Unicode 4.0 and lists none of these letters. The report is a withdrawn draft; the javadoc names kinds of folding of a finite table. Maintainers knowingly left the block static: in #4820 (LUCENE-3747) "Other sections were based either on AsciiFoldingFilter or UTR#30, neither of which has changed" and "It doesn't need to be perfect"; in #11274 the verbatim sections are "(basically hacks)"; in #2420 removing diacritics is called "language-specific". ICU's root collation keeps these letters, old and new, primary-distinct from their base letters (70 of the 72; only ґ/Ґ GHE WITH UPTURN is primary-equal). So this is a data-coverage enhancement.

Why it fails: the report did limit its foldings to Unicode 4.0, but it also says its table must grow as characters are added, so its 4.0 listing reads as a provisional snapshot rather than a decision to leave later letters unfolded; this is an argument, not proof. Lucene maintains its own copy: its Latin section comes from ASCIIFoldingFilter, and the file introduced in 2010 already had post-4.0 source code points in its other large sections, while the Cyrillic block had none. "Neither of which has changed" explains why the 2012 automation skipped the block; it is not a decision that newer letters stay unfolded. The language-specific and collation points apply equally to the older letters, which the filter already folds. Verdict: not intended, a coverage defect; the label (bug or improvement) is the committers' call.

Two options were applied in a scratch worktree at a9779ef3c193 and run through gradle with the upstream regeneration task; the others were not run.

Twelve lines go after 04CE>043C (DiacriticFolding.txt:626); each maps a small letter to the base letter named in its Unicode name, as 24 of the block's 26 existing lines do (for example 0493>0433 GHE WITH STROKE, 04A7>043F PE WITH MIDDLE HOOK); the other two are 04B7>04BC (CHE WITH DESCENDER to ABKHASIAN CHE, the separate bug) and 04CC>04BC (KHAKASSIAN CHE, whose name has no "WITH"). Capitals need no line: they are case-folded to the small letter first. Then ./gradlew :lucene:analysis:icu:regenerateUtr30DataFiles rebuilds utr30.nrm and updates the two SHA-1 values in src/generated/checksums/regenerateUtr30DataFiles.json (it compiles ICU4C 78.3 and downloads ICU's NFKC data). Add a test like the repro's first one to TestICUFoldingFilter.

--- a/lucene/analysis/icu/src/data/utr30/DiacriticFolding.txt
+++ b/lucene/analysis/icu/src/data/utr30/DiacriticFolding.txt
@@ -624,6 +624,18 @@
 04CA>043D
 04CC>04BC
 04CE>043C
+04F7>0433
+04FB>0433
+04FD>0445
+04FF>0445
+0513>043B
+0521>043B
+0523>043D
+0525>043F
+0527>04BB
+0529>043D
+052F>043B
+A68B>0442
 
 # Additional signs and diacritic, from examination of [:Mark:]&[:Lm:]
 # Rule: verbatim
(+ utr30.nrm and the two checksum values, produced by regenerateUtr30DataFiles)

Trade-offs

• Reindexing. Indexed terms change for text with these letters: documents indexed before the fix keep аԥсуа, new queries produce апсуа. Add a CHANGES entry (and, if the committers want, a MIGRATE note) that ICUFoldingFilter output changes for these 24 code points and affected fields need reindexing. The table is already regenerated at every ICU upgrade, and lucene/MIGRATE.md:873 is a precedent for an analysis change that asks users to reindex (Romanian analysis).
• No API change; no performance effect is expected (12 more mappings in a 61 KB table); not benchmarked.
• Copies of the data file outside Lucene need the same change separately.

What the checks validated

• Stock main a9779ef3c193 (gradle, seed DEADBEEF): the repro fails 2 of 2 with assertions on and off; TestICUFoldingFilter and TestICUFoldingFilterFactory pass (6 tests); the generated-file checksum check passes.
• With the 12 lines, the upstream regenerateUtr30DataFiles task (ICU4C 78.3 built from source, ICU's NFKC data downloaded) changes exactly DiacriticFolding.txt (+12 lines), utr30.nrm (61,776 to 61,888 bytes, SHA-1 83092802...) and two checksum values; that utr30.nrm is byte-identical to the one the hunt built with gennorm2 and icupkg -tb.
• An exhaustive comparison of the stock and the fixed utr30.nrm over all 1,112,064 Unicode scalar values, each normalized on its own, differs at exactly the 24 code points, each now folding to its base letter.
• With the fix: the repro passes 2 of 2 with assertions on and off; the two upstream suites pass (6 tests); the checksum check, ecjLintTest and checkGoogleJavaFormat pass; fix-with-test.diff applies cleanly with git apply, and all of :lucene:analysis:icu:test passes with it (17 classes, 160 tests, 0 failures, 10 skipped), with assertions on and off.

For one issue or PR on the Cyrillic block: the 12 lines above plus the separate fix that makes CHE WITH DESCENDER fold to ч instead of the different letter ҽ.

Trade-offs

Two independent changes in one PR; each can be reviewed and reverted on its own. Reindexing covers 26 code points instead of 24.

What the checks validated

• The combined diff applies cleanly to a9779ef3c193. Regenerated by the same task, the table differs from the stock one at exactly 26 code points (the 24 above, plus U+04B6/U+04B7, now folding to ч); this bug's repro, the sibling's repro and the two upstream suites pass together (4 classes, 11 tests, 0 failures); the checksum check passes.

In GenerateUTR30DataFiles, map each Cyrillic letter named "<base> WITH <hook, stroke, descender, tail, ...>" whose base letter exists to that base, so future Unicode versions are covered automatically. Not run.

Trade-offs

A larger change to the generator; the name patterns need care (YERU WITH BACK YER is a ligature-like letter, not a base plus a mark); and the maintainers chose hand-curated lists for this data on purpose.

Say in the javadoc that the Cyrillic diacritic table covers the letters of the UTR#30 draft data (Unicode 4.0). Not run.

Trade-offs

Keeps indexes stable but leaves the ҧ/ԥ inconsistency in place.

Map the 24 letters to their base letters before the filter (for example with a MappingCharFilter), or pass a custom Normalizer2 to the two-argument constructor. Not run as a workaround. One pitfall was executed: if the custom normalizer is a fixed utr30.nrm loaded with Normalizer2.getInstance(InputStream, name, mode), give it a name other than "utr30". ICU4J caches these instances by name, so once ICUFoldingFilter.NORMALIZER is loaded, a second load under "utr30" returns the stock normalizer, which leaves ԥ unchanged.

Trade-offs

Both change indexed terms, so they also need a reindex. The factory's filter argument cannot help: it can only exclude characters from folding.

Method: bounded-exhaustive translation validation of the folding data (semi-formal tier), in a hunt unit that checked the scope of ICUFoldingFilter's diacritic removal against the UTR#30 domain and against collation primaries.

• The harness ran the stock main jar (ICU4J 78.3, Unicode 17) over every Unicode scalar value and, for each Latin, Greek and Cyrillic letter whose name is "<base> WITH <diacritic>" with an existing base letter, compared fold(letter) with fold(base), bucketed by Unicode age. An earlier unit's name oracle had listed the unfolded letters without filing them; the age split was found here (2026-10-28).
• Cyrillic: 48 of 48 letters of this kind from Unicode 1.1-3.2 lose the mark (46 to the base letter, 2 to a wrong letter, a separate bug), 0 of 24 from 4.1-7.0. The 3 Unicode 15.0 subscript and modifier letters of this kind fold through their compatibility decomposition. Latin has 89 unfolded letters of this kind (76 from Unicode 6.0-16.0, 13 older ones that the Latin block keeps as commented-out placeholders), not claimed because the Latin block deliberately leaves letters out.
• The finding was reduced to a public-API JUnit and checked by an independent reproducer with its own oracle, reading-only reviews by a model from another family (OpenAI GPT-5.6), an intended-behaviour devil's advocate, an adjudication, a history, prior-report and security follow-up, and an audit.