ICUFoldingFilter folds the Tajik letter ҷ to the Abkhaz letter ҽ instead of ч, so "Точикистон" does not find "Тоҷикистон"

ICUFoldingFilter (module analysis-icu) folds text for search; its javadoc lists "Diacritic removal (including stroke, hook, descender)". Its data, utr30.nrm, is compiled from Lucene's own folding tables and ICU's NFKC data. One hand-copied line of DiacriticFolding.txt, 04B7>04BC, maps U+04B7 CYRILLIC SMALL LETTER CHE WITH DESCENDER (ҷ, a letter of Tajik and Abkhaz) to U+04BC CYRILLIC CAPITAL LETTER ABKHASIAN CHE, which case folding turns into ҽ U+04BD. Removing the descender gives ч U+0447, which is what every sibling WITH DESCENDER or WITH STROKE line of the same block does (җ to ж, қ to к, ҳ to х, ҹ to ч). So the Tajik phrase "Ҷумҳурии Тоҷикистон" is indexed as "ҽумхурии тоҽикистон", the query "Чумхурии Точикистон" (the same words with plain letters) becomes "чумхурии точикистон", and it finds 0 documents; in Abkhaz, which writes ҷ and ҽ as separate letters, the two fold to one term. There is no error, and the exact spelling still matches itself. The line was copied in 2010 from the 2004 draft data of UTR#30 (since withdrawn) and is in every release from 3.1.0 to 10.6.0 and on main. The fix is a one-line data change plus a regenerated utr30.nrm; it changes indexed terms for these letters.

• ICUFoldingFilter (module analysis-icu): a token filter for "search term folding". Its normalizer NORMALIZER loads the compiled data file utr30.nrm (ICUFoldingFilter.java:69-77); the one-argument constructor uses it, and so does ICUFoldingFilterFactory (SPI name icuFolding), directly or, with a filter argument, wrapped in a FilteredNormalizer2.
• utr30.nrm: compiled by the gradle task regenerateUtr30DataFiles from src/data/utr30/*.txt with ICU's gennorm2 and icupkg. The task regenerates the rule-based parts of the tables and keeps the data lines after a # Rule: verbatim marker as written, up to the next # Rule: line.
• The Cyrillic verbatim block (DiacriticFolding.txt:597-626), headed "Cyrillic script \"composed\" that do not further decompose, so decompose here", "These are from UTR#30 DiacriticFolding.txt", "Rule: verbatim". Each line is source>target in hex.

The block has 26 data lines (:601-626), with the same mappings as the small-letter lines of the 2004 UTR#30 draft data file. 24 map to the base letter of the character's Unicode name (for :620 that base is ABKHASIAN CHE itself); only :618 and :625 map to a different letter, ABKHASIAN CHE.

Ҷ U+04B6 is case-folded to ҷ U+04B7; line 618 maps ҷ to Ҽ U+04BC; case folding maps Ҽ to ҽ U+04BD. getRawDecomposition(U+04B7) is U+04BC, and normalize gives U+04BD for Ҷ ҷ Ҽ ҽ Ҿ ҿ Ӌ ӌ alike, while Ч and ч give ч. U+04B7 has no NFKC decomposition, so the order in which gennorm2 reads the files (a separate bug, LFV-857) does not matter here.

Eight Cyrillic descender letters added to Unicode after the 2004 draft (GHE, PE, SHHA and EL WITH DESCENDER, small and capital) keep their descender (the capitals are only case-folded). That is a separate completeness gap with a different fix (missing lines, not a wrong one).

• Who is exposed. Analysis chains with ICUFoldingFilter: built by hand, with CustomAnalyzer or the icuFolding SPI name, or through a downstream product that wraps the filter. No Lucene analyzer adds it by default. The default normalizer is enough; no option is involved.
• Which text. Tajik and Abkhaz Cyrillic text: Unicode's NamesList marks U+04B7 "Abkhasian, Tajik". Tajik writes, for example, "Тоҷикистон" (Tajikistan).
• Effect. Deterministic for every token with ҷ/Ҷ, which folds to ҽ; what a search then gets wrong depends on the query: one typed with plain ч (for example on a keyboard layout without Tajik letters) does not match the indexed term, so the document is missed; in Abkhaz text a word with ҷ and the same word with ҽ become one term, so they match each other. No exception, no index damage, nothing in the logs; the exact spelling still matches itself.
• How likely. Rare for most indexes, because the text must be Tajik or Abkhaz, but systematic for a Tajik-language search application. No public report was found (see Prior reports), although the line has shipped since 3.1.0 (2011).
• Fixing it changes indexed terms: fields that contain ҷ/Ҷ need a reindex after the fix.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<[ч]> but was:<[ҽ]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.icu.TestICUFoldingCheWithDescender.testCheWithDescenderFoldsToChe(TestICUFoldingCheWithDescender.java:25)

• Contract. ICUFoldingFilter.java:33: "Diacritic removal (including stroke, hook, descender)". The module overview: search term folding "removes distinctions (such as accent marks) between similar characters".
• The block's stated purpose. "Cyrillic script \"composed\" that do not further decompose, so decompose here" (DiacriticFolding.txt:597). ABKHASIAN CHE is not a decomposition of CHE WITH DESCENDER: the Unicode name gives the base letter CHE, and the Unicode 1.0 names were "CHE WITH RIGHT DESCENDER" for U+04B7 and "IE HOOK" for U+04BD.
• The siblings. Every other WITH DESCENDER or WITH STROKE line of the block keeps the base letter of the character's name, including CHE WITH VERTICAL STROKE (:619, to ч) and ABKHASIAN CHE WITH DESCENDER (:620, to ABKHASIAN CHE); the only other line that sends a different letter to ABKHASIAN CHE is :625 (KHAKASSIAN CHE, same origin).
• The copied source is inconsistent and withdrawn. The UTR#30 draft data ("DRAFT 1 - 2004-07-14 ... Subject to change") maps capital Ҹ U+04B8 to ABKHASIAN CHE but small ҹ U+04B9 to CHE. Lucene copied only the small-letter lines, so it already folds Ҹ to ч, not to ҽ. The report defines diacritic removal as mapping to "related base characters", so that users can "search for words with accented characters in them by supplying the equivalent word spelled in base letters only"; the Unicode site lists the draft as withdrawn.
• No test or decision records it. No upstream test asserts any Cyrillic folding, and no javadoc, CHANGES or MIGRATE entry or issue discusses ҷ. LUCENE-1343 (#2420), which added the table, lists five intended differences from ASCIIFoldingFilter, none Cyrillic. In LUCENE-10238 (#11274) the verbatim sections are called "basically hacks" and edited: maintained data, not a frozen copy.

The strongest case for "intended": the block says "These are from UTR#30 DiacriticFolding.txt" and "Rule: verbatim", and line 618 has the same mapping as the draft (which maps both U+04B6 and U+04B7 to U+04BC). The javadoc names a category, not a promise per character; eight newer Cyrillic descender letters keep their descender. ҷ is a letter of its own in Tajik, so a maintainer may prefer not to fold it, and any change forces a reindex.

Why it fails: Lucene does not reproduce the draft anyway (it differs for Ҹ, and the draft maps a case pair to two different letters); the block's stated job is to decompose, and the target is a third letter that Abkhaz uses next to ҷ, which no reading of "descender removal" yields. "Do not fold ҷ" is a fair position. No current normative Unicode source mandates U+0447 (UTR#30 is a withdrawn draft), so do not claim one; ask the maintainers to choose between ч and not folding ҷ, since either removes the merge with ҽ. Reindexing is the normal cost of a change to this data; every ICU upgrade regenerates utr30.nrm. Framing note from the independent reproducer: a plain-ч query would also miss ҷ if ҷ were left unfolded, so the distinctive harm of today's line is the merge with ҽ and the break from its siblings.

One fix and one variant were applied in a scratch worktree at a9779ef3c193 and run through gradle; the other options were not run.

One line of DiacriticFolding.txt changes. The binary utr30.nrm (61,776 bytes before and after) and the two SHA-1 values in src/generated/checksums/regenerateUtr30DataFiles.json follow from it: upstream, ./gradlew :lucene:analysis:icu:regenerateUtr30DataFiles produces both (the task runs GenerateUTR30DataFiles, downloads ICU's NFKC data files and compiles ICU4C 78.3). Here the file was rebuilt with only that task's last two steps (gennorm2 over the sorted *.txt, then icupkg -tb) and ICU4C 77.1, whose rebuild of the unchanged sources is byte-identical to the shipped file. An independent run of the full task, with only line 618 changed, produced a byte-identical utr30.nrm and the same two checksum lines. Add a test such as the repro's testCheWithDescenderFoldsToChe to TestICUFoldingFilter.

--- a/lucene/analysis/icu/src/data/utr30/DiacriticFolding.txt
+++ b/lucene/analysis/icu/src/data/utr30/DiacriticFolding.txt
@@ -615,7 +615,7 @@
 04AD>0442
 04B1>04AF
 04B3>0425
-04B7>04BC
+04B7>0447
 04B9>0447
 04BF>04BC
 04C4>043A
(+ utr30.nrm regenerated, + the two new SHA-1 values in regenerateUtr30DataFiles.json)

Trade-offs

• Reindexing. Indexed terms change for text with ҷ/Ҷ: documents indexed before the fix keep тоҽикистон, new queries produce точикистон. Add a CHANGES entry (and, if the committers want, a MIGRATE note) that ICUFoldingFilter output changes for U+04B6/U+04B7 and affected fields need reindexing. Earlier changes to this data (ICU upgrades, LUCENE-3747, LUCENE-10238) shipped with plain CHANGES entries and no reindex note; MIGRATE.md has a precedent for an analysis change that asks users to reindex (Romanian analysis, lucene/MIGRATE.md:873).
• No API change; no performance effect is expected (one mapping changes, the file size does not); not benchmarked.
• Copies outside Lucene need the same change separately: downstream copies of the text file, and Lucene.NET's compiled utr30.nrm, which would have to be regenerated.

What the checks validated

• Stock main a9779ef3c193 (gradle, seed DEADBEEF): the repro fails 2 of 3 with assertions on and off; TestICUFoldingFilter and TestICUFoldingFilterFactory pass (6 tests); the generated-file checksum check passes.
• With the fix: the repro passes 3 of 3 with assertions on and off; the two upstream suites pass (6 tests); all of :lucene:analysis:icu:test with the repro added passes (17 classes, 161 tests, 0 failures, 10 skipped); regenerateUtr30DataFilesChecksumCheck passes with the updated checksum file; ecjLintTest passes.
• An exhaustive comparison of the stock and the fixed utr30.nrm over all 1,112,064 non-surrogate code points, each normalized on its own, differs at exactly U+04B6 and U+04B7, which now fold to U+0447; an independent fact check repeated it with its own code and got the same 2 code points.
• The fixed utr30.nrm was rebuilt from the patched sources with gennorm2 and icupkg -tb; the same rebuild of the unchanged sources is byte-identical to the shipped file, and the full upstream task (ICU4C 78.3) with only line 618 changed gives a byte-identical fixed file (SHA-1 d14cf338...).
• checkGoogleJavaFormat flagged only the new test's line wrapping; the formatted test is the final repro file and was re-run: 2 of 3 fail on stock, 3 of 3 pass with the fix.
• After the fact checks, the repro's comments were corrected ("selected sibling letters" instead of "every letter"; code unchanged): google-java-format and ecjLintTest pass, it fails 2 of 3 on stock and passes 3 of 3 with the fix, assertions on and off, and fix-with-test.diff was rebuilt from fix.diff plus this file; it applies cleanly.

The finder's original sketch: the same change plus the neighbouring line with the same origin and the same target.

Trade-offs

Its Unicode 1.0 name was "CHE WITH LEFT DESCENDER", which points to ч, but its current name, KHAKASSIAN CHE, carries no diacritic, the 2004 draft grouped it with ABKHASIAN CHE, and no test asserts it. Treat it as a suspect companion mapping for maintainers to adjudication, not as part of the minimal fix or as a second bug.

What the checks validated

• With this variant: the repro passes 3 of 3 (assertions on), TestICUFoldingFilter and TestICUFoldingFilterFactory pass (6 tests), and the checksum check passes.
• Its utr30.nrm is byte-identical to the hunt's rebuilt file, whose exhaustive comparison with the stock one differs at exactly U+04B6, U+04B7, U+04CB and U+04CC.

For maintainers who want ҷ to stay distinct. Not run.

Trade-offs

Removes the wrong merge with ҽ, but a query with plain ч still misses ҷ, and it departs from every sibling line of the block.

Map ҷ to ч and Ҷ to Ч before the filter (for example with a MappingCharFilter), or exclude them with the factory's filter argument (a UnicodeSet of the characters to fold). Not run.

Trade-offs

Both change indexed terms too, so they also need a reindex; with filter, the excluded letters are not case-folded either. The filter route only removes the merge with ҽ: a plain-ч query still misses, as with not folding ҷ; only the char-filter mapping fixes that.

Method: bounded-exhaustive translation validation of the folding data (semi-formal tier), in a hunt unit that checked the UTR#30 verbatim tables against Unicode character names and the compiled .nrm.

• An independent model of the source tables (gennorm2's precedence by sorted file name, recursive mapping, canonical reordering) agreed with the shipped utr30.nrm and with gennorm2's own combined output on all 1,114,112 code points (0 mismatches); a mutation of lines 618 and 625 in the model was flagged at exactly the affected code points (U+04B7 and U+04CC in the raw mapping; U+04B6, U+04B7, U+04CB and U+04CC after recursion), and a gennorm2 rebuild of the unchanged sources is byte-identical to the shipped file.
• A name oracle (ICU4J 78.3, Unicode 17) took the 1,467 letters named "<base> WITH ..." whose base letter exists: 1,050 fold to the base letter's fold, 355 are left unfolded, 57 give the base plus something, and 5 fold to a wrong base letter. Two of the five are U+04B6/U+04B7, from line 618; the other three (U+06C0 and two presentation forms) follow Unicode's own canonical decomposition of U+06C0 into U+06D5 plus hamza above (Lucene's line 637 then removes the hamza as a diacritic), not a wrong Lucene line.
• The finding was reduced to a public-API JUnit and checked by an independent reproducer, reading-only reviews by a model from another family (OpenAI GPT-5.6), an intended-behaviour devil's advocate, an adjudication, a history, prior-report and security follow-up, and an audit. Found 2026-10-28 on main a9779ef3c193.