Since 9.0, ICUFoldingFilter turns ´ into a space inside the term: the build lets NFKC override Lucene's own folding tables

ICUFoldingFilter folds with utr30.nrm, which the build compiles with ICU's gennorm2 from ICU's nfc.txt, nfkc.txt and nfkc_cf.txt and Lucene's five UTR#30 tables. When two files map the same character, the later file wins. Up to 8.11.4 the build passed the NFKC files first and Lucene's tables last; a 2021 build change (010e3a1ba96, LUCENE-9900, PR #61), meant to make regeneration incremental, sorts the files by name instead, which puts Lucene's tables first. Since 9.0.0 the NFKC mapping wins on all 292 code points both map; on main's data 271 of them fold differently than under the old order. Lucene's "Remove diacritics" rule deletes ´ ¨ ¯ ¸, but NFKC maps them to a space plus a combining mark, so with keyword or whitespace tokenization O´Brien is indexed as o brien, and through StandardTokenizer カ゛ becomes カ , while ` and ^ (no NFKC mapping) are still deleted. About 240 of the 271 changes (modifier letters such as ʰ -> h) are arguably improvements. Silent: no error, no release note, no test pins either behaviour. Restoring the documented order and keeping today's superscript and subscript folds changes 35 characters back and passes the module's tests.

• utr30.nrm: the binary data behind ICUFoldingFilter.NORMALIZER (ICUFoldingFilter.java:69-77). The build compiles it with ICU's gennorm2 from eight text files in lucene/analysis/icu/src/data/utr30/: ICU's standard nfc.txt, nfkc.txt, nfkc_cf.txt, and Lucene's UTR#30 tables BasicFoldings.txt, DiacriticFolding.txt, DingbatFolding.txt, HanRadicalFolding.txt, NativeDigitFolding.txt.
• Later file wins: ICU User Guide, "Mappings in one text file can override mappings in previous files of the same gennorm2 invocation." ICU's nfkc.txt:14: "Use this file as the second gennorm2 input file after nfc.txt."
• The conflict: DiacriticFolding.txt:41 maps ´ to nothing (00B4>, under "## Remove diacritics", rule [:Diacritic:] >); nfkc.txt:24 maps it to space + combining acute (00B4>0020 0301), and the combining acute is then deleted, which leaves the space.

1. build.gradle:92-94 sorts the inputs by file name ("gennorm generates file order-dependent output, so make it constant here.") and :122 passes them in that order. Upper case sorts first: Lucene's five tables, then nfc, nfkc, nfkc_cf.
2. So for every character both sides map, NFKC's mapping wins: on main the stored mapping is NFKC's for all 288 conflicts whose two mappings differ (4 are identical); 271 of the 292 fold differently than Lucene's mapping would, 21 the same.
3. From 2010 (LUCENE-1343) to the first gradle port (LUCENE-9767, February 2021) the build listed ICU's files first (nfkc nfkc_cf, with nfc added in front in 2012) and Lucene's tables last. Commit 010e3a1ba96 (LUCENE-9900, PR #61, 2021-04-02) replaced the list by the sort and committed the regenerated utr30.nrm; it changed no source table.
4. The regeneration checksums store one hash per file and no order (the same commit created them), and no test asserts any of the changed characters, so nothing noticed.

and ^ are in the same rule but have no NFKC mapping, so they are still deleted: ab` -> ab, while a´b -> a b`.

The table compares two orders on today's data. The last column folds the same 271 code points with the real 8.11.4 utr30.nrm (ICU4J 62.1): main matches it on 36, the order-only fix on 117 (it restores the 8.x order, not the 8.x data), the recommended fix on 71.

• Who is exposed. ICUFoldingFilter, ICUFoldingFilterFactory (icuFolding) or ICUFoldingFilter.NORMALIZER with the default data, in the optional analysis-icu module, on every release from 9.0.0 to 10.6.0 and main. Nothing to configure beyond choosing the filter; anything that wraps the filter gets the same terms.
• Which text. A changed character must stay inside a token: keyword or whitespace tokenization (keyword-normalized fields; the factory's own javadoc example uses WhitespaceTokenizerFactory, ICUFoldingFilterFactory.java:29-35) of an acute typed as an apostrophe (O´Brien, don´t) or another spacing accent between letters; the spacing katakana marks ゛ ゜ even with StandardTokenizer or ICUTokenizer (カ゛ -> カ ); modifier letters (ʰ, ᵉ) with any tokenizer that keeps them in the token (StandardTokenizer and ICUTokenizer do). The default tokenizers split at the Latin spacing accents (O´Brien -> o, brien on main and 8.11.4).
• What happens. The term differs from the accent-free spelling and from 8.x: a keyword field holding o brien does not match obrien, and an index folded with 8.x disagrees with 9.x query folding on these characters. The same analyzer at index and query time hides most of it.
• Scale. Deterministic, any index size, no error; each changed code point maps to at most two UTF-16 chars.
• How likely. Rare: no report was found in the searched sources since 9.0.0 (2021).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<a[]b> but was:<a[ ]b>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.icu.TestICUFoldingSpacingAcute.testSpacingAcuteIsRemoved(TestICUFoldingSpacingAcute.java:30)

Intent is not proven: the points below make an unintended precedence inversion likely, and the committers decide which folding semantics they want.

• Lucene's table says delete. DiacriticFolding.txt:35-43: "## Remove diacritics", "# Rule: [:Diacritic:] >", 005E>, 0060>, 00A8>, 00AF>, 00B4>, 00B7..00B8>, 02B0..034E>; the javadoc lists "Diacritic removal" (ICUFoldingFilter.java:33). rmuir on LUCENE-1343 (2010): CARET "folds to CIRCUMFLEX ACCENT, which should be deleted as its [:Diacritic:]".
• The tables were designed to sit on top of NFKC. Headings such as "## Spacing Accents folding (done by kd)" (BasicFoldings.txt:86) and a space rule that leaves out whatever NFKC_CF changes (:83); on LUCENE-3747 rmuir called the "additions to kd" sections "the mappings in utr30 minus what nfkc does".
• The order was explicit for 11 years. The ant build (LUCENE-1343, 2010; nfc.txt added in front in 2012) and the first gradle port (LUCENE-9767, 2021-02-14) used nfc nfkc nfkc_cf then Lucene's tables.
• The change was treated as a pure build change. 010e3a1ba96 touched only build files, two new checksum files and the binary (58,720 -> 59,200 bytes), no source table. In PR #61 dweiss asked: "gennorm seems to be order-sensitive - I changed the order to sorted file names ... Is this order somehow important? I can revert to what it was before but this way it's simpler." rmuir replied after the merge: "it is my understanding order doesnt matter ... this may cause a data file change even though semanticw are unchanged? just my guess", about chained mappings, not two files mapping one character.
• Silent. No CHANGES or MIGRATE entry for 9.0.0; later ICU upgrades (#447, #13239, #14386, #15417) regenerated under the sorted order without mentioning it.

The strongest case for "intended": the javadoc also lists "Spacing Accents folding", "Subscript folding" and "Superscript folding" and says text "is normalized to NFKC" (ICUFoldingFilter.java:47-49, :57); UTR#30 defines Spacing Accents folding as [KD] (space + mark) for 00AF, 00B4, 00B8, 02D8..02DD, ..., 309B..309C and Superscript folding as [KD] plus modifier letters, so today's output is what those foldings plus accent removal give, and 8.x's Mᵐᵉ -> m contradicted "Superscript folding". The order was discussed in review, and the file was regenerated twice since under the sorted order. Two more points weigh for keeping today's output: the text tables are not the public API and the javadoc specifies no precedence between files (at runtime the shipped utr30.nrm is the contract), and five years of stable 9.x-10.x output (33 release tags) plus the reindex cost make keeping it a legitimate compatibility choice.

What answers it: the PR merged with no answer on the order (its three reviews are comments with empty bodies); the only reply came four hours after the merge and rested on a guess that semantics were unchanged ("just my guess"), while the ICU page linked in that comment says the later file overrides; no comparison of the output is recorded in the PR or the later regenerations; nothing announced a change; the sorted order leaves Lucene's tables unable to override any mapping that NFKC also defines, which defeats their ability to tailor those; and one rule now gives two results (` ^ deleted, ´ ¨ ¯ ¸ -> space). The adjudication, the independent reproducer and the devil's advocate rejected the intended reading; a cross-model reviewer called the contract "genuinely ambiguous" but said a maintainer "cannot convincingly call the 2021 precedence change documented". It survives as a question of fix direction, which the recommended fix answers by keeping the superscript and subscript folds.

Both order fixes were made with the build's own regenerateUtr30DataFiles task (it downloads and compiles the matching ICU4C) in a scratch worktree of a9779ef3c193 and run through gradle; options B and C were not run. Every option changes indexed terms for some characters, so it needs a CHANGES/MIGRATE note suggesting a reindex of fields that use ICUFoldingFilter.

Option A's build.gradle change (below), plus one rule line in DiacriticFolding.txt. With Lucene's tables last again, a character they do not map keeps its NFKC mapping, so taking superscript and subscript letters out of the deletion rule leaves them to NFKC, which folds them to their compatibility text (usually the base letter), as UTR#30's "Superscript folding" and "Subscript folding" (both in the javadoc, ICUFoldingFilter.java:48-49) describe. The regeneration rewrote that rule's generated range lines (+8/-12; +9/-13 with the rule line), the checksum file and utr30.nrm (61,776 -> 61,408 bytes).

New output against today: exactly 35 code points change, all back to 8.x (deleted): the 31 that leave a space (´ ¨ ¯ ¸ ˘ ˙ ˚ ˛ ˜ ˝, the Greek spacing accents, ゛ ゜, the fullwidth macron, U+2017, U+037A), U+FE4D..FE4F (_ today) and U+0345 (ι today). The 236 modifier letters keep today's folds (Mᵐᵉ -> mme). Against the real 8.11.4 file it matches on 71 of the 271 (these 35, and U+1D9B..1DBE, which 8.11.4 already folds like main); option A on 117.

// lucene/analysis/icu/build.gradle (condensed): an explicit order instead of the name sort
def inputOrder = ["nfc.txt", "nfkc.txt", "nfkc_cf.txt", "BasicFoldings.txt",
    "DiacriticFolding.txt", "DingbatFolding.txt", "HanRadicalFolding.txt", "NativeDigitFolding.txt"]
def inputFiles = inputOrder.collect { name -> new File(utr30DataDir, name) }
inputs.property "gennorm2InputOrder", inputOrder.join(" ")

# lucene/analysis/icu/src/data/utr30/DiacriticFolding.txt, then regenerateUtr30DataFiles
 ## Remove diacritics
-# Rule: [:Diacritic:] >
+# Rule: [[:Diacritic:]-[:Decomposition_Type=Super:]-[:Decomposition_Type=Sub:]] >

Trade-offs

• It edits a generated rule, so it fixes the regression and also settles the superscript direction; reviewers may want the two in separate commits.
• It is a policy, not only an order repair: it exempts every character whose decomposition type is Super or Sub (some fold to two letters or to punctuation, such as U+107B7 -> ||), and later Unicode versions can add to that set.
• U+FE4D..FE4F go back to deleted: Lucene's Underline line FE4D..FE4F>005E (BasicFoldings.txt:95, copied from UTR#30) maps them to ^, which is then deleted; changing it to 005F would keep _ (not run). U+037A stays deleted, as in 8.x.
• Indexed terms change for the 35 characters; no CHANGES entry is drafted.

What the checks validated

• regenerateUtr30DataFiles accepted the rule (ICU UnicodeSet syntax) and wrote the new lines, checksums and file; the other data files are unchanged.
• Gradle: all of :lucene:analysis:icu:test (18 classes, 165 tests including the new test and the finder's, 10 skipped, 0 failures), regenerateUtr30DataFilesChecksumCheck, and :lucene:analysis:icu:check -x test (google-java-format check, ecjLint, forbiddenApis) pass.
• With the new utr30.nrm first on the classpath of the stock main jars: the new test 4/4 with assertions on and off, the finder's 3/3, the independent reproducer's 9/9.
• Exhaustive diffs over all code points: 35 against the shipped file, 236 (the modifier letters) against option A.

build.gradle +20/-3: an explicit list nfc.txt, nfkc.txt, nfkc_cf.txt, BasicFoldings.txt, DiacriticFolding.txt, DingbatFolding.txt, HanRadicalFolding.txt, NativeDigitFolding.txt; the order as a task input property, so it enters the regeneration checksums and a future change shows up as a checksum diff; and a doFirst check that fails a regeneration run if a *.txt file in the data directory is not in the list (it runs only when the task executes: a file missing from the list is not a task input, so adding one does not by itself trigger the task). Regenerated: checksum file and utr30.nrm (61,776 -> 60,608 bytes).

New output against today: exactly the 271 code points change, all to their result under the pre-2021 order (deleted); nothing else changes. That is the 8.x order, not the 8.x data: against the real 8.11.4 file it matches on 117 of the 271 and deletes 154 that 8.11.4 does not delete (36 that 8.11.4 folds like main, 118 newer than its Unicode data).

  // gennorm2 output depends on the order of its input files: when two files map the same code
  // point, the later file wins. The Unicode data comes first, in the order its headers ask for
  // (nfc.txt, nfkc.txt, nfkc_cf.txt), then Lucene's UTR#30 tables, so that they can override it.
  def inputOrder = [
    "nfc.txt",
    "nfkc.txt",
    "nfkc_cf.txt",
    "BasicFoldings.txt",
    "DiacriticFolding.txt",
    "DingbatFolding.txt",
    "HanRadicalFolding.txt",
    "NativeDigitFolding.txt"
  ]
  def inputFiles = inputOrder.collect { name -> new File(utr30DataDir, name) }
  ...
  inputs.property "gennorm2InputOrder", inputOrder.join(" ")
  ...
  doFirst {
    def unlisted = fileTree(dir: utr30DataDir, include: "*.txt").files.collect { it.name } - inputOrder
    if (!unlisted.isEmpty()) {
      throw new GradleException("Add these utr30 data files to the gennorm2 input order: " + unlisted)
    }

Trade-offs

Smallest change and the 8.x order, but it also deletes what today's order arguably improved: 236 modifier letters (2ᵉ -> 2, Mᵐᵉ -> m again; 154 of them were not deleted by 8.11.4 either), U+FE4D..FE4F and U+0345. Indexes built with 9.x-10.x see 271 code points change.

What the checks validated

• The regenerated utr30.nrm is byte-identical to the finder's independent rebuild in the pre-2021 order with an older ICU4C (77.1); the data files are unchanged.
• Gradle: the new test and the finder's, 7 run, 5 failed on stock and 0 failed with the fix; all of :lucene:analysis:icu:test (165 tests, 10 skipped, 0 failures) and the checksum check pass.
• With the new utr30.nrm first on the classpath of the stock main jars: new test 4/4 (assertions on and off), finder's 3/3, independent reproducer's 9/9; the release probe prints the 8.11.4 outputs for its probe characters.

Keep NFKC last, sorted or explicit, remove Lucene's mappings for the 292 code points that NFKC overrides, and decide the spacing-accent, U+0345 and U+037A cases explicitly.

Trade-offs

No indexed term changes, which respects five years of stable 9.x-10.x output; but the five current tables (all sorted before nfkc_cf.txt) cannot override any NFKC mapping, so the 31 space-in-term cases would need a new override file that sorts after nfkc_cf.txt, or an explicit order that ends with a Lucene file (ICU lets a later file override an earlier one); and "Remove diacritics" stops meaning what it says for those characters.

Compute the winning mapping of every code point over all eight files in the declared order (the later file wins, also among Lucene's own tables) and assert that ICUFoldingFilter.NORMALIZER.getRawDecomposition(cp) equals it, as the hunt's translation validation did for the sorted order (0 mismatches) when it found the 292. Comparing each table line on its own does not work: BasicFoldings.txt:109-112 are overridden by DiacriticFolding.txt's generated deletion in either order, and 37 DingbatFolding digits by NativeDigitFolding.

Trade-offs

With the order of A or A2 declared, it should fail on today's file and pass on the fixed one (not run); under any option it catches the next silent order change. It needs the table files, or a list derived from them, at test time.

• Method (semi-formal): bounded-exhaustive translation validation of utr30.nrm against its source tables, in the hunt's unit on the UTR#30 data. An independent model of gennorm2's precedence (inputs in the build's sorted order, a later file overrides an earlier one) matched gennorm2's own combined output and the shipped utr30.nrm on all 1,114,112 code points, 0 mismatches (the raw comparison with the shipped file skips the algorithmic Hangul syllables), and showed that 292 code points mapped by Lucene's tables are overridden by the nfc/nfkc/nfkc_cf data.
• Attribution. The finder rebuilt utr30.nrm with gennorm2 in the sorted order (byte-identical to the shipped file) and in the pre-2021 order, and compared the two over all code points: 271 fold differently, all deleted under the old order. The same comparison of the files on either side of 010e3a1ba96 gives 118.
• Then a public-API JUnit on the stock jars, and independent checkers (see "Independent checks").