IrishLowerCaseFilter leaves capital letters outside the BMP unchanged, so IrishAnalyzer misses their lower-case form

The javadoc of IrishLowerCaseFilter says it "Normalises token text to lower case"; it is the only lower-casing step of IrishAnalyzer, for indexing and in normalize. Its loop passes one UTF-16 char at a time to Character.toLowerCase (IrishLowerCaseFilter.java:55). Letters outside the Basic Multilingual Plane (BMP) are stored as two chars, a surrogate pair, and each half alone has no case, so the 282 such capitals (Adlam, Deseret, Garay, Medefaidrin, Old Hungarian, Osage, Vithkuqi, Warang Citi) stay upper-case while BMP letters in the same token are lower-cased. A document indexed with IrishAnalyzer that contains the Adlam text U+1E900 U+1E923 is found by that exact spelling but not by its lower-case form U+1E922 U+1E923 (0 hits instead of 1). Nothing throws, and an index built from such text passes CheckIndex. Irish text itself has no such letters, so only mixed-script content in an Irish field is affected. The code has been there since the class was added in 3.6.0 (2012); reading the code point instead of the char fixes it.

• UTF-16 char: Java stores text as 16-bit char values. Most letters, including every Irish one, fit in one char: they are in the Basic Multilingual Plane (BMP).
• Supplementary character: a code point above U+FFFF, stored as two chars, a surrogate pair (a high and a low surrogate). The Adlam capital alif U+1E900 is the pair U+D83A U+DD00. A surrogate alone is not a letter and has no case.
• Code point lower-casing: Character.toLowerCase(int) maps a whole code point (U+1E900 to U+1E922). The char overload cannot: the JDK documents that it "cannot handle supplementary characters".

1. IrishLowerCaseFilter.incrementToken first handles the Irish prefixes (n-eclipsis and t-prothesis: nAthair becomes n-athair), then lower-cases the rest of the term: i += Character.toChars(Character.toLowerCase(chArray[i]), chArray, i); (IrishLowerCaseFilter.java:54-56 on main).
2. chArray[i] is one char. For U+D83A, Character.toLowerCase returns U+D83A; toChars writes it back and returns 1; the low half U+DD00 goes the same way. The capital letter is unchanged.
3. The loop is otherwise written for code points: it advances by the number of chars toChars wrote, as GreekLowerCaseFilter does (GreekLowerCaseFilter.java:46, which reads Character.codePointAt(chArray, i, chLen)). LowerCaseFilter lower-cases each code point through CharacterUtils.toLowerCase (CharacterUtils.java:47-63).
4. IrishAnalyzer runs StandardTokenizer, which keeps words of every script, and this filter is its only lower-casing step (IrishAnalyzer.java:126-135); IrishAnalyzer.normalize, which the classic query parser calls for range, wildcard, regexp, prefix and fuzzy terms (QueryParserBase.java:550-729, read; not run end to end), runs the same filter (:138-142).

Every code point whose lower case differs under Character.toLowerCase(int): 1,178 in the BMP, all handled correctly, and 282 outside it, all left unchanged: Adlam 34, Deseret 40, Garay 22, Medefaidrin 32, Old Hungarian 51, Osage 36, Vithkuqi 35, Warang Citi 32 (JDK 25 tables). Emoji and CJK ideographs have no case and are not affected. No lower-case mapping crosses between the BMP and the supplementary planes, so the term length never changes.

• Who is exposed. The stock IrishAnalyzer with any constructor (indexing and normalize), any chain with IrishLowerCaseFilter, and the irishLowercase factory (IrishLowerCaseFilterFactory.java:40; its normalize is the same filter, :60-64).
• What it takes. An upper-case letter of Adlam (a living script for Fulani), Osage, Deseret, Old Hungarian, Warang Citi, Medefaidrin, Vithkuqi or Garay in an Irish-analysed field. Irish orthography has none, so it takes mixed-script content: a quoted name or word, or a historic-script passage.
• What changes for the user. The capitalised and the lower-case spellings become two different terms, so a case-insensitive search misses in both directions (0 hits instead of 1 in the repro). In the intended-behaviour check's probe, "Tá X agus Y anseo" analysed to [tá, U+1E900 U+1E923, U+104B0, anseo] with capital Adlam and Osage words and [tá, U+1E922 U+1E923, U+104D8, anseo] with lower-case ones. Exact-case matches still work.
• What does not happen. No exception, no change in term length, valid UTF-16 terms and no index damage: an index built by the stock IrishAnalyzer from such text passes CheckIndex in two segments and after forceMerge(1), and every term decodes as strict UTF-8 (assertions on and off). BMP letters in the same token are lower-cased.
• Scale. Deterministic, with assertions on or off, at any index size: every occurrence of such a capital is indexed (and queried) unchanged.
• JDK dependence. The affected set follows the JDK's Unicode tables: 282 code points with JDK 25. Deseret letters exist in every JDK Lucene supports; scripts added in later Unicode versions (Osage and Adlam in 9.0, Garay in 16.0) need a newer JDK. Only JDK 25 was run.
• How likely. Rare: no public report of it was found (searches of 2026-10-27 and 2026-10-28), and the trigger needs mixed-script text in an Irish field.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<U+104[28]> but was:<U+104[00]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.ga.TestIrishLowerCaseSupplementary.testSupplementaryUpperCaseIsLowerCased(TestIrishLowerCaseSupplementary.java:29)

• Contract. "Normalises token text to lower case, handling t-prothesis and n-eclipsis (i.e., that 'nAthair' should become 'n-athair')" (IrishLowerCaseFilter.java:24-27, word for word as in 3.6.0). No script or BMP restriction; LowerCaseFilter's javadoc is nearly the same ("Normalizes token text to lower case.", LowerCaseFilter.java:22) and it lower-cases these letters.
• Policy before the class existed. lucene/CHANGES.txt:17927-17928 (3.1.0): "LUCENE-2069: Added Unicode 4 support to LowerCaseFilter. Due to the switch to Java 5, supplementary characters are now lowercased correctly." TurkishLowerCaseFilter (LUCENE-2102, 2009) and GreekLowerCaseFilter (LUCENE-2463, 2010) map code points. In 3.6.0, the release that added this filter, LowerCaseFilter already mapped by code point: the 3.6.0 jar lower-cases U+10400, U+104B0 and U+1E900 (executed on JDK 25). Of these, only Deseret U+10400 (Unicode 3.1) was in the JDKs of 2012; Osage and Adlam entered Unicode in 9.0 (2016).
• Same defect class treated as a bug. lucene/CHANGES.txt:1198-1200 (10.5.0): "GITHUB#15899, GITHUB#15900: Improve TruncateTokenFilter to truncate on codepoints not chars and no longer produce half surrogates". On #15899 uschindler wrote "We modified all other similar filters in the past (like ngrams), they all work correctly". That is precedent for treating char-wise handling as a bug, not proof of a complete audit: CapitalizationFilter (LFV-533) is still char-wise too.
• Nothing presents it as a choice. The class was new in 3.6.0 with no legacy switch; the upstream tests assert expected output only for BMP input (TestIrishAnalyzer.testRandomStrings feeds random text but checks only token-stream consistency, not case); the LUCENE-3883 thread (#4956, 25 comments) never mentions supplementary characters; the loop is written in the code-point idiom (i += Character.toChars(...)) but reads a char, and the JDK documents that the char overload "cannot handle supplementary characters".

The strongest case for "intended" (the intended-behaviour check and the cross-model reviewer): the constructor says it "normalises Irish token text" (IrishLowerCaseFilter.java:31), Irish is written in Latin script with no supplementary letters, so Adlam or Osage text is out of scope; and the factory says "this will 'mostly work', except for special cases, just like most other filters" (IrishLowerCaseFilterFactory.java:60).

Why it fails: the class javadoc promises lower case without restriction; IrishAnalyzer tokenises every script and has no other lower-casing step; the filter does lower-case non-Irish BMP scripts (Greek, Cyrillic, Armenian, executed by the intended-behaviour check), so the only gap is the supplementary planes, which comes from the char API, not from a choice of domain; TurkishLowerCaseFilter uses the same wording ("Normalizes Turkish token text to lower case", TurkishLowerCaseFilter.java:25) and maps code points; and the factory comment came with 44695f0f3f5 ("LUCENE-3883: add multitermaware to the irish case filter") and is about multi-term normalisation, not supplementary characters. Every verifier rejected the intended-behaviour argument, and the cross-model reviewer rated it weak twice. A later cross-model fact check of this doc rated it moderate: domain scope, BMP-only tests, the reindex cost and the factory comment give it some weight, although no evidence shows that the BMP limit was considered and chosen. What survives is the narrow reach and the reindex cost: reasons for low priority and a CHANGES note, not for keeping the behaviour.

Two equivalent fixes were built and validated in a scratch worktree at a9779ef3c193; A is the smallest diff and the one every checker ran.

The finder's one-line fix, wrapped by google-java-format. The loop already advances by the number of chars written, so nothing else changes.

      for (int i = idx; i < chLen; ) {
-        i += Character.toChars(Character.toLowerCase(chArray[i]), chArray, i);
+        i +=
+            Character.toChars(
+                Character.toLowerCase(Character.codePointAt(chArray, i, chLen)), chArray, i);
      }

Trade-offs

• Changes output only for the 282 supplementary capitals: every BMP code point already matched, and no lower-case mapping crosses the BMP boundary, so the term length never changes.
• Indexed terms change for text with these letters: indexes built with IrishAnalyzer need reindexing for exact consistency. For TruncateTokenFilter the project added factory parameters for the old behaviour (#15900); an option seems unnecessary here, but that is the maintainers' call.

What the checks validated

• Stock main: the paste-ready test fails 3 of 4, assertions on and off; the 10 upstream Irish tests pass.
• With fix A, org.apache.lucene.analysis.ga.*: 4 classes, 14 tests, 0 failures, assertions on and off.
• With fix A, the whole :lucene:analysis:common:test task: 340 classes, 1,913 tests, 0 failures, 18 skipped (skips not examined).
• ecjLintMain, ecjLintTest and checkGoogleJavaFormat of the module pass.
• Earlier checks: the finder's test (4 of 4), the independent reproduction's (7 of 7, and 17 of 17 with the upstream Irish tests), the cross-model reviewer's (5 of 5), the intended-behaviour check's (8 of 8), and the history probe on 3.6.0 and main all pass with the same change.

Replace the loop with the shared helper and import org.apache.lucene.analysis.CharacterUtils (+2/-3 lines). It maps each code point with Character.toLowerCase(int), with an ASCII fast path, so the output is the same as A.

-      for (int i = idx; i < chLen; ) {
-        i += Character.toChars(Character.toLowerCase(chArray[i]), chArray, i);
-      }
+      CharacterUtils.toLowerCase(chArray, idx, chLen);

Trade-offs

Reuses one documented code path instead of a second hand-written loop; a slightly larger diff than A, and only this doc's runs checked it.

What the checks validated

Validated the same way: ga.* 14 tests, 0 failures, assertions on and off; the whole module 340 classes, 1,913 tests, 0 failures, 18 skipped; lint and format pass; the standalone runner passes on main with -ea and -da.

A javadoc note that only BMP letters are lower-cased.

Trade-offs

Cheap, but contradicts LowerCaseFilter, the Turkish and Greek filters and the project's own past fixes, for no benefit.

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier). The finder wrote a reference for each language lower-case filter from its javadoc and listed mappings: lower case means Character.toLowerCase(int) per code point, as LowerCaseFilter does, plus each filter's documented special rules. It compared the stock Greek, Irish and Turkish filters with the reference:

• on every code point 0..0x10FFFF alone (1,112,064 checks per filter): Greek 0 and Turkish 0 mismatches, Irish 282, all supplementary upper-case letters;
• on 12 Latin, Greek and combining blocks in alone, initial, medial, final, after-n, after-t and after-I positions (15,984 checks per filter): Irish 0 mismatches, so the n-eclipsis and t-prothesis rules and every BMP letter behave as documented.

A public-API JUnit on the stock main jar then showed the effect through the filter and through an IrishAnalyzer index and search (found 2026-10-27). IrishLowerCaseFilter was one of the rule-table classes in the hunt unit's scope; the bug came from its every-code-point sweep.