CapitalizationFilter never changes the case of letters outside the BMP, so Adlam, Osage and Deseret words are neither capitalized nor lower-cased

The javadoc of CapitalizationFilter says "It will make the first letter capital and the rest lower case" (CapitalizationFilter.java:26-28). Its processWord changes case with Character.toUpperCase(char) and Character.toLowerCase(char), one UTF-16 char at a time (:152, :159, :190, :193). Letters outside the Basic Multilingual Plane (BMP) are stored as two chars, a surrogate pair, and each half alone has no case, so the 564 cased code points of Adlam, Deseret, Garay, Medefaidrin, Old Hungarian, Osage, Vithkuqi and Warang Citi are never capitalized and never lower-cased, while BMP letters in the same token are. In a field analysed with KeywordTokenizer + CapitalizationFilter, PARIS and paris become one term, Paris, but the Adlam letters alif + daali in capitals and in small letters stay two terms, neither capitalized, and a query for one spelling finds 1 of the 2 documents. No run threw, and the output is valid UTF-16 of the same length. The char-wise lines come from the Solr original (2007-2008), were moved into Lucene unchanged in 2010 and are in every release since 4.0.0-ALPHA; mapping code points, as LowerCaseFilter and UpperCaseFilter do, fixes it.

• UTF-16 char: Java stores text as 16-bit char values. Most letters fit in one char: they are in the Basic Multilingual Plane (BMP).
• Supplementary character: a code point above U+FFFF, stored as two chars, a surrogate pair. The Adlam small alif U+1E922 is U+D83A U+DD22. A surrogate alone is not a letter and has no case.
• Code-point case mapping: Character.toUpperCase(int) and toLowerCase(int) map a whole code point (U+1E922 to U+1E900). The char overloads cannot: the JDK documents that they "cannot handle supplementary characters".

1. incrementToken splits the term into words at chars <= ' ' and . and calls processWord on each word, editing the term buffer in place (CapitalizationFilter.java:102-143).
2. processWord (:145-196) maps case with the char overloads at four places: :152, every char of a later word when onlyFirstWord is true (the default); :159, the first char of a keep word with forceFirstLetter; :190, buffer[offset] = Character.toUpperCase(buffer[offset]);, the first letter; :193, every other char of the word, lower-cased.
3. For a surrogate both overloads return the char unchanged (all 2,048 surrogate units checked), so a supplementary letter keeps its case. BMP letters in the same word are mapped.
4. The code-point helpers exist next door: CharacterUtils.toLowerCase and toUpperCase (CharacterUtils.java:54-70, :105-113), used by LowerCaseFilter (:38) and UpperCaseFilter (:48). This filter's own case-insensitive keep set compares per code point too (CharArrayMap.java:231-290).

In an index (KeywordTokenizer + CapitalizationFilter, four documents PARIS, paris, U+1E900 U+1E901, U+1E922 U+1E923): expected terms [Paris, U+1E900 U+1E923], actual [Paris, U+1E900 U+1E901, U+1E922 U+1E923]; the query paris finds 2 documents, the query U+1E922 U+1E923 finds 1 instead of 2.

The supplementary code points with a case mapping (JDK 25 tables): 564 in eight scripts, Adlam 68, Deseret 80, Garay 44, Medefaidrin 64, Old Hungarian 102, Osage 72, Vithkuqi 70, Warang Citi 64. Emoji and CJK ideographs have no case and are not affected.

• Who is exposed. Chains built with the constructors, the capitalization factory in a CustomAnalyzer (reached with the stock standard, whitespace and keyword tokenizers: 9 of 9 cases unmapped, while lowercase in the same pipeline lower-cases the capitals) or a schema field type like the factory's javadoc example. Every path that changes case is affected (the default settings, onlyFirstWord=false and a keep word with forceFirstLetter were run). The filter is opt-in and in no stock analyzer.
• What it takes. Text in Adlam (a living script for Fulani), Osage, or historic or rarely used Deseret, Old Hungarian, Warang Citi, Medefaidrin, Vithkuqi or Garay.
• What changes for the user. Usually, case variants of such words stay separate terms (3 instead of 2 in the repro), separate facet values, and a query in one case misses the other (1 hit instead of 2). The capitalized display form the filter exists for is never produced for them. In one configuration the opposite also happens: with a case-sensitive keep list and forceFirstLetter, the stock filter turns [a, U+10400] into the kept word [A, U+10400] instead of [A, U+10428], merging case variants that the fixed filter keeps apart (35 stock terms in that configuration of the security probe, none in the other four).
• What does not happen. In a bounded probe over 7,380 tokens and 5 configurations, assertions on and off: no exception, no change in term length, valid UTF-16; the filter keeps no state across tokens or documents.
• Scale. Deterministic, with assertions on or off. The result depends only on the token text and the settings, so index size should not matter; only small indexes were run.
• JDK dependence. Deseret (Unicode 3.1) shows it on every JDK Lucene supported; Adlam and Osage need Unicode 9.0 tables, which first came with JDK 11 (JDK 9 and 10 have Unicode 8.0; JDK 11 has Unicode 10.0, JEP 327). Only JDK 25 was run for the repro; a JDK 11 sanity run maps Adlam and Osage too.
• How likely. Rare: the documented searches found no public report since Solr 1.3.0 (2008), and it needs those scripts in a field with this filter.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[1e900, 1e923]> but was:<[1e922, 1e923]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.miscellaneous.TestCapitalizationFilterSupplementary.testCapitalizesFirstLetterOutsideBmp(TestCapitalizationFilterSupplementary.java:37)

• Contract. "A filter to apply normal capitalization rules to Tokens. It will make the first letter capital and the rest lower case." (CapitalizationFilter.java:26-28). No script, BMP or UTF-16-unit limit in the class or the factory javadoc. Where Lucene means UTF-16 units it says so: LengthFilter.java:26, "Length is calculated as the number of UTF-16 code units".
• The API used cannot do it. The JDK javadoc of Character.toUpperCase(char) and toLowerCase(char): "This method cannot handle supplementary characters. To support all Unicode characters, including supplementary characters, use the toUpperCase(int) method."
• Siblings and their tests. TestAnalyzers.testLowerCaseFilter and testUpperCaseFilter ("handles entire unicode range correctly") require Deseret to be mapped (TestAnalyzers.java:135-169). The filter's own keepIgnoreCase set folds case per code point.
• Policy before the class moved in. CHANGES.txt:17927-17928 (3.1.0): "LUCENE-2069: Added Unicode 4 support to LowerCaseFilter. Due to the switch to Java 5, supplementary characters are now lowercased correctly." That commit (2009) is an ancestor of the 2010 move of this class into Lucene, which copied the char loop verbatim.
• Same defect class treated as a bug. CHANGES.txt:1198-1200 (10.5.0 Bug Fixes): "GITHUB#15899, GITHUB#15900: Improve TruncateTokenFilter to truncate on codepoints not chars and no longer produce half surrogates"; on #15899 uschindler wrote "We modified all other similar filters in the past (like ngrams), they all work correctly". CHANGES.txt:8852: "LUCENE-6914: Fixed DecimalDigitFilter in case of supplementary code points."
• Nothing presents it as a choice. No test, javadoc, CHANGES or MIGRATE entry, commit message or JIRA discussion (SOLR-248, SOLR-468 and LUCENE-2413, read in full) marks the char loop as BMP-only. TestCapitalizationFilter.testRandomString checks token-stream consistency, not case.

The strongest case for "intended" (the intended-behaviour check and the cross-model reviewer): a legacy, display-oriented char-buffer filter from Solr 1.3 ("useful to build nice looking facet parameters"), never updated, whose fix changes indexed output; on #15900 uschindler argued at one point that half surrogates from char-based truncation are "expected", "exactly the same like if you use String#substring(n,m)" (earlier in the same thread, uschindler had called that filter "somehow broken in general"), and both LUCENE-2069 and #15900 kept a legacy path.

Why it fails: nothing documents a BMP limit; the JDK documents the overloads used as unable to do the job; the sibling case filters and this filter's own keep set work per code point; on #15900 rmuir wrote "Nobody expects that behavior and doing this based on codepoints is a better api" (before the "expected" remark), and that fix shipped under Bug Fixes. Every verifier who assessed the argument rated it weak (the intended-behaviour check: no intended argument survives). What survives is the narrow reach and the reindex cost: reasons for low priority and a CHANGES note, perhaps a legacy option, not for keeping the behaviour.

One fix, the finder's patch, was rebuilt and validated in a scratch worktree at a9779ef3c193; it applies unchanged.

Replace the two char loops with CharacterUtils.toLowerCase(buffer, from, to) and the two first-letter statements with a small helper that reads the first code point, upper-cases it with Character.toUpperCase(int) and writes it back with Character.toChars. All three paths of processWord are covered.

--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/CapitalizationFilter.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/CapitalizationFilter.java
@@ -19,6 +19,7 @@
 import java.io.IOException;
 import java.util.Collection;
 import org.apache.lucene.analysis.CharArraySet;
+import org.apache.lucene.analysis.CharacterUtils;
 import org.apache.lucene.analysis.TokenFilter;
 import org.apache.lucene.analysis.TokenStream;
 import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
@@ -148,15 +149,13 @@
     }
 
     if (onlyFirstWord && wordCount > 0) {
-      for (int i = 0; i < length; i++) {
-        buffer[offset + i] = Character.toLowerCase(buffer[offset + i]);
-      }
+      CharacterUtils.toLowerCase(buffer, offset, offset + length);
       return;
     }
 
     if (keep != null && keep.contains(buffer, offset, length)) {
       if (wordCount == 0 && forceFirstLetter) {
-        buffer[offset] = Character.toUpperCase(buffer[offset]);
+        upperCaseFirstCodePoint(buffer, offset, length);
       }
       return;
     }
@@ -187,11 +186,14 @@
     /*char[] chars = w.toCharArray();
     StringBuilder word = new StringBuilder( w.length() );
     word.append( Character.toUpperCase( chars[0] ) );*/
-    buffer[offset] = Character.toUpperCase(buffer[offset]);
-
-    for (int i = 1; i < length; i++) {
-      buffer[offset + i] = Character.toLowerCase(buffer[offset + i]);
-    }
+    int firstLen = upperCaseFirstCodePoint(buffer, offset, length);
+    CharacterUtils.toLowerCase(buffer, offset + firstLen, offset + length);
     // return word.toString();
   }
+
+  /** Upper-cases the first code point of the word; returns its length in chars. */
+  private static int upperCaseFirstCodePoint(char[] buffer, int offset, int length) {
+    int cp = Character.codePointAt(buffer, offset, offset + length);
+    return Character.toChars(Character.toUpperCase(cp), buffer, offset);
+  }
 }

Trade-offs

• Output changes only for supplementary letters; BMP text gives the same chars as before. Indexes built with this filter need reindexing for exact consistency, so a CHANGES note should say so.
• The in-place edit relies on case mapping keeping each code point's UTF-16 length; that holds for all 1,112,064 non-surrogate code points with JDK 25, and CharacterUtils.toLowerCase (used by LowerCaseFilter) relies on the same.
• With a case-sensitive keep list, the stock filter maps [a, U+10400] onto a kept word [A, U+10400]; the fixed filter keeps them apart ([A, U+10428]). Worth one sentence in the PR.

What the checks validated

• Stock main: the paste-ready test fails 3 of 4, assertions on and off; the 29 upstream TestCapitalizationFilter* tests pass.
• With fix A, TestCapitalizationFilter*: 3 classes, 33 tests, 0 failures, assertions on and off.
• With fix A, the whole :lucene:analysis:common:test task: 340 classes, 1,913 tests, 0 failures, 18 skipped (skips not examined).
• ecjLintMain, ecjLintTest, checkGoogleJavaFormat, forbiddenApisMain and forbiddenApisTest of the module pass.
• The standalone runner with the fixed class shadowed: 4 of 4 on main, -ea and -da; earlier, the finder's test (4 of 4) and the 29 upstream tests on ea57360912ed.
• Malformed UTF-16: all 137,257 strings of 0-6 chars over {a, B, space, ., U+D801, U+DCB0, U+DCD8} (128,166 with an unpaired surrogate) through the stock and the fixed filter, three configurations, -ea and -da: no exception, the same length, lone surrogates unchanged, and without a valid pair the fixed output equals the stock output.

Keep the char-wise path behind an option, as #15900 did for TruncateTokenFilter ("new factory parameters available to configure legacy prefix chars and new codepoint behaviour") or LUCENE-2069 did with a Version argument.

Trade-offs

More API surface for 564 code points; only worth it if maintainers want old terms to stay exactly as they are without reindexing.

A javadoc note that only BMP letters change case.

Trade-offs

Cheap, but contradicts LowerCaseFilter, UpperCaseFilter, the filter's own keep set and the project's past fixes.

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier). The finder wrote two references for CapitalizationFilter from its javadoc, one that maps case per char and one per code point, and compared the stock filter with them:

• with the per-char reference, on every token of up to 5 chars over {a, B, space, '.', U+0131} under 1,440 parameter combinations (7,874,496 checks): the mismatches found the sibling word-boundary bug (LFV-535);
• with the per-code-point reference, on every token of up to 4 code points over {a, B, space, U+10428, U+10400}, with the default parameters and with onlyFirstWord=false, restricted to checks (a token with one onlyFirstWord setting) where the per-char reference agrees, so the sibling bug is excluded: 836 of 1,414 checks differ, each at a supplementary letter; 880 of all 1,562 checks (781 tokens x 2 settings) still differ after the sibling's fix.

A public-API JUnit on the stock main jar then confirmed it (found 2026-10-26). The same reference was used to find and to check, so the exhaustive count is not independent evidence; the verifiers wrote their own tests.