RegExp CASE_INSENSITIVE matches dotted and dotless i one way only: /İ/ matches "i", but /i/ does not match "İ"

RegExp.CASE_INSENSITIVE (and the deprecated ASCII_CASE_INSENSITIVE, which takes the same path since 10.2.0) is not symmetric for two letters of the Turkish and Azerbaijani alphabets: the pattern İ (U+0130) matches the text i, but i does not match İ, and the pattern ı (U+0131) matches I, but I does not match ı. It is not transitive either: İ matches i and i matches I, but İ does not match I. CaseExpansion.expand adds Java's simple upper and lower case mapping of every pattern letter, and for exactly these two code points those mappings lie outside the simple case-folding closure that the ICU-generated data is meant to equal; nothing removes them. Neither model the javadoc names behaves this way: C+S simple case folding makes İ and ı match only themselves, and java.util.regex makes all four letters match each other. The cleanup in #14384, which removed the Turkish mappings on purpose, left these two one-way edges behind. Every release from 10.2.0 to 10.6.0 and main is affected; a validated fix lets the generator skip the out-of-closure mappings.

• i U+0069, I U+0049, İ U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE, ı U+0131 LATIN SMALL LETTER DOTLESS I.
• C+S simple case folding: the CaseFolding.txt mappings with status C (common) and S (simple). For the i letters: 0049; C; 0069 (line 71), 0049; T; 0131 (line 72), 0130; F; 0069 0307 and 0130; T; 0069 (lines 146-147); U+0131 has no row. "The default option is to exclude" T mappings (lines 52-53). So under C+S, İ and ı match only themselves (Unicode 16.0.0 files).
• Closure: ICU's UnicodeSet(c, c).closeOver(SIMPLE_CASE_INSENSITIVE): {I, i} for i and I, {İ} for İ, {ı} for ı (executed with ICU4J 78.3, whose Unicode 17 data CaseExpansion.java was generated from, :25-26; the CaseFolding.txt rows cited above are Unicode 16.0.0).

CaseExpansion.expand(c) (CaseExpansion.java:32-44, a generated file) accepts c, then Character.toUpperCase(c) if it differs (:36-39), then Character.toLowerCase(c) if it differs (:41-44), then a generated switch of extras (:46 onward), which has no case for any of the four letters. Its javadoc says "c itself and its scf mappings" (:31).

The generator (GenerateCaseExpansion.groovy:17-24) emits, per code point, only the closure members that are not c, its upper case or its lower case; the template (:71-85) writes the upper/lower adds, guarded only by != c, with no check against the closure. That equals the closure only if the upper and lower case are inside it, which fails for exactly two code points. From UnicodeData.txt, Java gives toLowerCase(U+0130) = i (line 305) and toUpperCase(U+0131) = I (line 306), while i and I map only to each other:

pattern \ text   i  I  İ  ı
i                1  1  0  0
I                1  1  0  0
İ                1  0  1  0     <- matches i, but i does not match İ
ı                0  1  0  1     <- matches I, but I does not match ı
java.util.regex (CASE_INSENSITIVE|UNICODE_CASE): all 16 pairs match

(main 645d4fe14a20, both flags; the 10.6.0 jar gives the same matrix.)

Every case-insensitive entry point calls expand: RegExp single characters (RegExp.java:682-687, via Automata.makeCaseInsensitiveChar), strings (:701-705, Automata.makeCaseInsensitiveString), character classes (:1338), CASE_INSENSITIVE_RANGE (:773), and the public Automata.makeCaseInsensitiveChar/String (Automata.java:121-127, :569-592, :700-709; public since 10.5.0, #15753). RegexpQuery on an index gives the asymmetric hit sets of the steps above on main and 10.6.0. With CASE_INSENSITIVE_RANGE, [İ-ı] matches i and I, but [h-i] does not match İ and [H-I] does not match ı.

An exhaustive sweep of 0 to 0x10FFFF through Automata.makeCaseInsensitiveChar finds 2,940 code points that accept more than one code point and exactly 2 one-way edges, U+0130 -> U+0069 and U+0131 -> U+0049.

• Trigger. An opt-in match flag plus text or a pattern containing İ or ı. No Lucene query parser sets the flags (in non-test code only util/automaton uses them); applications pass them to RegexpQuery(term, syntaxFlags, matchFlags) (main; in 10.x the 4-argument constructor with a determinize work limit), RegExp, or call Automata.makeCaseInsensitiveChar/String directly.
• The deprecated flag too. ASCII_CASE_INSENSITIVE has taken the same path since 10.2.0 (check(ASCII_CASE_INSENSITIVE | CASE_INSENSITIVE)), so applications that set it before 10.2.0 got Unicode case-insensitivity, and this asymmetry, on upgrade.
• Scale. Exactly 2 code points, but every word containing them; in Turkish and Azerbaijani these are everyday letters (for example İstanbul). For other languages it practically never occurs.
• Effect. Silent wrong match sets: a query with İ finds documents with i, but a query with i does not find documents with İ; the same for ı and I. No exception, no index damage.
• Downstream (read, not tested): Elasticsearch #153513 (merged 2026-07-11) routes its case_insensitive term, prefix and wildcard queries through Automata.makeCaseInsensitiveChar/String and its regexp query through RegExp.CASE_INSENSITIVE; both reach expand.
• API status. RegExp, Automata and RegexpQuery are marked @lucene.experimental (RegExp.java:352, Automata.java:44, RegexpQuery.java:40), so a behaviour change in a minor release is allowed; this weakens a compatibility argument, not the inconsistency.

/\u0130/ is run on the text "i" and /i/ on "\u0130", both with RegExp.CASE_INSENSITIVE, and the test asserts the two answers are equal. Deterministic, single thread; LuceneTestCase is only the base class, and the result does not depend on the random locale (Java's code-point case mapping is locale-independent). The source is ASCII-only, with \u escapes.

Regression test: this same class (or its one method added to TestRegExp); it passes with the recommended fix.

cp $REPRO/TestRegExpCaseInsensitiveDottedI.java $LUCENE/lucene/core/src/test/org/apache/lucene/util/automaton/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestRegExpCaseInsensitiveDottedI -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
   >     java.lang.AssertionError: CASE_INSENSITIVE /U+0130/ on "i" vs /i/ on "U+0130" expected:<true> but was:<false>
   >         at __randomizedtesting.SeedInfo.seed([DEADBEEF:B536C22A3EB52FDB]:0)
   >         at org.junit.Assert.assertEquals(Assert.java:120)
   >         at org.apache.lucene.util.automaton.TestRegExpCaseInsensitiveDottedI.testDottedCapitalIMatchesSymmetrically(TestRegExpCaseInsensitiveDottedI.java:25)

:lucene:core:test (FAILURE): 1 test, 1  failure

# standalone runner, seed DEADBEEF: main 645d4fe14a20, 10.2.0, 10.5.2 and 10.6.0 fail with -ea and -da, same message

• Documented model. RegExp.java:431-433: "We support common case folding in addition to simple case folding as defined by the common (C) and simple (S) mappings in .../CaseFolding.txt"; :430-431: "the attempt is to reach parity with java.util.regex.Pattern Pattern.CASE_INSENSITIVE and Pattern.UNICODE_CASE". Under C+S neither edge exists; under Pattern all 16 pairs match (executed). Either way one of today's two answers is wrong.
• API wording. Automata.java:121-124: "case-insensitive equivalents"; CaseExpansion.java:31: "c itself and its scf mappings", and i is not a simple case folding of İ.
• The author's design. #14384 (rmuir): "Remove turkish mappings, these were wrongly mixed in and only make things worse ... it should not be mixed in with normal mappings" (commit message) and "the CaseFolding class now just gives you UnicodeSet(ch).closeOver(UnicodeSet.SIMPLE_CASE_INSENSITIVE)" (PR comment, 2025-03-21). Earlier, as rmuir wrote on #14350, "dotless and dotted I are DIFFERENT, but it mixes all these characters up and conflates them", and in the #14192 review: "we want the simple \"L1\" 1-1 case folding, no expansion/contraction/normalization/turkish-azeri alt". The surviving edges are the Turkic pairing, next to the default {I, i}.
• Lucene's folding. UnicodeUtil.foldCase ("Returns the Unicode simple case folding", UnicodeUtil.java:690-693) maps U+0130 to U+0130 (SimpleCaseFolding.java:43-44) and U+0131 to itself; as neoremind noted in their #16371 review, U+0130 has "no Common/Simple mapping (only Full and Turkic ones)".
• No test pins it, and the parity test cannot see it. No U+0130/U+0131 in the automaton tests. TestRegExp case 14 (TestRegExp.java:263-282, asserted at :294-318) switches each letter's case with Character.toUpperCase/toLowerCase and requires both Pattern and RegExp to match the original text, but its palette (:171-173) has neither letter. Replayed for single code points over the 2,933 with a case-switched form, RegExp's half fails for exactly U+0130 and U+0131; Pattern's half fails only for U+1E9E (pattern ß on text ẞ, a JDK quirk). Under the recommended fix RegExp's half still fails for both letters: case 14 encodes java.util.regex parity, so it explains why upstream tests never caught this but is not a regression test for that fix.
• What this settles. These facts make the asymmetry a poor fit for every documented model and point to an incomplete #14384 cleanup; they do not settle which symmetric behaviour maintainers want.

The strongest case for "intended": the class-3 paragraph says "lowercase matching on these values is supported: 0x00DF, 0x0130, ..." (RegExp.java:464-466), which reads as "İ matches its lowercase"; the javadoc only promises "most Unicode characters" and "a best effort ... there may be differences" (:428, :440-441); and in OpenSearch PR #20928 msfroh drew exactly these one-way edges as the "Unicode ROOT locale" case conversions, naming Automata.makeCaseInsensitiveChar() as the rules for that mode.

Why it fails: the class-3 sentence was written in #14192, when İ matched all of i, I and ı, and #14384 left it unchanged; it names no direction and never says i must fail to match İ. One-way is a property of toLowerCase, not of a match relation, and "best effort" allows gaps against Pattern, not an answer that depends on which side holds the letter. The C+S paragraph, the "equivalents" wording and the author's statements all describe a symmetric relation. A cross-model refute review rated the argument strong but returned only a procedural "not real" (it could not run Java); the tie-break rejected it. Expect the fix-direction question, not doubt about the bug.

One fix was written and validated in a scratch worktree of 645d4fe14a20; the alternatives are described but not run. Which symmetric behaviour Lucene wants is the maintainers' choice; the recommended one follows the documented model and the #14384 intent.

+41/-6 in 4 files. GenerateCaseExpansion.groovy gets generateClosureOnly() (+24) and a call in the template (+2): for each code point whose ICU UCharacter.toUpperCase or toLowerCase is not in closeOver(SIMPLE_CASE_INSENSITIVE), it emits a case that accepts the other closure members and returns before the Java mappings are added. Regenerating with ./gradlew :lucene:core:regenerateCaseExpansion :lucene:core:tidy adds exactly the two cases below to CaseExpansion.java (+9) and updates its checksum file; RegExp.java:466 drops 0x0130 from the class-3 list.

    // add codepoint
    fn.accept(c);
    // code points whose upper/lowercase is outside their simple case folding closure
    switch (c) {
      case 0x0130: // LATIN CAPITAL LETTER I WITH DOT ABOVE
        return;
      case 0x0131: // LATIN SMALL LETTER DOTLESS I
        return;
      default:
        break;
    }
    // add uppercase from tables

Trade-offs

• Behaviour change for Turkish text: /İstanbul/ no longer matches "istanbul" and /ı/ no longer matches "I". That is what C+S, ICU's closure, UnicodeUtil.foldCase and #14384 say, but it moves further from java.util.regex for these letters (it chooses the C+S closure over Pattern parity; TestRegExp case 14 would fail for İ/ı under it); worth a CHANGES entry.
• Data-driven for ICU: a later ICU upgrade that adds such a code point is handled by regeneration. The generator never reads the JDK's mappings and the task's only declared input is the ICU version (lucene/analysis/icu/build.gradle:287-312), so a JDK-only Unicode change is not seen; a test that sweeps all code points for one-way edges would catch it.
• Build nit: the generator script is not a declared input of regenerateCaseExpansion, so after editing it the task printed SKIPPED until the checksum file was deleted.

What the checks validated

Gradle --no-daemon in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3 (2026-10-31). Stock: the repro fails with assertions on and off. With the fix:

• the repro passes with assertions on and off;
• org.apache.lucene.util.automaton.* plus TestRegexpQuery, TestRegexpRandom, TestRegexpRandom2, TestAutomatonQuery, TestAutomatonQueryUnicode, TestWildcardQuery, TestWildcardRandom, TestPrefixQuery, TestPrefixRandom: 280 tests, 0 failures, 2 skipped, seeds DEADBEEF and CAFEBABE;
• checkGoogleJavaFormat, forbiddenApisMain, ecjLintMain and regenerateCaseExpansionChecksumCheck for core pass;
• the tie-break probe on the fixed jar: 0 asymmetric pairs among the four letters with both flags, accepted sets equal to ICU's closure, 0 one-way edges over every code point (2,938 code points accept more than one, down from 2,940: exactly İ and ı), and RegexpQuery /İstanbul/ hits only "İstanbul".

The whole :lucene:core:test was not run.

Restore what #14192 shipped: i, I, İ and ı mutually equal.

Trade-offs

Symmetric and transitive, matches Pattern and the class-3 sentence; but it is the Turkish conflation #14384 removed ("wrongly mixed in and only make things worse"), and it makes /i/ match "ı" and /I/ match "İ" in every language.

Add i -> İ and I -> ı.

Trade-offs

Symmetric, but still not transitive (İ ~ i ~ I, yet İ does not match I), and it mixes Turkic pairs into the default too.

The generator emits every closure member; expand no longer calls Character.toUpperCase/toLowerCase.

Trade-offs

Also removes the dependence on the running JDK's Unicode version (the study saw 56 Unicode 17 code points that JDK 25 does not map, a separate symmetric info item); costs a switch of about 3,000 cases (2,994 code points with a non-trivial closure under ICU 78.3) instead of today's 87 case labels.

Method: semi-formal (bounded-exhaustive check on the real jar).

• Finder. The study's fuzzing and property-based unit (run 1, main 7cecf79523, 2026-10-22) used Jazzer, jqwik and JQF, plus an exhaustive enumeration of RegExp CASE_INSENSITIVE over all 1,112,064 code points, each compared with an independent oracle (ICU4J case folding, java.util.regex). The enumeration took 1.4 s and found exactly 2 asymmetric pairs.
• Verifier. A separate verifier wrote its own 5-method JUnit (2 characterisation tests that pass, 3 expected-property tests that fail), with ICU4J 78.3 as oracle and RegexpQuery on an index, and ran the 4x4 matrix on 10.1.0 (0 pairs) through main (4 pairs).
• Hunt. A one-test minimal repro, a current-main re-check, a cross-model refute review and a second adjudication with an independent public-API probe followed (see the checks below).