ASCIIFoldingFilter folds six letters to the ASCII letter of the opposite case, so "naǧib" is indexed as "naGib" and the query "nagib" finds nothing

ASCIIFoldingFilter converts non-ASCII characters "into their ASCII equivalents, if one exists". Its mapping table, the switch in foldToASCII with 1,242 case labels, groups the labels by output: one block for G, one for g, and so on. Six labels sit in the block of the opposite case: ǥ U+01E5 and ǧ U+01E7 fold to G, the insular s pair is swapped (small ꞅ U+A785 folds to S, capital Ꞅ U+A784 to s), capital Ɐ U+2C6F folds to a and capital Ꜿ U+A73E to c. Each of these lines carries a comment with the letter's Unicode name, which names the other case; the case partners of four of them (Ǥ, Ǧ, ɐ, ꜿ) sit in the right block, and the insular s pair is swapped. A chain that lower-cases and then folds therefore indexes "naǧib" as "naGib", and the lower-cased ASCII query "nagib" finds nothing, with no error. The entries have been in the table since it was added in 2008 and are in every release from 2.9.0 to 10.6.0 (10.6.0 by reading) and on main. A seventh entry, ʗ U+0297 LATIN LETTER STRETCHED C (folds to C), is contested and is raised only as a question.

• The fold table: foldToASCII(char[], int, char[], int, int) (public static since 3.1, tagged @lucene.internal, ASCIIFoldingFilter.java:188-190, called by the filter at :153) is one switch (c) (:200). Each block lists case labels, each with a comment // <char> [UNICODE NAME], and then writes one output, for example output[outputPos++] = 'G'; break;. There are 1,242 case labels.
• Case partner: the capital or small form of the same letter (Ǧ U+01E6 for ǧ U+01E7).

The other insular pairs are placed by case: Ꝺ/ꝺ (:419/:439), Ꝼ/ꝼ (:555/:566), Ᵹ/ᵹ (:611/:624), Ꞃ/ꞃ (:1171/:1195). Line numbers are at main a9779ef3c193.

• Lower-case, then fold: ǧ, ǥ and ꞅ come out as capitals; Ǧ and Ꞅ are lower-cased to ǧ and ꞅ first and then also come out as capitals. Ɐ and Ꜿ are lower-cased to ɐ and ꜿ, which fold correctly, so this order hides those two.
• Fold only (a field that keeps case, like the factory's own javadoc example, WhitespaceTokenizerFactory + ASCIIFoldingFilterFactory): all six fold to the wrong case.
• Fold, then lower-case: every result is correct, because the output is lower-cased afterwards.

On main, WhitespaceTokenizer + LowerCaseFilter + ASCIIFoldingFilter turns "naǧib NAǦIB aǥa ꞅet Ꞅet" into [naGib, naGib, aGa, Set, Set] instead of [nagib, nagib, aga, set, set]; an index of "ağa" and "naǧib" answers the query "aga" with 1 hit and "nagib" with 0. Without lower-casing, ǥ ǧ ꞅ Ꞅ Ɐ Ꜿ fold to G G S s a c instead of g g s S A C, while the controls ğ Ǧ ɐ ꜿ fold to g G a c. An exhaustive fold of all 65,536 BMP code units through the stock jar finds exactly these seven case violations among 891 mapped letters (a check of each letter's JDK case property against the case of its output; letters whose name contains SMALL CAPITAL excluded, see below); with the six-label fix only U+0297 remains, with the seven-label variant none.

• ẛ U+1E9B LATIN SMALL LETTER LONG S WITH DOT ABOVE folds to f (:564); a committer already called it one of the "bugs in ASCIIFoldingFilter that should be fixed" in LUCENE-1343 (#2420, 2010), and it was never changed.
• LFV-819: the ligature ſt U+FB05 has no entry at all, while its sibling st U+FB06 folds to st.

• Who is exposed. Analysis chains that use ASCIIFoldingFilter: built by hand, with CustomAnalyzer, or through the asciiFolding SPI factory; no analyzer shipped with Lucene includes it. The chain must either lower-case before folding or keep case in the folded field. Lucene's own javadoc shows the lower-case-then-fold order: TruncateTokenFilterFactory recommends TurkishLowerCaseFilterFactory followed by ASCIIFoldingFilterFactory for diacritics-insensitive Turkish search (Turkish itself uses ğ, which folds correctly). Both orders occur: Lucene's own TestCustomAnalyzer folds first and lower-cases afterwards, the order that hides the bug in lower-cased fields, while on LUCENE-7525 (#8576) a committer wrote "So ASCIIFoldingFilter is in most analysis chains the very last filter!"
• Which text. ǧ and ǥ have documented modern use, for example in Romani, Skolt Sami (ǥ, as in šiõǥǥ "good"), Lakota, Berber Latin, and scholarly romanization of Arabic, Persian and Pashto names (DIN 31635 uses ǧ), per the reference articles the hunt cited. The insular s pair, turned A and reversed C with dot are mainly specialist (medieval or phonetic) letters.
• Effect. Deterministic for every such token: in the affected chain the ASCII query term (nagib) does not match the indexed term (naGib), so a search on that field misses the document; a query with the same non-ASCII letter still matches. No exception, no index damage, nothing in the logs. preserveOriginal=true keeps the unfolded token next to the wrong one, which does not help an ASCII query.
• How likely. Rare by accident, because these letters are rare in most corpora, but systematic in collections where they occur (transliterated names in scholarship, Sami or Romani text).
• Also affected. Code that calls the public static foldToASCII(...) directly (from 3.1; it is tagged @lucene.internal, so not a supported API contract), and the copies of the table outside Lucene (see the downstream note).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<[g]> but was:<[G]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.miscellaneous.TestASCIIFoldingOppositeCase.testSmallGWithCaron(TestASCIIFoldingOppositeCase.java:25)

• The comments were the verification aid. In LUCENE-1390 (#2464) the contributor who added the Unicode-name comments wrote that the mappings were generated by script "and then moved things around to the appropriate places by hand. I guess this is one weakness of this patch: it's large enough that manual verification is tough. It's my hope that adding the Unicode character descriptions will allow for at least improved verifiability." Six of those descriptions name the case opposite to their block.
• Placement by case was the intent. A later patch revision in the same issue "moves all schwa characters to the 'A' or 'a' depending on their case", and an earlier one "fixes a couple of character mappings that were out of order in the mapped-to-capital-"B" section": hand-placement slips were already being fixed in review.
• The design rule gives the other answer. The stated rule was to "convert characters to those that look like them in ASCII"; a small ǧ looks like a small g, and capital Ɐ and Ꜿ look like capitals.
• Contract and tests. The class javadoc promises "their ASCII equivalents" and its one example keeps case ("'à' will be replaced by 'a'"); the hand-written testLatin1Accents asserts both cases ("À" -> "A", "à" -> "a"). ǧ's canonical decomposition is g + combining caron (NFD U+0067 U+030C).
• The generated test is no evidence of intent. testAllFoldings asserts these outputs, but its comment says "The following Perl script generated the foldings[] array automatically from ASCIIFoldingFilter.java".
• Precedent, and its limits. In LUCENE-1343 (#2420) a committer listed a self-inconsistent entry (U+1E9B) among the "bugs in ASCIIFoldingFilter that should be fixed", but that entry was never changed (ẛ still folds to f, :564). No mapping in the table has changed since 2008 (it is identical at all 21 revisions of the file), so this would be the first table change. LUCENE-7468 (6.3.0, "Bug Fixes") changed the filter's output only under preserveOriginal=true, where it stopped emitting a duplicate token: a limited precedent.

The intended-behaviour checker built the strongest case for "intended": the javadoc has no case clause (its predecessor ISOLatin1AccentFilter said "The case will not be altered", and that sentence was not carried over); the table folds all 39 SMALL CAPITAL letters to upper case on purpose ("ᴄat" becomes Cat even after LowerCaseFilter, executed); the generated test has asserted these outputs for almost 18 years, and maintainers may treat it as a compatibility oracle; in LUCENE-1029 (#2105) participants objected to changing the older accent filter, one because it "would break indexes out there that use it", and a committer wrote that its functionality "shouldn't change in a way that wouldn't be b[a]ckward compatible"; and nobody reported these entries in almost 18 years.

Why it fails: the claim is not "case must be preserved" but six entries placed against their own names (and, for four of them, against their correctly placed partners). Small capitals are a uniform, deliberate choice under the look-alike rule; these six are isolated exceptions in blocks that otherwise follow case. The test was generated from the table. The back-compat objection was about deliberate semantics and argues for a CHANGES/MIGRATE note, not for intent, but it is a real cost: no table mapping has ever changed, so propose the fix for 11.0. The 2010 comparison with ICU folding could not see case errors because ICU folding lower-cases its output. Verdict: not intended for the six. The case survives only for ʗ U+0297. For a bug: Unicode gives it General Category Ll (a lower-case letter), which is why the case check flags it. For intent: its name, LATIN LETTER STRETCHED C, has no SMALL/CAPITAL, it has no case partner (upper-casing returns it unchanged), and C fits the look-alike rule like the small capitals. Raise it as a question only.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF), and the seven-label variant was run too; the documentation-only option and the workaround were not run.

In ASCIIFoldingFilter.java, U+01E5 and U+01E7 move from the G block to the g block, U+A785 and U+A784 swap blocks, U+2C6F moves to A and U+A73E to C, each kept in code-point order inside its block. In TestASCIIFoldingFilter.testAllFoldings, the same six entries move between the same blocks of the generated array, which its comment says was generated from the table. The change is a pure permutation of existing lines: per file, six lines out and the same six lines in (24 changed lines in two files). U+0297 is left alone.

           case '\u01E4': // Ǥ  [LATIN CAPITAL LETTER G WITH STROKE]
-          case '\u01E5': // ǥ  [LATIN SMALL LETTER G WITH STROKE]
           case '\u01E6': // Ǧ  [LATIN CAPITAL LETTER G WITH CARON]
-          case '\u01E7': // ǧ  [LATIN SMALL LETTER G WITH CARON]
 ...      (the rest of the 'G' block; output 'G')
           case '\u0123': // ģ  [LATIN SMALL LETTER G WITH CEDILLA]
+          case '\u01E5': // ǥ  [LATIN SMALL LETTER G WITH STROKE]
+          case '\u01E7': // ǧ  [LATIN SMALL LETTER G WITH CARON]
           case '\u01F5': // ǵ  [LATIN SMALL LETTER G WITH ACUTE]
 ...      (the same move for U+A785/U+A784 between 'S' and 's', U+2C6F to 'A', U+A73E to 'C',
 ...       and for the same six entries in TestASCIIFoldingFilter.testAllFoldings)

Trade-offs

• Reindexing. Indexed terms change for text with these letters: documents indexed with the old table keep naGib, new queries produce nagib. Expect a back-compat reply: no mapping of this table has changed since 2008 (LUCENE-7468 in 6.3.0 changed only the duplicate token under preserveOriginal), and testAllFoldings has asserted these outputs all along. Suggest the fix for 11.0 with a CHANGES entry and a MIGRATE.md note (reindex fields folded by ASCIIFoldingFilter that may contain ǥ, ǧ, ꞅ, Ꞅ, Ɐ or Ꜿ). A 10.x backport is the committers' call.
• No API change. No performance effect is expected (the same switch, the same 1,242 labels and the same output lengths); not benchmarked.
• Copies of the table outside Lucene need the same change separately.

What the checks validated

• Stock main a9779ef3c193 (gradle, seed DEADBEEF): the repro fails 3 of 3 with assertions on and off; TestASCIIFoldingFilter and TestAsciiFoldingFilterFactory pass (6 tests).
• With the fix: the repro and the two upstream suites pass with assertions on and off (9 tests, 0 failures each way, final test file); all of :lucene:analysis:common:test with the repro added passes (340 classes, 1,912 tests, 0 failures, 18 skipped); ecjLintMain, ecjLintTest and checkGoogleJavaFormat pass for the module with the fix, and again for the final test file.
• Run order, for the record: the module-wide run and the fix's lint run used the test before its class javadoc was reworded (a comment-only change); after the rewording, google-java-format left the file unchanged, and the stock and fixed repro runs above were repeated with it.
• The table change alone is not enough: with only ASCIIFoldingFilter.java changed, the unchanged upstream TestASCIIFoldingFilter fails 1 of 5 (testAllFoldings), so the generated array must move too.
• Through the classpath runner (fix shadowed onto the main jars): the repro passes 3 of 3 with -ea and -da; the finder's broader JUnit then fails only on U+0297.
• The fix, the fix with the repro as a new test file, and the seven-label variant all apply cleanly to a clean a9779ef3c193 tree.

The finder's original sketch: the same six moves plus U+0297 from the C block to the c block, in the table and in the generated test.

Trade-offs

ʗ (LATIN LETTER STRETCHED C) is General Category Ll in Unicode, but it has no case partner (upper-casing returns it unchanged) and no SMALL/CAPITAL in its name, so C may be intended under the look-alike rule, like the small capitals. Better raised as a question for the reviewers than bundled into the fix.

What the checks validated

• This doc's gradle run: the repro, the finder's JUnit, TestASCIIFoldingFilter and TestAsciiFoldingFilterFactory pass (13 tests, 0 failures).
• In the hunt, through the classpath runner: 10 tests, 0 failures, two seeds; the exhaustive check then reports 0 case violations.

Say in the javadoc that a few entries fold across case, and recommend folding before lower-casing.

Trade-offs

Not run. A reasonable choice for 10.x if committers do not want indexed terms to change in a minor release (it could pair with the fix in 11.0); on its own it leaves fields that keep case wrong and the six comments contradicting their blocks.

Put LowerCaseFilter after ASCIIFoldingFilter; that hides all six in lower-cased fields. A field that keeps case can use, for example, a MappingCharFilter override for these letters.

Trade-offs

Not run. Changes the analysis chain, so it also needs reindexing, and it does not help other users of the table.

Method: bounded-exhaustive checking of the stock jar against an exact reference (semi-formal tier), in a hunt unit that checked the ASCII folding table against the Unicode Character Database.

• A planning probe first folded every BMP code unit through the stock jar's public static ASCIIFoldingFilter.foldToASCII and flagged letters whose ASCII output has the opposite Unicode case.
• The hunt then turned this into a proof by exhaustion over all 65,536 BMP code units on the stock jar, against JDK 25 Unicode data (NFKD, case properties, character names) and ICU4J 78.3 UTR#30 folding, with an independently parsed model of the source table that agrees with the jar on 65,536 of 65,536 code units. The case property (a mapped lower-case letter folds to lower-case ASCII and vice versa; letters whose name contains SMALL CAPITAL excluded as a uniform rule) has exactly 7 violations among 891 mapped letters, one more (U+0297) than the probe listed; with the seven-label fix, 0. This case check is a heuristic (the JDK case property of the letter against the case of a single-case output); that six of the seven sit opposite the case named in their own comments was then read from the source.
• The finding was reduced to a public-API JUnit and checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a history, prior-report and security follow-up, and an audit. Found 2026-10-28 on main a9779ef3c193.