ASCIIFoldingFilter folds the ligature st to "st" but has no entry for its sibling ſt (U+FB05), so "faſt" is not found by the query "fast"

ASCIIFoldingFilter converts non-ASCII characters "into their ASCII equivalents, if one exists", and its javadoc lists the Alphabetic Presentation Forms block. Its table folds every Latin ligature of that block, ff fi fl ffi ffl (U+FB00-FB04) and st (U+FB06 to st), and the long s ſ (U+017F to s), but it has no case label for ſt U+FB05 LATIN SMALL LIGATURE LONG S T, whose Unicode compatibility decomposition (NFKD) is "st" like U+FB06's. ſt therefore falls to the default: branch and is copied unchanged: "faſt" is indexed as faſt, and the ASCII query "fast" finds the documents spelled "faſt" and "fast" but not "faſt", with no error. The table never had the entry (LUCENE-1390, 2008), so every release from 2.9.0 to 10.6.0 (10.6.0 by reading) and main is affected. For users of the filter it replaced it was an unnoticed regression: ISOLatin1AccentFilter folded U+FB05 (to ft), and U+FB05 is the only one of its 72 mappings that ASCIIFoldingFilter lacks. The fix is one case label plus the matching entry of the generated test; it changes indexed terms for text with ſt.

• The fold table: foldToASCII(char[], int, char[], int, int) (public static, tagged @lucene.internal, ASCIIFoldingFilter.java:188-190) is one switch (c) (:200). Each block lists case labels, each with a comment // <char> [UNICODE NAME], and then writes the output characters. incrementToken() (:105) calls it through foldToASCII(char[], int) (:122, :146) for every token that has a character at or above U+0080.
• NFKD: the Unicode compatibility decomposition. It turns st and ſt into "st" and ſ into "s" (the JDK's java.text.Normalizer).

Line numbers are at main a9779ef3c193.

• With StandardTokenizer, LowerCaseFilter and ASCIIFoldingFilter at index and query time, "faſt" and "fast" are indexed as fast and "faſt" as faſt. The query "fast" (body:fast) counts 2 of 3 documents; the query "faſt" still finds the third.
• preserveOriginal=true adds the original token only when folding changed it (needToPreserve, :166). For "faſt" nothing changed, so the index still holds only faſt, and "fast" again finds 2 of 3.
• The CustomAnalyzer chain "standard", "lowercase", "asciiFolding" gives the same 2 of 3.

An exhaustive fold of all 65,536 BMP code units through the stock jar took, in the 16 blocks the javadoc lists, every letter or digit whose NFKD (nonspacing marks removed) is ASCII: 631 code points, 583 folded, 48 not. 47 of the 48 have an explanation other than an oversight. 39 are whole classes the table never covers: 2 ordinal indicators (ª º) and 37 MODIFIER LETTER forms (the table has no ORDINAL or MODIFIER LETTER entry). 8 are subscripts U+2095-209C, added in Unicode 6.0 (2010), after the table was generated from "the Unicode 5.1 version of UnicodeData.txt" in 2008; the older subscripts U+2090-2094 are folded (:272, :273, :540, :1058, :1490). U+FB05 is the only gap that already existed when the table was made (Unicode 1.1, like U+FB06) and has a folded sibling of the same shape in its block. An independent scan of U+FB00-FB4F finds U+FB05 as the only unfolded letter there with an ASCII NFKD.

• LFV-818: six letters fold to the ASCII letter of the opposite case (for example ǧ to G), and a seventh (ʗ U+0297 to C) is contested.
• ẛ U+1E9B LATIN SMALL LETTER LONG S WITH DOT ABOVE folds to f (:564); a committer listed it among the "bugs in ASCIIFoldingFilter that should be fixed" in LUCENE-1343 (#2420, 2010).

• Who is exposed. Analysis chains that use ASCIIFoldingFilter: built by hand, with CustomAnalyzer, or through the asciiFolding SPI factory; no analyzer shipped with Lucene includes it. The text (documents or queries) must contain the precomposed ligature ſt U+FB05.
• Which text. U+FB05 appears mainly in transcriptions and digitizations of early-modern long-s printing and in text extracted from documents set with historical ligatures. Ordinary text writes "st" with two letters or uses U+FB06.
• Effect. Deterministic for every token that contains ſt: the ASCII query term (fast) does not match the indexed term (faſt), while the same word written with st or ſ is found. A query that itself contains ſt still matches. No exception, no index damage, nothing in the logs.
• Settings. Assertions on or off make no difference, and preserveOriginal=true does not help.
• How likely. Rare: nobody has reported it in the 17 years since 2.9.0. But systematic in collections where the ligature occurs.
• Also affected. Code that calls the public static foldToASCII(...) directly (tagged @lucene.internal, so not a supported contract), and the copies of the table outside Lucene (see the downstream note).
• Workarounds today. ICUFoldingFilter instead of ASCIIFoldingFilter, or core CaseFoldingFilter (new in 10.6.0; it maps U+FB05 to U+FB06) in front of ASCIIFoldingFilter: both find 3 of 3 on the example. Either changes the analysis chain, so it needs a reindex too.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<fa[st]> but was:<fa[ſt]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.miscellaneous.TestASCIIFoldingLongSTLigature.testLongSTLigatureFoldsToST(TestASCIIFoldingLongSTLigature.java:25)

• The table's own rows set the rule. It folds ſ U+017F to s (:1224), st U+FB06 to st (:1255-1258) and every other Latin ligature of the block (:575-596). By its own standard st is a "reasonable ASCII alternative" for ſt, which is ſ + t. U+FB05 is the only hole among U+FB00-FB06.
• Unicode and Lucene's other folding code agree on st. NFKD(U+FB05) = NFKD(U+FB06) = "st" (JDK Normalizer, executed). Lucene's ICU folding data maps both to "st" (utr30/nfkc_cf.txt:3031, FB05..FB06>0073 0074), and core SimpleCaseFolding maps U+FB05 to U+FB06 (:459-460).
• A silent regression for users who migrated. LUCENE-1351 (2.4.0, filed under "Bug fixes": "ISOLatin1AccentFilter now cleans additional ligatures") added U+FB05 -> ft to ISOLatin1AccentFilter. Four months later LUCENE-1390 added ASCIIFoldingFilter and deprecated the old filter "in favor of ASCIIFoldingFilter which covers a superset of Latin 1"; from 2.9.0 to 3.6.2 the new javadoc said "The set of character conversions supported by this class is a superset of those supported by Lucene's ISOLatin1AccentFilter". The CHANGES entry on main still says the old filter "handles a subset of this filter", a contributor wrote in the issue: "This class includes all of ISOLatin1AccentFilter.", and in February 2011 a co-author repeated on the dev list that "ASCIIFoldingFilter provides a superset of it mappings". Of the old filter's 72 mappings, 71 are identical in the new table and U+FB05 is the only one missing; a scan of the 3.6.2 jars finds U+FB05 as the only character the old filter folds and the new one leaves unchanged. What migrating users lost was a match on "faft" (the old filter gave ft); the query "fast" has never matched "faſt" through either filter. The superset promise covered 2.9.0-3.6.2 only, so for current versions the case rests on the table's own consistency, not on a broken guarantee.
• No record of a decision. The issue thread never mentions U+FB05, long s or ligatures; the final patch in the JIRA archive already lacks the entry; no test, CHANGES or MIGRATE entry, issue or list mail found says it was left out on purpose. No upstream test asserts that U+FB05 passes through (testUnmodifiedLetters covers only § ¦ ¤), and testAllFoldings was generated from the table by a Perl script (TestASCIIFoldingFilter.java:150-151), so it mirrors the table. The contributor called the table generated from UnicodeData.txt "and then moved things around to the appropriate places by hand. I guess this is one weakness of this patch: it's large enough that manual verification is tough."
• Why it was left out is not recorded. One plausible mechanical cause (an inference, not documented): U+FB05's one-level decomposition is U+017F U+0074, which is not ASCII, while U+FB06's is U+0073 U+0074.
• What not to claim. The current javadoc does not promise this mapping: the superset sentence left together with ISOLatin1AccentFilter in 4.0 (a58c26978f5, LUCENE-2781, 2010-11-30).

The intended-behaviour checker built the strongest case for "intended": the javadoc says "only those characters with reasonable ASCII alternatives are converted", so the table is a curated selection with no completeness promise; the ICU module calls ICUFoldingFilter "a more thorough replacement for ASCIIFoldingFilter and LowerCaseFilter that applies the same ideas to many more languages", so gaps are expected and the remedy is to switch filters; the table has not changed since 2008 and any change alters indexed terms; and the contributor's rule, to "convert characters to those that look like them in ASCII", makes ſt ambiguous between ft (what it looks like, and what the old filter produced) and st (what it is).

Why it fails: the table already chose meaning over look for the long s (ſ, ẜ U+1E9C and ẝ U+1E9D fold to s) and folds st to st, so st is reasonable by its own standard, and NFKD, ICU folding and core case folding agree. Nothing indicates the omission was a choice: the 2.9.0-3.6.2 javadoc promised a superset of the old filter, which folded ſt, and U+FB05 is the only break. The ICU note says the ICU filter is broader, not that the table leaves a Latin presentation form of a block its javadoc lists out on purpose. The table is stable in practice (no mapping has changed since 2008, not even ẛ U+1E9B to f, called a bug in #2420 in 2010), but no freeze policy was found in the code, CHANGES, MIGRATE or the issue threads examined; back-compat is a real cost (a reindex note) and a likely review question, not evidence of intent. Verdict: the evidence favours an accidental omission; nothing found shows U+FB05 was left out on purpose. What survives is a priority argument (a committer may answer "use ICUFoldingFilter") and the ft versus st question: propose st and say plainly that the old filter gave ft.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF), with the table line alone and with the generated-test entry; the alternatives were not run.

In ASCIIFoldingFilter.java, one new case label case '\uFB05': // ſt [LATIN SMALL LIGATURE LONG S T] right above the U+FB06 label, so both write s, t. In TestASCIIFoldingFilter.testAllFoldings, the generated array's st entry gains "ſt" next to "st" (the comment above the array says it was generated from the table by a Perl script). 2 files, +3/-1 lines.

           case '\u00DF': // ß  [LATIN SMALL LETTER SHARP S]
             output[outputPos++] = 's';
             output[outputPos++] = 's';
             break;
+          case '\uFB05': // ſt  [LATIN SMALL LIGATURE LONG S T]
           case '\uFB06': // st  [LATIN SMALL LIGATURE ST]
             output[outputPos++] = 's';
             output[outputPos++] = 't';
             break;
 ...      (and in TestASCIIFoldingFilter.testAllFoldings: "ſt" // U+FB05 ... + "st" // U+FB06 ..., folded "st")

Trade-offs

• Reindexing. Indexed terms change for text with ſt: documents indexed before the fix keep faſt, new queries produce fast. Suggest a CHANGES entry that says affected fields need a reindex; whether it goes to 10.x or only 11.0 is the committers' call. No mapping of this table has changed since 2008, so expect a back-compat question.
• The value. st matches NFKD, the ICU folding data, core SimpleCaseFolding and the table's own ſ and st rows; the old ISOLatin1AccentFilter produced ft.
• No API change. No performance effect is expected (one more label in the same switch, a two-character output like U+FB06's, within the 4 x length buffer); not benchmarked.
• Copies of the table outside Lucene need the same line separately.

What the checks validated

• Stock main a9779ef3c193 (gradle, seed DEADBEEF): the repro fails 2 of 3 with assertions on and off; TestASCIIFoldingFilter and TestAsciiFoldingFilterFactory pass (6 tests).
• Table line only: the repro passes 3 of 3 with assertions on and off, and the two upstream suites pass (6 tests). testAllFoldings checks only the entries it lists, so it does not fail without the test change; the test change adds coverage for the new entry.
• Full fix: the repro passes 3 of 3 with assertions on and off; the two upstream suites pass (6 tests); all of :lucene:analysis:common:test with the repro added passes (1,912 tests, 0 failures, 18 skipped); ecjLintMain, ecjLintTest and checkGoogleJavaFormat pass for the module, and the repro needed no reformatting.
• Through the classpath runner (fix shadowed onto the main jars): the repro passes 3 of 3 with -ea and -da. In the hunt, the finder's and the independent reproducer's JUnits pass with the fix (2 of 2, 6 of 6), and the finder's JUnit plus the two upstream suites pass 8 of 8 with both parts.
• The fix, the table line alone, and the fix with the repro as a new test file all apply cleanly to a clean a9779ef3c193 tree.

The same one-line change in a group that writes f, t.

Trade-offs

Not run. Fits the look-alike rule and restores the old filter's output exactly, but disagrees with Unicode and with every other Lucene folding component, and the query "fast" would still miss "faſt".

Say in the javadoc that U+FB05 is not folded.

Trade-offs

Not run. Avoids changing indexed terms, but leaves the hole and the search miss.

Use ICUFoldingFilter instead of ASCIIFoldingFilter, or put core CaseFoldingFilter (10.6.0 and later) in front of it; both give 3 of 3 on the example.

Trade-offs

Changes the analysis chain, so it also needs a reindex, and it does not help other users of the table.

Method: bounded-exhaustive checking of the stock jar against an exact reference (semi-formal tier), in a hunt unit that checked the ASCII folding table against the Unicode Character Database.

• A planning probe first folded every letter or digit of the 16 javadoc-listed blocks whose NFKD (nonspacing marks removed) is ASCII through the stock jar's public static ASCIIFoldingFilter.foldToASCII and listed the ones left unchanged.
• The hunt then turned this into a proof by exhaustion over all 65,536 BMP code units on the stock jar, against JDK 25 NFKD, character names and Unicode ages and ICU4J 78.3 UTR#30 folding, with an independently parsed model of the source table that agrees with the jar on 65,536 of 65,536 code units. Its coverage property has 631 such code points, 583 folded and 48 not; U+FB05 is the only one of the 48 that is neither a class exclusion (2 ordinal indicators, 37 MODIFIER LETTER forms) nor added to Unicode after the table (8 subscripts U+2095-209C, Unicode 6.0, while the older subscripts U+2090-2094 are folded), and it has a folded sibling of the same shape in its block. This coverage property (P7) is one of several the harness checks; what the others flagged includes the separate table bugs listed under "Related entries in the same table".
• The finding was reduced to a public-API JUnit and checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a history, prior-report and security follow-up, and an audit. Found 2026-10-28 on main a9779ef3c193.