WordDelimiterGraphFilterFactory: a types rule, even a no-op one, also changes how the unlisted characters ¹ ² ³ ¼ ½ ¾ ª º are tokenized

WordDelimiterGraphFilter gives every character a type (lower-case, upper-case, digit or delimiter) and splits a token at type changes such as letter to digit or lower to upper case. Its factory, and the deprecated WordDelimiterFilterFactory, accept a types file whose rules override the type of single characters. Without the option, characters below U+0100 are looked up in DEFAULT_WORD_DELIM_TABLE, built from Character.isLowerCase, isUpperCase and isDigit. With the option, parseTypes fills every entry of a new table, including all 256 Latin-1 ones, from WordDelimiterIterator.getType(i), which maps the Unicode general category, and only then applies the rules (WordDelimiterGraphFilterFactory.java:179-186 on main a9779ef3c193). The two tables disagree on exactly 8 characters: ¹ ² ³ ¼ ½ ¾ are delimiters by default and digits in any types table that does not list them, and ª º are lower-case letters by default and caseless letters (ALPHA) in any types table that does not list them (on Java 8 and later). So a types file whose only rule is - => SUBWORD_DELIM, the type - already has, changes a-b m² 1½ ¼ ªB from [a, b, m, 1, ª, B] to [a, b, m, ², 1½, ¼, ªB], with no error. Only applications that set the opt-in option are hit; with one analyzer at index and query time the terms stay consistent. Every Lucene release from 4.0.0-BETA to 10.6.0 is affected (the graph factory from 6.5.0; 15 releases or heads run, the rest inferred from unchanged code), and seeding the Latin-1 entries from the default table fixes both factories without changing default tokenization.

• Character type: LOWER, UPPER, ALPHA (= LOWER | UPPER), DIGIT, ALPHANUM or SUBWORD_DELIM (a delimiter). The filter splits a token at type changes such as letter to digit or lower to upper case, unless the two types share a bit or a flag turns that split off (isBreak, WordDelimiterIterator.java:227-245), and it drops delimiters.
• Default table: WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE, 256 entries for U+0000-U+00FF: LOWER if Character.isLowerCase, else UPPER if isUpperCase, else DIGIT if isDigit, else SUBWORD_DELIM (WordDelimiterIterator.java:88-105). Above U+00FF the filter calls getType (:302-307).
• getType: a switch on Character.getType, the Unicode general category: OTHER_LETTER -> ALPHA (:322-328), OTHER_NUMBER -> DIGIT (:330-333) (:315-360).
• types option: types="wdfftypes.txt" names files of rules such as $ => DIGIT (the factory javadoc example, WordDelimiterGraphFilterFactory.java:62).

1. inform reads the types files and calls typeTable = parseTypes(wlist) (WordDelimiterGraphFilterFactory.java:139-147). create passes typeTable == null ? DEFAULT_WORD_DELIM_TABLE : typeTable (:155).
2. parseTypes (:164-188) allocates a table of at least 256 entries ("ensure the table is always at least as big as DEFAULT_WORD_DELIM_TABLE for performance", :179), fills every entry with WordDelimiterIterator.getType(i) (:184), then writes the rules (:185-186).
3. Below U+0100, getType and the default table disagree on 8 characters (exhaustive check):

1. So the terms change. A trailing delimiter is dropped (m² -> [m], 1½ -> [1], a lone ¼ -> nothing); as a digit, ² is a new number part after a letter ([m, ²]) and ½ joins the digit before it ([1½]). Two characters whose types share a bit are never split (WordDelimiterIterator.java:228-230): ALPHA shares the upper-case bit with B, so ªB stays one term, while LOWER then UPPER is a case change ([ª, B]). That is also why ª and º change only directly before an upper-case letter (1ª is [1, ª] and Bª is unchanged either way).
2. The deprecated WordDelimiterFilterFactory has the same code (WordDelimiterFilterFactory.java:151, :160-184).

With the no-op rule, an exhaustive sweep of U+0000-U+00FF (each character alone and between letters) finds exactly the 8 characters changed, in both factories. Code that builds the filter directly with DEFAULT_WORD_DELIM_TABLE plus its own entries is not affected; only the factory path is.

• Who is hit. Users of WordDelimiterGraphFilterFactory or the deprecated WordDelimiterFilterFactory with a types file that has any rule; only the characters the file lists keep the type it gives them. The option is off by default, and no built-in Lucene analyzer uses these filters.
• What they get. Different terms for the same text, deterministic and silent, the same with assertions on and off: m² gains the term ², 1½ is indexed as 1½ instead of 1, a lone ¼ becomes a term, ªB is no longer split. Mostly an unannounced tokenization change when the first rule is added, even a rule unrelated to these characters.
• Search results. With the same analyzer (with the types file) at index and query time, terms match either way. Missed or extra hits need analyzers that differ in having a types file (index with types: 1½ -> [1½]; query without: [1]), or a rule added without reindexing.
• Which characters. Superscript one to three, the fractions ¼ ½ ¾ (on every JDK), and the ordinal indicators ª º (on Java 8 and later, whose Unicode data, 6.1 and later, makes them category Lo; on Java 5 to 7 they were Ll and unaffected; Lucene 6.0 and later require Java 8). Characters above U+00FF are not affected.
• Severity. Minor: opt-in configuration, uncommon characters, consistent terms in the usual setup; registry priority low. No prior report was found.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[m]> but was:<[m, ²]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.miscellaneous.TestWdgfTypesRuleKeepsOtherTypes.testNoOpRuleKeepsSuperscriptTwo(TestWdgfTypesRuleKeepsOtherTypes.java:58)

• No written contract either way. Lucene documents the option only as an attribute in the factory javadoc example (WordDelimiterGraphFilterFactory.java:54-64), and no Lucene test sets it. The filter class javadoc says only that delimiters are "by default, all non alpha-numeric characters" (WordDelimiterGraphFilter.java:47, WordDelimiterFilter.java:40), which does not say whether ² or ½ count as numeric. So this is an inconsistency, not a broken documented promise.
• The code comment. "ensure the table is always at least as big as DEFAULT_WORD_DELIM_TABLE for performance" (WordDelimiterGraphFilterFactory.java:179; WordDelimiterFilterFactory.java:175): the full-size table is for speed, which implies the author treated getType(i) and the default table as interchangeable below 256. They are not, for 8 characters.
• The author's description. SOLR-2059 (2010, added the option): "By default, WordDelimiterFilter assigns 'types' to each character (computed from Unicode Properties) ... I think you should be able to customize the defaults with a configuration file". Its CHANGES entry: it "allows you to customize how WordDelimiterFilter tokenizes text with a configuration file". SOLR-3102 (2012), the same author quoting the wiki: "In some cases you might want to adjust how WordDelimiterFilter splits on a per-character basis".
• A no-op rule. - => SUBWORD_DELIM restates the type - already has, and still changes 8 other characters. The SOLR-2059 test (testCustomTypes) checked a sentence of letters, digits and interest-rate whose only special characters were the mapped $ % . ,, plus U+200D; it has none of the 8 characters, and no test, CHANGES entry or discussion records a change of unlisted Latin-1 characters.
• The ª º half came from Unicode data. In Unicode 4.0 to 6.0 (Java 5 to 7, the JDKs of 2010) ª and º are category Ll, so the two tables agreed on them; Unicode 6.1 made them Lo (Java 8 and later) while isLowerCase stays true (unicode.org UnicodeData; no old JDK was run). The 6 number characters differed already in 2010. Lucene 6.0 and later require Java 8, so on every JDK they support, and so for the graph factory, all 8 differ.

Strongest case: the types path is the consistent one. The Solr Reference Guide says "The default for any character without a customized mapping is computed from Unicode character properties" (the option's sample file says the same), which is what getType does; getType is already the filter's classifier above U+00FF; and by default the filter disagrees with itself (x² gives [x] but x⁴ gives [x, ⁴]; 1½ gives [1] but 1⅓ gives [1⅓]). Reviews from another model family (OpenAI GPT-5.6) rated this strong: one from the record alone, without reading the code (its verdict: not real, low confidence), and the read-only fact check of this doc, which still called the 8-character side effect a legitimate consistency defect, though not conclusively.

Our answer: the default table is computed from Unicode properties too (Lowercase, Uppercase, Nd), so that sentence does not pick a side. Whichever table is right, a rule for $, or a rule that changes nothing, that switches 8 other characters from one table to the other is hard to read as intended. We treat it as a consistency defect; maintainers may still prefer the complete getType table as the intended meaning, and that choice is fix B (change the default table, with a reindex note), not "no change". Every reviewer that read the code rated it real (the source-reading cross-model review had seen verifiers' verdicts; the adjudication's confidence is medium). Verdict: a consistency defect with a weak written contract.

Two fixes were applied in a scratch worktree at a9779ef3c193 and run through gradle, and shadowed onto the stock jars for every repro of the hunt. A keeps default tokenization; B changes it. C is not a fix.

Both factories' parseTypes, +18/-12 lines in total: copy the default table into the new one and compute getType only above it, then apply the rules as before. The same behaviour as the finder's fix (i < 256 ? DEFAULT_WORD_DELIM_TABLE[i] : getType(i)), written as a copy. The comment now says why the table starts from the defaults. Recommended if keeping default tokenization matters most.

// parseTypes in both factories (WordDelimiterGraphFilterFactory.java:180-184); comment lines omitted
-    byte[] types =
-        new byte
-            [Math.max(
-                typeMap.lastKey() + 1, WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE.length)];
-    for (int i = 0; i < types.length; i++) types[i] = WordDelimiterIterator.getType(i);
+    byte[] defaults = WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE;
+    byte[] types = new byte[Math.max(typeMap.lastKey() + 1, defaults.length)];
+    System.arraycopy(defaults, 0, types, 0, defaults.length);
+    for (int i = defaults.length; i < types.length; i++) {
+      types[i] = WordDelimiterIterator.getType(i);
+    }

Trade-offs

• A types file now changes exactly the characters it lists; default tokenization (no types file) does not change.
• Users who already have a types file see ¹ ² ³ ¼ ½ ¾ ª º go back to their default types, a one-time tokenization change for those characters that a CHANGES note should mention (reindex if they matter).
• No runtime cost: the table is built once per factory.

What the checks validated

• Stock main a9779ef3c193, gradle: the repro fails 4 of 4 with assertions on and off.
• With fix A, gradle: the repro passes 4 of 4 with assertions on and off; the whole lucene/analysis/common test task with assertions on, repro included, runs 340 classes, 1,913 tests, 0 failures, 18 skipped (among them TestWordDelimiterGraphFilter 28, TestWordDelimiterFilter 16 and TestCustomAnalyzer 26 tests); with the test in place ecjLintMain, ecjLintTest, checkGoogleJavaFormat and forbiddenApisMain ran and passed, and forbiddenApisTest, which had passed in the first fix A run, was UP-TO-DATE in the final one. A first version with Arrays.copyOf failed forbiddenApisMain ("Prefer using ArrayUtil"), hence System.arraycopy.
• With fix A shadowed onto the stock jars: the finder's test 2 of 2, the independent reproducer's 4 of 4, the cross-model reviewer's probe 3 of 3 and the audit's probe 2 of 2 pass.
• Not run: the full Lucene suite, a benchmark (the change only runs when a factory is initialized).

WordDelimiterIterator.java only: the static block becomes tab[i] = getType(i) for i < 256. Both paths then agree, and the default also matches how the filter treats characters above U+00FF (x² like x⁴, 1½ like 1⅓).

Trade-offs

• Changes default tokenization of the 8 characters for every user of the filter, with or without the factory (DEFAULT_WORD_DELIM_TABLE is a public array, though of the @lucene.internal class WordDelimiterIterator; after a Lucene upgrade also the downstream default paths): m² gains the term ², 1½ becomes one term, ªB is no longer split. Needs a reindex note in CHANGES and MIGRATE, so probably only for a major release.
• Keeps the ª º behaviour tied to the JDK's Unicode data, as it already is above U+00FF.

What the checks validated

• Gradle: the repro passes 4 of 4 (assertions on), and the whole lucene/analysis/common test task runs 340 classes, 1,913 tests, 0 failures, 18 skipped, so no existing test pins the default tokenization of the 8 characters. Lint was not run.
• Shadowed onto the stock jars: the independent reproducer's test, the reviewer's probe and the audit's probe pass; the finder's control fails as it must (it asserts 8 table disagreements, and B leaves 0).

Say in the factory javadoc that a types file switches every Latin-1 character it does not list to getType.

Trade-offs

Not a fix: a no-op rule would still change 8 other characters.

Method: semi-formal (an exhaustive table comparison after code reading; no formal tool produced the first signal).

• A hunt unit on the apostrophe and hyphen character tables of Lucene's analysis filters, compared with Unicode, extracted the character tables from the stock main jar and compared them over all code points.
• While building that table model, reading parseTypes suggested that the types path and the default path use different tables. The exhaustive comparison DEFAULT_WORD_DELIM_TABLE[c] versus getType(c) for every c < 256 found exactly 8 disagreements: U+00AA and U+00BA (LOWER versus ALPHA), U+00B2, U+00B3, U+00B9, U+00BC, U+00BD and U+00BE (SUBWORD_DELIM versus DIGIT).
• A public-API JUnit through CustomAnalyzer, with and without the types file $ => DIGIT, turned that into a failing test on the stock jar (2026-10-28, main a9779ef3c193).
• Then checked by an independent reproducer, a reviewer-designed probe, an intended-behaviour devil's advocate, reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a history and prior-report follow-up, a security screen and a credibility audit. This doc added the compact JUnit, the gradle runs and the validation of fixes A and B.