WikipediaTokenizer drops Chinese and Japanese text inside wiki markup

WikipediaTokenizer tokenizes plain Chinese and Japanese text (one <CJ> token per character of its CJ class), and it tokenizes Latin and Korean text inside wiki markup with the markup's type ([[Japan]] gives Japan|il, [[한국]] gives 한국|il). But the Chinese and Japanese characters inside markup (the grammar's CJ class: Han, Hiragana, Katakana and a few other ranges) are skipped without an error: [[日本]], [[Category:日本]], == 歴史 ==, {{日本}} and <ref>日本</ref> give no token, [http://example.org 日本] gives only the URL, and x ''日本'' y gives x and y, with y typed bold. In 東京は[[日本]]の首都である。 9 of its 11 Han and kana characters become tokens; with the title in bold, a common pattern in MediaWiki lead sentences, only 6 of 11, and a search for 東京 or 日本 misses the page. The grammar's only rule that emits these characters ({CJ}) is for plain text, while each of the nine markup lexical states has one word-token rule, for {ALPHANUM}, and skips what no rule matches, and CJ shares no character with ALPHANUM. All three output modes are affected. It dates from the tokenizer's first commit (LUCENE-1103, 2008; first release 2.3.0) and is in every release up to 10.5.2 and on main. Adding {CJ} to the nine per-state token rules and regenerating the scanner with the build's own task fixes it: the repro, an exact-token test and the whole analysis-common test suite pass, with and without assertions.

• Lexical state: the tokenizer's scanner is generated from a JFlex grammar, WikipediaTokenizerImpl.jflex, and reads in one mode at a time. Plain text is YYINITIAL; [[, [[Category:, [, '', ==, {{ and <ref> switch to a markup state (:208-214), and each state has its own token rules. A character that matches no token rule of the current state is skipped.
• ALPHANUM and CJ: the grammar's word classes. ALPHANUM is letters (a hard-coded list that ends at U+1FFF, plus U+FFA0-FFDC), digits, and Hangul syllables and jamo (:111, :154-158). CJ is Hiragana, Katakana, Bopomofo, Hangul compatibility jamo, CJK ideographs, part of the CJK Compatibility block (U+3300-337F) and halfwidth Katakana (:161). They share no character.
• Markup types: text inside markup is typed with the markup: il (internal link), c (category), h and sh (headings), i, b, bi (italics, bold, both), ci (template, citation), elu and el (external-link URL and label). Plain CJ text is typed <CJ>, one character per token.

1. The only CJ token rule is <YYINITIAL>{CJ} (WikipediaTokenizerImpl.jflex:201).
2. Each of the nine markup states has one word-token rule, for {ALPHANUM} (:222, :231, :237, :246, :257, :269, :281, :288, :300), and an ignore rule for what no rule matches ([^], or for external links the global ignore rule at :331). The only other rules there that return a token, the external-link URL (:230) and STRING's | (:307), cannot match a CJ character. So every CJ character inside markup is skipped.
3. The {ALPHANUM} rule also moves the scanner on. In bold, only a word moves THREE_SINGLE_QUOTES_STATE to STRING (:257), the one state that recognises the closing ''' (:297). After a bold CJ title the scanner stays in the bold state, which it leaves only on a Latin word or a link opener (:259-261), and keeps skipping the plain text after it: '''日本'''は東アジアの国である。 gives no token at all. In x ''日本'' y the closing '' is read as a bold opener, so y is typed b.
4. The loss happens in the scanner, so the UNTOKENIZED_ONLY and BOTH modes have nothing to collapse. reset() returns the scanner to YYINITIAL (WikipediaTokenizer.java:318-324), so the next value starts clean.

Scope. "Chinese and Japanese" here means the CJ class. Some Han characters are outside it: 々 (U+3005), 〇 (U+3007) and supplementary-plane Han are lost in plain text too (時々 gives only 時), which is a separate limit of the hard-coded classes, not this bug; fix A does not change it.

Search over three documents, the plain sentence, the one with the link and the one with the bold title: TermQuery(body:日) and PhraseQuery(body:"日 本") find 1 of 3 on main (3 of 3 with the fix), and PhraseQuery(body:"東 京") 2 of 3. With UNTOKENIZED_ONLY or BOTH and every markup type untokenized, [[日本]] and == 歴史 == still give no token.

• Who is hit. Users of WikipediaTokenizer from analysis-common: its public constructors, or the SPI name wikipedia (WikipediaTokenizerFactory.java:41), whose javadoc shows a Solr schema example (:28-33). No Lucene analyzer uses it, and it is @lucene.experimental (WikipediaTokenizer.java:40). Within that use no special setting is needed: the default constructor and the factory default (TOKENS_ONLY, WikipediaTokenizerFactory.java:52) are affected, and so are UNTOKENIZED_ONLY and BOTH.
• What triggers it. Chinese or Japanese text in links, categories, headings, bold, italics, templates, <ref> citations or external-link labels. A bold title in the first sentence, a common MediaWiki pattern, also loses the plain text after it up to the next link or Latin word. No large input or unusual character is needed. Korean (Hangul syllables) is kept; Hangul compatibility jamo, which the grammar puts in CJ, are lost.
• What goes wrong. The lost words are not in the field's terms, so term and phrase queries on them miss the page (only positive queries were run). Silent and deterministic: no exception, the stored value keeps the text, the same with -ea and -da.
• How much. In the example sentence 2 of 11 Han and kana characters, with a bold title 5 of 11. On a synthetic three-paragraph article with a bold title, 51 of 51 Chinese/Japanese characters were lost. Real articles were not measured.
• Not more than that. A 1,350,000-character input with 400,000 CJ characters inside markup tokenized in 68-113 ms with no exception (no benchmark or memory study); other fields and documents are not affected, because reset() returns the scanner to its initial state.
• Long-lived. Every release since 2.3.0 (2008). A fix changes the tokens of values with CJ text inside markup only, so existing indexes need a reindex.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[日本]> but was:<[]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.wikipedia.TestWikipediaTokenizerLinkCjk.testCjkInsideInternalLink(TestWikipediaTokenizerLinkCjk.java:37)

• The class's purpose. "Extension of StandardTokenizer that is aware of Wikipedia syntax" (WikipediaTokenizer.java:35-38). The grammar treats Chinese/Japanese as a token type in plain text (WikipediaTokenizerImpl.jflex:201, type <CJ> at WikipediaTokenizer.java:85); the CJ and KOREAN macros and the {CJ} rule were copied from StandardTokenizer's grammar of the time.
• Markup content is meant to become typed tokens. Upstream testHandwritten (TestWikipediaTokenizer.java:66-285) and testLinks (:315-340) assert that link, category, italics, bold, heading, citation and external-link text comes out typed il, c, i, b, h, ci, el. Those tests are ASCII-only; neither upstream test file contains a CJK character.
• Only CJ disappears. Latin and Korean inside all ten markup constructs are kept (the tested forms), and so are the letters and digits of the APOSTROPHE, ACRONYM, COMPANY, EMAIL, HOST and NUM token classes (split into ALPHANUM pieces).
• No sign it was decided. The nine states got their {ALPHANUM} rules in the tokenizer's first commit (f715fc60317, LUCENE-1103); git log -S'{CJ}' on the grammar finds only that commit, and later edits of these lines (position increments, LUCENE-1133's counters, explicit break;, LUCENE-5357's [^]) never touched CJ. LUCENE-1103's description and comments never mention CJK, Chinese, Japanese or scripts.
• Keep the claim precise. The contract is implied (purpose, upstream tests, consistency with Latin, Korean and plain-text CJ), not quoted. The mistyping of following words is not CJ-specific: x '''!!!''' y z already gives y|b z|b, with or without the fix; only the loss of text is.
• Maintainers may prefer a bigger change. In LUCENE-6993 (#8050), sarowe wrote that "Performing an effective upgrade would probably mean cloning the current StandardTokenizer and then re-layering the wiki syntax". Maintainers may prefer such a rewrite, or deprecation, to a local fix.

Strongest case for "intended" (the independent reproduction: moderate-to-weak; the cross-model reviews: moderate): the javadoc says the tokenizer "is based off of the Wikipedia tutorial ... but it may not be complete" and marks it @lucene.experimental (WikipediaTokenizer.java:36-40); its author tested it only on the English Wikipedia dump ("I have only tested it with the benchmarking EnwikiDocMaker wikipedia stuff", LUCENE-1103); the grammar's own design sketch defines every markup body as ALPHANUM runs (//INTERNAL_LINK = "["{2}({ALPHANUM}+{WHITESPACE}*)+"]"{2}, WikipediaTokenizerImpl.jflex:334-343) with an ignore catch-all in every state; and the tokenizer is little maintained (LUCENE-6993 kept it on Unicode 3.0; it carries @IgnoreRandomChains(reason = "TODO: it seems to mess up offsets!?"), WikipediaTokenizer.java:42).

Why it fails: "may not be complete" is about which Wikipedia syntax is recognised, and here the syntax is recognised and the text inside it thrown away; @lucene.experimental warns that the API or grammar may change (LUCENE-1135); the sketch's ALPHANUM includes KOREAN, so the design keeps non-Latin text inside markup, and nothing explains dropping only Chinese and Japanese; LUCENE-6993 is about the Unicode version of the character data, not about which rules each state has. Verdict: not intended, a defect (the intended-behaviour check, the adjudication and the audit). Expect maintainers may still file the change as an improvement to an experimental, English-tested tokenizer.

Option A was validated in a scratch worktree at a9779ef3c193 (OpenJDK 25.0.3): grammar edit, ./gradlew :lucene:analysis:common:regenerateWikipediaTokenizer, then the repro, an exact-token test, the whole analysis-common test suite and its static checks, with and without assertions. Options B and C were not run. The fix was not benchmarked. Until a fix ships there is no workaround within WikipediaTokenizer's options: StandardTokenizer keeps all the text (<IDEOGRAPHIC>, <HIRAGANA> tokens) but loses the markup types and also indexes markup keywords such as Category. The sibling defect in the same grammar (the hard-coded LETTER/DIGIT ranges, which lose letters such as µ, ª, º and fullwidth digits in plain text) is separate: neither fix fixes the other.

In each of the nine per-state token rules (WikipediaTokenizerImpl.jflex:222, :231, :237, :246, :257, :269, :281, :288, :300), match ({ALPHANUM}|{CJ}) instead of {ALPHANUM}, with the same action, and regenerate the scanner with ./gradlew :lucene:analysis:common:regenerateWikipediaTokenizer, which also rewrites the checksum file src/generated/checksums/regenerateWikipediaTokenizer.json. 9 grammar lines; the generated WikipediaTokenizerImpl.java changes in about 380 lines of tables.

The result: Chinese/Japanese inside markup becomes one token per character typed with the markup's type, as plain CJ text is one <CJ> token per character (== 歴史 == gives 歴|h 史|h, [[Category:日本]] 日|c 本|c). Because CJ now fires the same action as ALPHANUM, the state transitions happen too: '''東京'''は[[日本]]の首都である。 keeps all 11 Han and kana characters (東|b 京|b, は as <CJ>, 日|il 本|il, then の首都である), and in x ''日本'' y the y is <ALPHANUM> again. Offsets are per character with position increment 1; the first external-link label token shares the URL's position, as Japan does in [http://example.org Japan]. The untokenized modes work: UNTOKENIZED_ONLY with il gives 日本|il [2,4) for [[日本]] and 東京 Tokyo|il [2,10) for [[東京 Tokyo]]; BOTH gives the collapsed token, then 日 (increment 0) and 本. Plain text and Latin and Korean inside markup are unchanged.

 <INTERNAL_LINK_STATE>{
-  {ALPHANUM} {yybegin(INTERNAL_LINK_STATE); numWikiTokensSeen++; return currentTokType;}
+  ({ALPHANUM}|{CJ}) {yybegin(INTERNAL_LINK_STATE); numWikiTokensSeen++; return currentTokType;}
 ...
 <THREE_SINGLE_QUOTES_STATE>{
-  {ALPHANUM} {yybegin(STRING); numWikiTokensSeen++; return currentTokType;}
+  ({ALPHANUM}|{CJ}) {yybegin(STRING); numWikiTokensSeen++; return currentTokType;}
 ...
 // the same change in the other seven states (9 rules in all), then:
 ./gradlew :lucene:analysis:common:regenerateWikipediaTokenizer

Trade-offs

• Token changes beyond the lost text. CJ shares no character with any macro the nine states use (ALPHANUM, HOST, P, the markup delimiters), so the new alternative matches only characters that today fall to an ignore rule. But the state changes and counters it now triggers also change tokens main already emits in the same value: x ''日本'' y gives y|<ALPHANUM> instead of y|b; in [http://example.org 日本 Japan] the label token Japan gets position increment 1 instead of 0; and UNTOKENIZED_ONLY turns [[東京 Tokyo]] from Tokyo|il [5,10) into 東京 Tokyo|il [2,10). Inputs without CJ characters inside markup are unchanged; the module's tests agree. Existing indexes need a reindex.
• CJ inside markup gets the markup type (il, c, h, b, ...), not <CJ>. That matches what the tokenizer does for Latin and Korean and lets the untokenized modes collapse it, but a filter that selects <CJ> tokens will not see these (option B is the other way).
• Also covered: Hangul compatibility jamo, the part of the CJK Compatibility block in CJ (U+3300-337F) and halfwidth Katakana ([[ㄱㄴ]] gives ㄱ|il ㄴ|il). Not covered: characters outside CJ, such as 々, 〇 and supplementary-plane Han, which are lost in plain text too.
• Not covered, pre-existing: bold or italics whose content is neither ALPHANUM nor CJ (x '''!!!''' y z) still type the following words b; the hard-coded LETTER/DIGIT ranges (the sibling defect) are untouched.
• No benchmark was run.
• Maintainers may prefer a rewrite on top of the current StandardTokenizer, or deprecation (see "Why it is not intended").

What the checks validated

• The gradle regenerate task ran cleanly and updated the checksum file.
• The repro: 8 of 9 fail on stock, 9 of 9 pass with the fix (gradle and the repro runner, -ea and -da). The finder's repro passes 8 of 8 and the independent reproduction's test 17 of 17 with the regenerated scanner.
• An exact-token test (TestWikipediaTokenizerCjkInMarkupTokens, 5 tests: tokens, offsets, types and position increments for the link sentence, the bold-title sentence, italics, an external-link label and the two untokenized modes): 5 of 5 pass with the fix (-ea and -da), 5 of 5 fail on stock.
• :lucene:analysis:common:test, the whole module with both new tests: 341 suites, 1,923 tests, 0 failed, 18 skipped, with assertions on (seed 5EED0708) and off (seed 5EED0709); upstream TestWikipediaTokenizer 8 of 8 and TestWikipediaTokenizerFactory 6 of 6. Stock with both new tests: 13 failed, all in the new tests.
• :lucene:analysis:common:check -x test (format, ecjLint, forbiddenApis, every regenerate checksum check) passes; analysis.tests TestBugInSomething (its testCuriousWikipediaString uses WikipediaTokenizer): 0 failed.

Add per-state rules that return the CJ type but make the same transition as ALPHANUM (for example THREE_SINGLE_QUOTES_STATE to STRING). Not run.

Trade-offs

Keeps the plain-text type, but loses the markup information that is the class's purpose, and the untokenized modes would not collapse these tokens with the rest of the link (they collapse runs of one token type, WikipediaTokenizer.java:228-230, :273-275). More rules to get right.

One macro change. Not run.

Trade-offs

It also changes plain text: a run of CJ characters would become one <ALPHANUM> token instead of one <CJ> token per character, and runs mixed with Latin letters would merge.

Method: bounded-exhaustive content-coverage sweep of the stock jar (the hunt's semi-formal tier).

• The sweep (2026-10-27, main a9779ef3c193). A hunt unit on letters lost by tokenizers ran every string of length 1-3 over a 107-character alphabet, 1,236,599 strings per tokenizer configuration, with the oracle "every letter or digit of the input lies in some token". For WikipediaTokenizer it reported Han, Kana and Hangul compatibility-jamo letters lost after '' (for example ''一).
• Root cause. A probe over the markup constructs showed the same loss in links, categories, headings, templates, citations and external-link labels, and a read of the grammar's lexical states found the missing {CJ} rules. The finder reduced it to a public-API JUnit.
• Checks. Claude verifiers wrote their own tests (the independent reproduction: 17 tests with an index check and a regenerated-scanner fix check; the intended-behaviour check: a devil's advocate who found the bold-title case). Review records by a model from another family (OpenAI GPT-5.6, executed mode) wrote and ran their own tests: both "real, high". An adjudication re-ran the independent reproduction's test with a fresh seed, and an independent audit ran its own probe.