SmartChineseAnalyzer indexes THIN SPACE, EM SPACE, FORM FEED and 11 other whitespace characters as one-character terms

SmartChineseAnalyzer's segmenter has a character class for "Characters that act as a space" (CharType.SPACE_LIKE), which it skips, but Utility.getCharType puts only space, tab, CR, LF and U+3000 in it. VERTICAL TAB, FORM FEED, OGHAM SPACE MARK and the eleven spaces U+2000-U+200A (EN QUAD to HAIR SPACE, among them EN, EM and THIN SPACE) fall through to OTHER, and every OTHER character is emitted as a one-character term that the default stop list does not remove. So hello<U+2009>world is indexed as [hello, U+2009, world], an exact phrase typed with a normal space misses it, a TermQuery for the space character alone finds it, and 中<U+2009>国 is no longer segmented as the word 中国. Nothing throws. It has been there since the original contribution (2.9.0, 2009) and is still on main.

1. The class table. Utility.getCharType (Utility.java:140-159) returns SPACE_LIKE only for ' ', '\t', '\r', '\n' and U+3000 (:146-147). The "Punctuation Marks" ranges come next: 0x0021-0x00BB, 0x2010-0x2642, 0x3001-0x301E (:148-151), then full-width letters and digits and 0xFE30-0xFF63 (:153-157). U+000B and U+000C lie below 0x0021, U+1680 is in no range, and U+2000-U+200A lie just below 0x2010, so all 14 reach return CharType.OTHER (:158).
2. What the classes mean. CharType.SPACE_LIKE: "Characters that act as a space" (CharType.java:38-39); CharType.OTHER: "Other (not fitting any of the other categories)" (:47-48).
3. The segmenter. HHMMSegmenter.createSegGraph skips SPACE_LIKE characters (HHMMSegmenter.java:56-58) and steps over them while it extends a Chinese word (:92), which is why 中 国 becomes 中国. Any other class ends the word. The default branch, which OTHER reaches, makes a one-character WordType.STRING token ("Treat the unrecognized char symbol as unknown string. For example, any symbol not in GB2312 is treated as one of these.", :156-165).
4. The term text. WordSegmenter.convertSegToken copies the original character back into a STRING token (WordSegmenter.java:69-75), so the term is the whitespace character itself.
5. No stop entry. SmartChineseAnalyzer.createComponents chains HMMChineseTokenizer, PorterStemFilter and a StopFilter with the default stop set (SmartChineseAnalyzer.java:124-135). That set "is simply a list of punctuation" (:103-104); it lists U+3000 with the comment "Used as a space in Chinese" (stopwords.txt:54-55) and none of the 14 characters, so the term is indexed.
6. Why exactly 14. Lucene's pinned Unicode 17 White_Space set (UnicodeProps.WHITESPACE, UnicodeProps.java:33-40) has 25 code points: 5 are SPACE_LIKE; U+0085, U+00A0, U+2028, U+2029, U+202F and U+205F fall into the DELIMITER ranges and are removed like a comma; the other 14 fall through to OTHER.

U+000B VERTICAL TAB, U+000C FORM FEED, U+1680 OGHAM SPACE MARK, U+2000 EN QUAD, U+2001 EM QUAD, U+2002 EN SPACE, U+2003 EM SPACE, U+2004 THREE-PER-EM SPACE, U+2005 FOUR-PER-EM SPACE, U+2006 SIX-PER-EM SPACE, U+2007 FIGURE SPACE, U+2008 PUNCTUATION SPACE, U+2009 THIN SPACE, U+200A HAIR SPACE.

Not claimed: the no-break spaces U+00A0 and U+202F, U+205F MEDIUM MATHEMATICAL SPACE, and U+0085, U+2028, U+2029 are removed exactly like a comma (hello<U+00A0>world gives [hello/1, world/2], as hello,world does). That position gap after removed punctuation is tested design (TestSmartChineseAnalyzer.java:119-128), and Lucene has no single rule for no-break spaces. (The third no-break space, U+2007 FIGURE SPACE, is one of the 14.)

• Who is hit. Users of the optional analysis-smartcn module: SmartChineseAnalyzer with any constructor, or HMMChineseTokenizer / the hmmChinese factory with the SmartChinese stop list. Default settings, any scale. Only text that contains one of the 14 characters, in documents or queries.
• What goes wrong. (1) A whitespace-only term is indexed, and a TermQuery for that character finds the document. (2) An exact phrase typed with an ordinary space misses words on both sides of the character; slop 1 still matches, because the extra term takes one position. (3) Chinese characters are no longer joined into a word across it: 中<U+2009>国 is indexed as 中, U+2009, 国, so the term 中国 is not indexed for that text. A FORM FEED inside a word splits it the same way (道<U+000C>具 gives 道, U+000C, 具, where a newline keeps 道具). (4) Each extra term also counts toward the field length that the norms store (IndexingChain.java:2055), so such a document's scores shift slightly (read from the code, not run).
• How the characters arrive. As literal characters (typographic spacing, text extracted with form feeds), or through HTML when an application puts Lucene's HTMLStripCharFilter in front of the analyzer (it is not part of the stock SmartChineseAnalyzer): the char filter decodes  ,   and   to U+2003, U+2002 and U+2009, so an    paragraph indent adds two U+2003 terms.   decodes to U+0020 and is not a trigger.
• Silent. Nothing throws, the index stays valid, and assertions on or off behave the same.
• Likelihood: rare. Most occurrences, such as a paragraph indent, only add an extra term; a wrong match needs a query that spans the character (an exact phrase across it, or a Chinese word split by it) or a query for the character itself. No report in the 17 years since 2.9.0.
• Not affected. Space, tab, CR, LF and U+3000 are skipped as intended. StandardAnalyzer, CJKAnalyzer and WhitespaceAnalyzer emit no whitespace-only term for the same inputs.
• After a fix. Indexed terms change for text with these characters, so affected fields need reindexing.
• Workaround today (from the code, not run): map these characters to U+0020 with a char filter (for example a MappingCharFilter) in front of HMMChineseTokenizer in a custom analyzer. A custom stop set that lists the 14 characters removes the terms too, but, like fix D, it leaves a position gap and does not join Chinese characters across them.

The test sits in org.apache.lucene.analysis.cn.smart like the module's own tests but uses only public API. One 11-character input, the default constructor (so the default stop list), single thread, deterministic. The source is plain ASCII: the THIN SPACE is written as a Java Unicode escape. It compiles from 9.1.0 on (org.apache.lucene.tests). For a release, compile and run the same file against that release's lucene-core, lucene-analysis-common, lucene-analysis-smartcn and lucene-test-framework jars.

Pass -Ptests.asserts explicitly: on current main its default is drawn from the seed and the module path ("Run with assertions for ~75% of all seeds"), and seed DEADBEEF turns assertions off for this module.

The [ ] in the failure message is the THIN SPACE term. The same test is the suggested regression test; it passes with the recommended fix.

# gradle, main 5adff78bb168, seed DEADBEEF, -Ptests.asserts=true
TestSmartChineseThinSpace > testThinSpaceIsSkippedLikeASpace FAILED
    org.junit.ComparisonFailure: term 1 expected:<[world]> but was:<[ ]>
        at org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
        ...
        at org.apache.lucene.analysis.cn.smart.TestSmartChineseThinSpace.testThinSpaceIsSkippedLikeASpace(TestSmartChineseThinSpace.java:27)

# the same failure: gradle with -Ptests.asserts=false; the JUnit runner on main (-ea and -da), 10.6.0 (-ea and -da), 10.5.2, 9.12.3 and 9.1.0
# main, 20 iterations (seed 5C0C2009): 20 of 20 fail
# main with the recommended fix: passes (assertions on and off)

This is the case for calling it a bug. It is medium confidence: no public contract promises how Unicode whitespace is handled, and maintainers may reasonably treat the change as an enhancement (see the devil's advocate below).

• Contract. CharType.SPACE_LIKE is "Characters that act as a space" and OTHER is "Other (not fitting any of the other categories)" (CharType.java:38-39, :47-48). THIN SPACE, EM SPACE and FORM FEED act as spaces, so OTHER contradicts the documented meaning of both classes.
• The stop-list design points the same way. The default stop list lists U+3000 "Used as a space in Chinese" (stopwords.txt:54-55) although U+3000 is already skipped (perhaps only redundancy, but it fits an intent to keep spaces out of the index); HMMChineseTokenizerFactory says the tokenizer "will currently emit tokens for punctuation" and tells users to remove them, for example with the SmartChinese stop list (:27-30); and in #3089 (LUCENE-2014, 2009) rmuir described smartcn as creating "individual tokens for each piece of punctuation (including things like whitespace)" and putting "these in the stopword list". None of the 14 characters is in that list.
• The list was meant to be open. Utility.SPACES reads "Space-like characters that need to be skipped: such as space, tab, newline, carriage return." (Utility.java:41-44); the original Chinese javadoc in e01aad89fe2 ends in "等等" ("etc."), and the original CharType comment says "space, \t\r\n and other space or line-break characters".
• Not a GB2312 rule. VT and FF are ASCII and encodable in GB2312, yet they are not SPACE_LIKE and become terms.
• Other Lucene analyzers tried do not do this. StandardAnalyzer, CJKAnalyzer and WhitespaceAnalyzer emit no whitespace-only term for the same inputs.
• Precedent. #9372 (LUCENE-8325, fixed in 7.4.0) fixed an earlier getCharType misclassification (surrogates turned into bad one-character tokens) as a bug.
• Nothing pins it. No upstream test, CHANGES or MIGRATE entry asserts or mentions a whitespace term.

The strongest case for "intended" came from an independent reviewer from another model family (OpenAI GPT-5.6, refute lens), which reproduced the behaviour, agreed on the cause and the fix, and still called the change an enhancement: Utility.SPACES lists exactly five characters; HHMMSegmenter says "any symbol not in GB2312" becomes an unknown-string token; testNonChinese says other text is split per character "(regardless of Unicode category)" (TestSmartChineseAnalyzer.java:182-185); the classes are @lucene.experimental; and where Lucene promises Unicode whitespace splitting it says so (UnicodeWhitespaceTokenizer).

Why it fails: Utility.SPACES has no reader on main (its last one, SentenceTokenizer, was removed in 6.0.0 by LUCENE-5969) and introduces its examples with "such as"; the GB2312 comment explains the fallback for unrecognized characters, not why VT and FF are unrecognized; testNonChinese is about letters of other writing systems (it asserts Arabic letters), not separators; the claim is "do not index whitespace as a term", not "split like UnicodeWhitespaceTokenizer"; and @lucene.experimental did not stop LUCENE-8325 from being fixed as a bug. What is left is that the contract is descriptive, not a sentence saying "whitespace is never indexed"; that is why the confirmation is medium confidence.

Four options. A was validated with gradle on main 5adff78bb168; B in standalone runs only; C was run by an earlier check and is incomplete; D was not run.

UnicodeProps.WHITESPACE is Lucene's pinned Unicode 17 White_Space set. At the fall-through it can only catch characters no earlier branch classified, which are exactly the 14. The six White_Space characters in the DELIMITER ranges (U+0085, U+00A0, U+2028, U+2029, U+202F, U+205F) keep today's behaviour; U+2007 FIGURE SPACE, a no-break space, is one of the 14 and changes (see the trade-offs). smartcn already requires analysis-common (module-info.java:21, build.gradle:24).

import org.apache.lucene.analysis.util.UnicodeProps;
...
    if (ch >= 0xFE30 && ch <= 0xFF63) return CharType.DELIMITER;
    // Other Unicode whitespace (VT, FF, U+1680, U+2000-U+200A) also acts as a space
    if (UnicodeProps.WHITESPACE.get(ch)) return CharType.SPACE_LIKE;
    return CharType.OTHER;

Trade-offs

• Changes indexed terms for text with these characters: reindex, and a CHANGES entry.
• The set follows Lucene's pinned Unicode version, not the JDK's.
• U+2007 FIGURE SPACE, a no-break space, becomes skipped like any space, so Chinese characters are joined across it (中<U+2007>国: [中/1, U+2007/1, 国/1] today, [中国/1] with the fix); the other no-break spaces, U+00A0 and U+202F, stay DELIMITER and still split, with a position gap (中<U+00A0>国 -> [中/1, 国/2] today and with the fix). In Latin text all three keep separating the words. Treating U+2007 like an ordinary space is a policy choice to flag for the maintainers.

What the checks validated

• Gradle in a scratch worktree of 5adff78bb168 (the script reverts its changes and records an empty git status afterwards), OpenJDK 25.0.3, seed DEADBEEF: stock, the repro fails with assertions on and off; with the fix it passes with assertions on and off.
• The whole module with the fix, :lucene:analysis:smartcn:test with assertions on: 22 tests, 0 failures (TestSmartChineseAnalyzer 19, TestHMMChineseTokenizerFactory 2, the new test 1).
• :lucene:analysis:smartcn:check -x test (formatting, ecj lint, forbidden APIs) passes with the fix and the test.
• On a9779ef3c193 the same change made the broader 7-test check pass 7 of 7 (assertions on and off) and the version driver report the bug absent.

Same result as A without the UnicodeProps dependency. With the patched class applied, the minimal test passes and the broader 7-test check passes 7 of 7 on 5adff78bb168; no gradle module run.

    // VT, FF, OGHAM SPACE MARK and EN QUAD to HAIR SPACE also act as a space
    if (ch == 0x000B || ch == 0x000C || ch == 0x1680 || (ch >= 0x2000 && ch <= 0x200A))
      return CharType.SPACE_LIKE;

Trade-offs

A hand-kept list instead of Lucene's maintained Unicode set.

The finder's first sketch. With it the broader 7-test check still fails 4 of 7.

Trade-offs

Misses VT and FF (it covers only Zs, Zl and Zp); also turns U+00A0, U+202F, U+205F, U+2028 and U+2029 from DELIMITER into SPACE_LIKE; depends on the runtime JDK's Unicode tables. LUCENE-6874 (#7932) chose the White_Space property over Character.isSpaceChar for the same reasons.

Removes the terms for users of the default stop list only.

Trade-offs

Leaves a position gap like removed punctuation, so the exact phrase still misses, and 中<U+2009>国 is still not joined into 中国 (reasoning from the code, not run).

Method: semi-formal, exhaustive over the BMP character table.

• A sweep ran all 65,536 BMP characters through the stock Utility.getCharType on the main jars and cross-tabulated the class against the JDK 25 Unicode general category. For the space separators (Zs, 17 characters): SPACE_LIKE 2, DELIMITER 3, OTHER 12.
• Each space separator was then driven through the stock SmartChineseAnalyzer and its terms printed; the OTHER ones came out as one-character terms.
• The finding was reduced to a public-API JUnit test on the real jars.
• The confirming check restated the scope over the Unicode White_Space property (Lucene's pinned UnicodeProps.WHITESPACE, checked over the whole BMP): that added VERTICAL TAB and FORM FEED and left out U+00A0, U+202F, U+205F, U+0085, U+2028 and U+2029, which behave like punctuation.