Nori's unigram mode ignores discardPunctuation: $, @, line breaks and spaces become index terms

Nori's KoreanTokenizer drops punctuation, symbols and whitespace by default (discardPunctuation=true). Its opt-in outputUnknownUnigrams=true splits each word the dictionary does not know into one-character tokens. With both on, the unigram branch of Viterbi.backtrace adds every character without the shouldFilterToken check that the other branch applies, so the punctuation, symbol and whitespace characters inside an unknown word become terms of their own: 가격: $100 gives [가격, $, 1, 0, 0], 다음\n\n문장 gives [다음, \n, \n, 문장], 50% 할인 gives [5, 0, %, ' ', 할인], and an e-mail address keeps its @ and .. The extra terms carry the SY tag of every unigram, so the default part-of-speech stop filter keeps them, and each one pushes later words one position on, so exact (zero-slop) phrases across them miss (가격 100 finds 0 documents instead of 1). No error. It is opt-in (the korean factory, the constructors and KoreanAnalyzer can turn it on; Elasticsearch and OpenSearch cannot). LUCENE-8784 removed the check in 8.2.0, in the same diff that made shouldFilterToken honour the new flag; 8.1.1 and kuromoji filter each unigram, and putting the check back fixes it.

• Nori: Lucene's Korean analysis module. KoreanTokenizer finds the cheapest split of the text into dictionary words with a Viterbi search; a stretch the dictionary does not know becomes an unknown word.
• discardPunctuation (default true): drop tokens that are punctuation, symbols or whitespace. The check is shouldFilterToken: the flag is on and the token's first character is punctuation, a separator, a control or format character, or a math, currency, modifier or other symbol (Viterbi.java:412-445).
• outputUnknownUnigrams (default false): emit each unknown word as one token per character instead of one token for the word.

1. Viterbi.backtrace turns the best path into tokens in two branches. The else branch (Viterbi.java:237 on) emits a word as one token; a single-morpheme word, or any word in mode NONE (:248-249), is added only if (shouldFilterToken(token) == false) (:250). Unknown words and dictionary punctuation such as : take this checked path (with unigrams off, : and $ are dropped).
2. With unigrams on, an unknown word takes the first branch (:214): it loops over the word's characters, builds one token per character from the NGRAM entry of the unknown dictionary, and calls pending.add(token) with no check (:232).
3. So every punctuation, symbol or whitespace character that is part of an unknown word is emitted, whatever discardPunctuation says.
4. Every unigram gets the part-of-speech tag SY (dict/UnknownDictionaryBuilder.java:31), which is not in KoreanPartOfSpeechStopFilter.DEFAULT_STOP_TAGS (:36-68). The leaked tokens pass KoreanAnalyzer's stop filter, and no part-of-speech filter can remove only them: letters and digits of unknown words carry the same tag.
5. Each leaked token takes a position, so every later token moves one position on per leaked token.

가격: $100: the best path is 가격 (a dictionary noun), : (a dictionary token, tag SC), $ (an unknown word after the space) and 100 (an unknown word). Without unigrams, : and $ fail the check and 100 stays whole: [가격, 100]. With unigrams, : still goes through the checked branch and is dropped, but $ and 100 are split and added unchecked: 가격@0, $@1, 1@2, 0@3, 0@4. With the fix: 가격@0, 1@1, 0@2, 0@3, which is what the phrase query "가격 1 0 0" (the analyzed 가격 100) needs.

Hello, world. does not leak: , and . are dictionary tokens there, handled by the checked branch, and a single space between words is a separator. Kuromoji's equivalent loop (EXTENDED mode) checks each unigram: if (!discardPunctuation || !isPunctuation(fragment[offset + i])) (ViterbiNBest.java:448).

• Who is exposed. Nori users who turn on unigram mode and keep the default flag: the 4-argument KoreanTokenizer constructor (it always passes discardPunctuation=true, KoreanTokenizer.java:104-110) or the 5-argument one with true, true; the korean factory with outputUnknownUnigrams="true" (discardPunctuation defaults to true, KoreanTokenizerFactory.java:96-97; Solr schemas expose both); KoreanAnalyzer(userDict, mode, stopTags, true), which has no discardPunctuation option and always uses true (KoreanAnalyzer.java:57-74). The classes are @lucene.experimental.
• Not exposed. The default configuration (unigrams off), and users who set discardPunctuation=false. Elasticsearch and OpenSearch pass a literal false for outputUnknownUnigrams in their nori tokenizer and analyzer, so their settings cannot reach it (read in their source; not executed).
• What triggers it. Ordinary text with a punctuation, symbol or whitespace character inside an unknown-word run: prices ($100, 50%), e-mail addresses, C++, ab#cd, tabs, line breaks, a space after a symbol, and since 8.10.0 doubled spaces. Deterministic: every such document, every time.
• Effect on the index. Extra one-code-point terms such as $, @, ., %, a newline or a space, which the default stop filter keeps; and shifted positions, so an exact (zero-slop) phrase across the character misses; slop can absorb the gap. Executed: the phrase 가격 100 misses 가격: $100; the phrase 회의 자료 ("meeting materials") misses documents with $, U+200B, a doubled space, two newlines or U+00AD between the words, and matches them with the fix; the same probe's slop-2 phrase matches the same documents with and without the fix.
• Effect on queries. Query text with such a character gains a junk term (createBooleanQuery("$") becomes body:$).
• Not affected. Term queries over punctuation-free words; no term the fixed code emits is lost; offsets are exact and CheckIndex is clean. No hang or amplification: 1M-character punctuation inputs ran at 0.97-1.24x an all-letter control, at most one token per character.
• No equivalent stock token-filter workaround. A token filter that empties punctuation-only tokens and drops empty ones removes the terms but keeps the position gaps (가격@0, 1@2, ..., executed), so exact phrases would still miss. A patternReplace char filter that turns [\p{P}\p{S}\p{Cc}\p{Cf}\p{Z}]+ into one space in front of the tokenizer gave gap-free output on the 7 inputs tried (가격: $100 -> [가격@0, 1@1, 0@2, 0@3], executed), but it changes the text the tokenizer segments, removes punctuation the dictionary would handle, was tried on 7 inputs only and must be in both the index-time and the query-time analyzer. Otherwise turn unigrams off, or apply the fix and reindex: existing indexes keep the extra terms until then.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 1 expected:<[1]> but was:<[$]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.ko.TestKoreanUnigramsDiscardPunctuation.testUnigramsKeepPunctuation(TestKoreanUnigramsDiscardPunctuation.java:34)

• The contract. "@param discardPunctuation true if punctuation tokens should be dropped from the output" (KoreanTokenizer.java:119, :150; the factory says the same, KoreanTokenizerFactory.java:61). outputUnknownUnigrams is documented only as "if true outputs unigrams for unknown words" (:102, :118, :149): how unknown words are split, not that punctuation is kept.
• The introducing change's own words. CHANGES 8.2.0: "LUCENE-8784: The KoreanTokenizer now preserves punctuations if discardPunctuation is set to false (defaults to true)." (lucene/CHANGES.txt:6318-6319).
• History. Nori's first release (LUCENE-8231, 7.4.0) checked every unigram; the check is in every tag from 7.4.0 to 8.1.1. LUCENE-8784 (a556925eb8a, 8.2.0) made shouldFilterToken return discardPunctuation && isPunctuation(...) and, in the same diff, replaced the checked add in the unigram branch with an unconditional one. The flag-aware check already returns false when the flag is false, so the option did not need the check removed: on 8.2.0 with the check restored, discardPunctuation=false still keeps every character (가격: $10.5 gives [가격, :, ' ', $, 1, 0, ., 5]), and the default drops punctuation again. Removing it changed only the default case.
• The record. Neither the commit message, the CHANGES entry nor any of the 17 comments of #9828 mentions the punctuation check in the unigram branch or a change to the default output. Two of the comments are the commit notices of db334c792be and its branch_8x copy, which a day later restored the NGRAM part-of-speech tag that the patch had changed "inadvertenly" (sic) on the unigram path; they did not touch the check. That shows one unintended side effect of the patch; it is circumstantial, not evidence about the removed check itself.
• Maintainers on record. "Punctuations are removed by default" (#10020) and "The tokenizer removes punctuations so it's expected that some characters are filtered" (#9739).
• The sibling. Kuromoji checks each EXTENDED-mode unigram (ViterbiNBest.java:448); executed (saved run of the verifier), JapaneseTokenizer gives 価格: $100 -> [価格, 1, 0, 0] and ab cd -> [a, b, c, d].
• Tests. The only deterministic unigram input in TestKoreanTokenizer is 2018 평창 동계올림픽대회, with no punctuation (two assertions on it, :366-372 and :374-405); its punctuation tests use discardPunctuation=false with unigrams off. No test asserts punctuation unigrams.

The strongest case for "works as designed" (the intended-behaviour checker): read literally, "outputs unigrams for unknown words" covers every character of an unknown word; a committer signed off on the removal; LUCENE-8784 was about keeping the decimal point, and unigram mode does keep the . of 10.1; the classes are @lucene.experimental; no report was found since 2019; the flag still drops separators and dictionary punctuation in unigram mode; users can filter the tokens themselves; and changing it now changes the analysis for anyone who relies on the punctuation unigrams.

Why it fails: the discardPunctuation javadoc is unconditional and the CHANGES entry keeps punctuation only when the flag is false. 7.4.0 to 8.1.1 filtered, and restoring only the check on 8.2.0 brings that back without hurting the new option. Kuromoji filters each unigram, and maintainers said punctuation is removed by default. The record shows no decision to change the default; the next-day NGRAM tag fix is circumstantial only. Keeping the . also when the flag is true goes beyond the issue's stated design, though that does not settle what the patch author intended. Filtering downstream does not work cleanly with token filters: the leaked tokens share the SY tag with every unigram, and a token filter leaves position gaps (a char filter avoids them on the inputs tried but changes the text the tokenizer sees). Few reports fit an opt-in option that two prominent downstream servers cannot turn on. The compatibility point and the explicit patch review are real: the fix changes analysis output for unigram users and needs a CHANGES line and a reindex note. Verdict: on balance not intended, a regression.

One fix was built and validated with gradle in a scratch worktree at a9779ef3c193; it is the same change as the finder's and the independent reproducer's. The alternatives and the workaround are for discussion.

Restores the 7.4.0 to 8.1.1 code in the unigram branch of Viterbi.backtrace (+5/-3 in Viterbi.java). It mirrors the other branch's check at :250; with discardPunctuation=false the check returns false, so LUCENE-8784's option is unchanged.

@@ -229,9 +229,11 @@ final class Viterbi
                   charLen,
                   backWordPos + i,
                   backWordPos + i + charLen);
-          pending.add(token);
-          if (VERBOSE) {
-            System.out.println("    add token=" + pending.get(pending.size() - 1));
+          if (shouldFilterToken(token) == false) {
+            pending.add(token);
+            if (VERBOSE) {
+              System.out.println("    add token=" + pending.get(pending.size() - 1));
+            }
           }
         }
       } else {

Trade-offs

• Behaviour: unigram mode with the default flag drops punctuation again, including the . of 10.1 ([1, 0, ., 1] becomes [1, 0, 1]). Users who want it set discardPunctuation=false, which keeps it with or without the fix, or follow the keepDecimalPoint proposal #15894.
• Existing indexes keep the extra terms until reindexed; after the upgrade, exact phrases across punctuation match newly indexed documents but still miss old ones (reasoning). Worth a CHANGES line.
• Cost: one check per unigram and fewer tokens; no slowdown seen in the follow-up's 1M-character runs (one shared host, so only "no measurable change" is claimed).
• Backport: on 8.x and 9.x the same block is in KoreanTokenizer.backtrace (the code moved to Viterbi in 10.0.0); only main was built and tested with gradle.
• Not covered (pre-existing, both branches): shouldFilterToken checks only the first char, so an emoji or other supplementary symbol is still emitted, and a BMP non-spacing mark such as U+FE0F is not in its classes either; with the fix the phrase 회의 자료 still misses 회의😀자료.

What the checks validated

• Gradle, JDK 25: the new test fails 4 of 6 on stock (assertions on and off) and passes 6 of 6 with the fix; with the fix TestKoreanTokenizer, TestKoreanTokenizerFactory, TestKoreanAnalyzer and the new test pass (38 tests, 2 nightly skipped); the whole module passes, 14 suites and 93 tests (87 upstream + 6 new; 4 skipped: 3 nightly and the @Ignored TestKoreanNumberFilter.testLargeData, so 83 upstream tests ran), with assertions on and off; the nightly run of TestKoreanTokenizer and the new test (checkRandomData over the unigram analyzer, strings of up to 8192 characters) passes 23 of 23, so 84 distinct upstream tests ran and passed; :lucene:analysis:nori:check passes (checkGoogleJavaFormat, ecjLint, forbiddenApis, tests); tidy changes nothing.
• Standalone runner with the fix: the brief's test 6/6 (assertions on and off), the finder's 4/4, the independent reproducer's 6/6; earlier, the follow-up's security probe 3/3 and a four-configuration token dump (discardPunctuation=false still keeps every character).
• On 8.2.0, restoring the same check makes the history probe and the 8.x JUnit port pass.

if (!discardPunctuation || !isPunctuation(fragment[fragmentOffset + i])) { ... pending.add(...) }, the form of ViterbiNBest.java:448.

Trade-offs

Same result for these inputs and skips allocating dropped tokens; the recommended form is the smaller diff and matches nori's other branch.

Keep the code and add a javadoc exception.

Trade-offs

Contradicts the 7.4.0 to 8.1.1 behaviour, kuromoji and the maintainers' statements, and leaves SY-tagged junk terms that no stock token filter was shown to remove without position gaps.

Turn outputUnknownUnigrams off. Or remove the terms with a token filter: a patternReplace filter that empties punctuation-only tokens (^[\p{P}\p{S}\p{Z}\p{Cc}\p{Cf}]+$ to "") followed by length with min=1 (executed). Or rewrite the text first with a patternReplace char filter that turns [\p{P}\p{S}\p{Cc}\p{Cf}\p{Z}]+ into one space in front of the korean tokenizer (executed on 7 inputs).

Trade-offs

The token filter keeps the position gaps (가격@0, 1@2, ..., executed), so exact phrases across the removed characters would still miss (reasoning). The char filter gave gap-free output on the 7 inputs tried, the same as the fixed tokenizer where compared, but it changes the text the tokenizer segments, removes punctuation the dictionary would handle and must be in both the index-time and the query-time analyzer (reasoning). Turning unigrams off changes how every unknown word is indexed.

Method: bounded-exhaustive buffer-edge sweep with metamorphic oracles (the hunt's "semi-formal" tier), hunt unit "SPI analysis-chain fuzzing", 2026-10-26.

• The sweep built 91 analysis chains only through SPI names and legal parameters (CustomAnalyzer) and analyzed, for every chain, filler alphabet (13), probe string (13) and insertion position around buffer and refill boundaries (255 to 16384), plus a control position far from any boundary: 900,263 checks per run, two runs.
• Eight oracles: no exception, offsets in range, start offsets non-decreasing, final offset, position increments and lengths, no unpaired surrogates, probe preservation (W7: the probe's first code point occurs in the emitted terms as often as at the control position), and reuse identity.
• W7 fired on many chains; the hunt triaged the other hits as by design (filters that multiply, merge or drop tokens, the 255-character token limit, dictionary lemmas and transliterations that contain spaces). The korean tokenizer with outputUnknownUnigrams=true and the space probe was the only W7 hit not explained by design and the only nori configuration with any violation: 71 rows in the first run (5 in the second), where the number of spaces in the emitted terms differed from the control position. That a space shows up in the terms at all is the leak.
• The finder saw that the leak does not depend on buffer edges, cut it down by hand to the general punctuation leak, found the cause by reading Viterbi.backtrace and the LUCENE-8784 diff, and wrote the public-API JUnit. Independent AI tools then reproduced and judged it (see the checks).