Nori user dictionary: a trailing space or inline comment on a noun's line can change how the next word is analysed

Nori's user dictionary stores, for each custom noun, a right context id that tells the tokenizer whether the noun's last syllable ends in a consonant (a coda), because the reading of the next word depends on it. UserDictionary takes the noun from the line's first whitespace-separated part but the id from the last character of the whole line (UserDictionary.java:94). A line with trailing whitespace, or with an inline # comment after a space (the comment is removed, the space stays), therefore gives the same noun the generic id 3533 instead of 3535 or 3534. The next morpheme can then get another part of speech, and KoreanAnalyzer's default stop filter keeps or drops other terms: 사람이나 ("person or") is [사람] with the line 사람 but [사람][이] with 사람 (9.12.3, 10.5.2 and main), and on 9.12.3 학교가방을 샀다 ("bought a school bag") loses the term 가방 ("bag") with 학교 . Silent and deterministic; opt-in (user dictionaries only); present since nori was added in 7.4.0. Fix: read the character from the noun, with a guard for an empty noun.

• Nori user dictionary: a text file with one custom noun per line, optionally followed by its parts (세종시 세종 시); # starts a comment. It is loaded with UserDictionary.open(Reader) and passed to KoreanTokenizer, KoreanAnalyzer or the korean factory's userDictionary file.
• Right context id: a number per dictionary entry that the tokenizer's Viterbi search uses to score the morpheme that follows. User nouns get one of three: 3535 when the last syllable has a final consonant (a coda, as in 사람), 3534 when it has none (학교), 3533 when the last character is not Hangul (UserDictionary.java:40-45). Korean particles depend on the coda: 사람이 but 학교가.
• Part-of-speech stop filter: KoreanAnalyzer drops tokens whose tag is in KoreanPartOfSpeechStopFilter.DEFAULT_STOP_TAGS (:36-68): endings such as EC and particles such as JX; the copula VCP is kept.

1. UserDictionary.open removes comments with line.replaceAll("#.*$", "") and keeps the line untrimmed (UserDictionary.java:56-64): 사람 # memo becomes 사람 .
2. The constructor takes the noun from entry.split("\\s+")[0] (:89-90). Java's split drops trailing empty strings, so 사람, 사람 and 사람\t give the same noun, no segmentation and the same FST key (:125-131).
3. But it picks the id from char lastChar = entry.charAt(entry.length() - 1);, the last character of the whole line (:94-103). A space or tab is not Hangul, so the noun gets 3533.
4. Viterbi stores that id on the noun's node and uses it as the left side of the connection cost to the next morpheme (morph/Viterbi.java:469-472, :530-539), so another reading of what follows can win, and the stop filter then keeps or drops other terms. In every executed probe the noun itself is still emitted (the code does not guarantee it: the id enters the cost of every path through the noun).
5. Secondary trigger, same line: a segmentation shorter than the noun, 세종시 세종, ends in 종 (coda), so 세종시 gets 3535 instead of 3534; only segmentations longer than the noun are rejected (:114-121).

On main, the cheapest rest of the path for 사람이나 after the noun: from 3535, 이나/JX costs -10 and 이/VCP 나/EC 321, so the particle wins and the terms are [사람]; from 3533, 이/VCP 나/EC costs 184 and 이나/JX 315, so the terms are [사람][이].

(tagger = KoreanTokenizer with DecompoundMode.NONE.) On 10.x and main a separate bug, LFV-658, leaves the three constants stale for the dictionary that 10.0 switched to, which is why the plain line's 학교가 reads as 가/VV ("go"). The claim is invariance: the same noun must get the same id however its line is written.

• Who is exposed. Applications that load a nori user dictionary: the KoreanTokenizer constructors that take a UserDictionary (KoreanTokenizer.java:104-106, :121-123, :154-159), KoreanAnalyzer(userDict, ...) (KoreanAnalyzer.java:59), or the korean factory's userDictionary file (KoreanTokenizerFactory.java:44-57, :111-123). The classes are @lucene.experimental. Without a user dictionary nothing changes.
• What it takes. A noun line that ends with whitespace, or with a # comment after a space. Both are easy to produce in a hand-edited file. A comment written right after the noun (사람# memo) is not affected.
• When it shows. The text must then continue the noun with something whose reading depends on the context. With correct context ids (9.12.3, or main with LFV-658's fix) an independent reproducer saw 0 of 336 well-formed noun + matching-particle sentences change, but continuations such as 이나, 다, 지, 나, 과 and noun + noun compounds such as 학교가방 do: 67 of 1,420 probe sentences on main with LFV-658's fix, 113 of 1,420 on stock main, which mixes in LFV-658. The finder's sweep of 16 nouns x 21 particles (plain line vs # memo) found 34 differences on 9.12.3 and 61 on main (reported by the finder; the raw output was not kept and the sweep was not re-run).
• Effect. Extra terms ([사람][이]), lost terms (가방 on 9.12.3) or another part of speech for the morpheme after the noun, at index and at query time alike. No exception, with assertions on or off.
• Scale. No threshold: one dictionary line is enough, and it applies to every occurrence of that noun. A fix changes the terms of fields analysed with an affected line, so those fields need a reindex.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[사람]> but was:<[사람, 이]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.ko.TestUserDictionaryTrailingSpace.testTrailingSpaceInUserDictionaryLine(TestUserDictionaryTrailingSpace.java:51)

• The contract. RIGHT_ID_T is the "NNG right with hangul and a coda on the last char", RIGHT_ID_F "... and no coda on the last char" (UserDictionary.java:42-45). The coda belongs to the noun's last syllable, and the noun is token; the class javadoc speaks of "adding custom nouns (세종) or compounds (세종시 세종 시)" (:32-35).
• The parser's own rules. Whitespace is a separator: split("\\s+") gives the same surface and the same (absent) segmentation for 사람 and 사람 (:89-90), and the FST key is the same (:125-131). The implementation removes # and everything after it anywhere on a line (// Remove comments above replaceAll("#.*$", ""), :57-58), although the line grammar is not documented; the space before # survives only because the line is not trimmed.
• Tests. userdict.txt and TestKoreanTokenizerFactory.java:72 use only whole-line comments and complete segmentations, and no test pins a user entry's right-id value or checks that it is the same however the line is written (TestKoreanTokenizer.testDuplicate only checks getRightId(3) != 0 to count deduplicated entries, TestKoreanTokenizer.java:563-577). With the fix, all 85 upstream nori tests that ran pass.
• History. The line is in nori's first commit (LUCENE-8231, e851b89cbeb1); neither #9278 (62 comments) nor any later commit explains reading the line's end. LUCENE-8603 (7.7.0, listed under "Bug fixes": "Fix the inversion of right ids for additional nouns in the Korean user dictionary", lucene/CHANGES.txt:6910-6911) fixed the values of the same three constants.
• Design evidence only. An independent C++ port of nori strips comments and whitespace and takes the coda from the surface term.

The strongest case for "not a bug" (the intended-behaviour checker, the cross-model intended lens and the adjudication): Lucene documents no line grammar for the nori user dictionary (open has no javadoc; the factory says only "User dictionary path"); the classes are @lucene.experimental; first-party tests use only whole-line comments; kuromoji accepts # comments only at line start since LUCENE-8596 (CHANGES.txt:4839-4841), so an inline comment may count as unsupported; and kuromoji rejects segmentations that do not add up to the surface (LUCENE-8933, #9976).

Why it fails: it does not cover plain trailing whitespace, which nori's own parser already treats as a separator, and nori strips # comments anywhere on the line (it never adopted LUCENE-8596's line-start rule). The constants' comments tie the id to the noun, and LUCENE-8603 treated wrong user-noun ids as a bug. The case is strong for the shorter-segmentation trigger (세종시 세종): the documented example is a complete segmentation, kuromoji requires the parts to add up to the surface, and the line is plausibly malformed input that nori accepts only because it does not validate it. Verdicts: devil's advocate real (medium-high for whitespace and comments, low for the segmentation); cross-model intended lens real (high), its intended argument "moderate" and failing for bare trailing whitespace; adjudication confirmed (high).

One fix was built and validated with gradle in a scratch worktree at a9779ef3c193; it is the finder's one-line change plus the guard that an independent reviewer showed it needs. The alternatives are for discussion.

Take lastChar from token, the surface form, instead of the cleaned line (+7/-1 in UserDictionary.java). The guard matters: a line that starts with whitespace ( 학교) gives token = "". Stock main rejects that line later with IllegalArgumentException: ... the segmentation is bigger than the surface form (); without the guard token.charAt(-1) would throw StringIndexOutOfBoundsException first. With the guard it stays an IllegalArgumentException, with a clearer message. Moving the right-id block below the segmentation check instead keeps the old message; that variant is the one the diagram's fix sketch shows, and it was also executed (every 사람 line form gives 3535 and [사람]; 사람 still throws IllegalArgumentException). It also fixes the secondary trigger (세종시 세종 now gets 시's id).

@@ -91,7 +91,13 @@ public final class UserDictionary
       if (token.equals(lastToken)) {
         continue;
       }
-      char lastChar = entry.charAt(entry.length() - 1);
+      if (token.isEmpty()) {
+        throw new IllegalArgumentException(
+            "Illegal user dictionary entry " + entry + " - the surface form is empty");
+      }
+      // the right context depends on the last syllable of the surface form, not on whatever
+      // ends the line (trailing whitespace, or the space left before a removed comment)
+      char lastChar = token.charAt(token.length() - 1);

Trade-offs

• On 10.x and main, until LFV-658 is fixed, lines with trailing whitespace move from the generic id to the stale coda ids, so their analysis changes, and not always for the better: after this fix 학교가 reads as 가/VV with every line form. Ship both fixes together; with both, the independent reproducer's sweeps show 0 differences.
• Fields analysed with an affected line need a reindex; worth a CHANGES line.
• Cost: none to speak of (one charAt at dictionary load; not benchmarked).

What the checks validated

• Gradle, JDK 25, scratch worktree: the new test fails 3 of 5 on stock (assertions on and off) and passes 5 of 5 with the fix; TestUserDictionary, TestKoreanTokenizer, TestKoreanAnalyzer, TestKoreanTokenizerFactory and the new test pass (39 tests, 2 nightly skipped); the whole module passes, 14 suites and 92 tests (87 upstream + 5 new; 4 skipped: 3 nightly and the @Ignored TestKoreanNumberFilter.testLargeData), with assertions on and off; the nightly run of TestKoreanTokenizer, TestKoreanAnalyzer and the new test passes 29 of 29; :lucene:analysis:nori:check passes (checkGoogleJavaFormat, ecjLint, forbiddenApis, tests); tidy changes nothing.
• Standalone runner with the fix: the new test 5 of 5 (assertions on and off), the finder's 2 of 2 (it includes 세종시 세종), the independent reproducer's 7 of 7; every token-dump row gives the plain line's output for every line form.
• Earlier: the one-line change without the guard passes the version-portable test on 7.4.0 and main (6 of 6).

line = line.replaceAll("#.*$", "").strip(); in UserDictionary.open.

Trade-offs

Fixes trailing whitespace and the inline comment, not the shorter segmentation; changes another behaviour (leading whitespace accepted instead of rejected); strip() also removes Unicode spaces such as U+3000 that the \s+ split does not treat as separators. Can be combined with the recommended fix.

What the checks validated

Standalone runner: the new test's three bug tests pass, but the control testLeadingWhitespaceIsRejected fails (4 of 5), because 학교 is now accepted.

char lastChar = token.charAt(token.length() - 1); with no guard (the finder's first patch).

Trade-offs

Correct for every well-formed line (its own check and 32 upstream nori tests pass), but turns the leading-whitespace IllegalArgumentException into a StringIndexOutOfBoundsException (executed separately).

As kuromoji does since LUCENE-8933.

Trade-offs

Would reject 세종시 세종 outright; it changes accepted input and touches the area of LFV-603 (part offsets), so keep it out of this fix. Not run.

Keep noun lines free of trailing whitespace and put comments on their own line, or right after the noun with no space (사람# memo behaves like 사람, executed). A loader that strips each line removes the trailing-space form but not 사람 # memo unless it also removes comments.

Method: differential probe of the stock jar (the hunt's "semi-formal" tier), during a hunt unit that checked the CJK tokenizers' Viterbi output against an exhaustive minimum-cost path over the same dictionary data (kuromoji first, then nori), 2026-10-27, main a9779ef3c193.

• While modelling nori's user-dictionary costs for that reference, the finder read UserDictionary.java:94 and saw that the right id comes from the cleaned line, not from the noun.
• A differential probe, NoriWsSweep, analysed 16 user nouns x 21 particles = 336 inputs, each with the plain line and with the same line plus # memo: the finder reports 34 differences on 9.12.3 and 61 on main (the raw output was not kept; not re-run).
• The finder then wrote a public-API JUnit (2 of 2 fail on main, 10.5.2, 9.12.3 and 9.0.0) and the one-line fix (later given an empty-noun guard). No solver or model checker produced the signal.
• Independent AI tools reproduced and judged it (see the checks). The same unit also found a separate root cause in the same class, the stale context ids (LFV-658), filed as its own bug.