Nori user dictionary: since 10.0, user nouns carry the old dictionary's context ids, so the particle after them is misread

Nori's user dictionary gives every custom noun a hard-coded left context id and one of three right context ids, which the tokenizer's Viterbi search uses to score the morpheme that follows: RIGHT_ID 3533 "NNG right", RIGHT_ID_T 3535 "... with hangul and a coda on the last char", RIGHT_ID_F 3534 "... no coda" (UserDictionary.java:40-45) and LEFT_ID 1781 "NNG left" (UserMorphData.java:25-26). These are the ids of mecab-ko-dic 2.0.3, the dictionary 9.x ships. LUCENE-10416 (76c9fd4e38a, 2022, first released in 10.0.0) moved nori to mecab-ko-dic-2.1.1-20180720, in which the same numbers name other contexts: 3534 is now the common noun with a final consonant, 3535 a "state change" subclass without one, 1781 the "state change" left class; the plain-noun ids are now 1780 and 3532/3533/3534. So on 10.x the coda flag is inverted: on the usual one-word line, a user noun ending in a vowel is scored as if it ended in a consonant, and the reverse. The particle after it is then misread: 학교가 gives 가/VV ("go") instead of 가/JKS, 학생이 gives 이/VCP (copula) instead of 이/JKS, and KoreanAnalyzer indexes [학교, 가] and [학생, 이]; a new word fares the same (깃허브가 있다 gives [깃허브, 가, 있]). Silent, opt-in (needs a user dictionary), in every 10.x release and main, not in 9.x, whose dictionary matches the ids. Fix: the four 2.1.1 numbers, plus a test that ties them to the bundled dictionary; fields indexed with a user dictionary then need a reindex.

• Nori user dictionary: a text file with one custom noun per line (optionally its parts, 세종시 세종 시), loaded with UserDictionary.open(Reader) and passed to KoreanTokenizer, KoreanAnalyzer or the korean factory's userDictionary file.
• Context ids: every dictionary entry has a left id (how it joins what comes before) and a right id (how it joins what comes after). The Viterbi search adds ConnectionCosts.get(rightId of the previous morpheme, leftId of the next one) from a matrix built from the bundled dictionary (morph/Viterbi.java:459, :469-472, :530-539; end of text :389).
• Coda: the final consonant of a Hangul syllable. Korean particles depend on it: the subject particle is 가 after a vowel (학교가) and 이 after a consonant (학생이). mecab-ko-dic marks it as F (no coda) or T (coda) in its id definitions.

1. UserDictionary stores a right id per entry, chosen from the last character of the entry's line (for the usual one-word line, the noun's last syllable): RIGHT_ID_T 3535 (Hangul with a coda), RIGHT_ID_F 3534 (Hangul without), else RIGHT_ID 3533 (UserDictionary.java:94-103; constants and comments :40-45). Every entry has LEFT_ID 1781, cost -100000, and tag NNG (UserMorphData.java:23-26, :61-69).
2. Those are the ids of mecab-ko-dic 2.0.3. Since 10.0.0 nori is built from mecab-ko-dic-2.1.1-20180720 (nori/build.gradle:51), whose left-id.def/right-id.def assign the numbers differently:

1. The running binary agrees: bundled 학교 and 나무 (no coda) are 1780/3533 and bundled 학생 and 가방 (coda) 1780/3534, but the same nouns as user entries get 1781/3534 and 1781/3535.
2. Told that 학교 ends in a consonant, the search finds the subject particle 가 expensive and takes the verb 가 "go"; told that 학생 ends in a vowel, it reads 이 as the copula. KoreanAnalyzer's stop filter drops JKS but keeps VV and VCP (KoreanPartOfSpeechStopFilter.java:36-68), so 가 and 이 become index terms.

Cost of the rest of the path when the particle ends the text, conn(R, left) + wordCost + conn(right, end), on main:

Only the right ids cause the observed analysis effects: an independent reproducer's three particle and term tests pass with just the right ids corrected and are unchanged with just LEFT_ID corrected (the noun was at the start of the text in its probes); its separate id test still fails with only the right ids corrected, because LEFT_ID stays 1781. LEFT_ID is stale by the same definitions and belongs in the fix, but no probe isolates its effect with a morpheme before the noun, so that part rests on the definitions. The stale ids still index inside the cost matrix, so nothing throws.

• Who is exposed. Applications that load a nori user dictionary on 10.0.0 or later: the KoreanTokenizer constructors that take a UserDictionary (KoreanTokenizer.java:104-110, :121-136, :154-163), KoreanAnalyzer(userDict, ...) (KoreanAnalyzer.java:59-69), or the korean factory's userDictionary file (KoreanTokenizerFactory.java:44-57, :111-123). Downstream products whose nori plugins expose a user-dictionary setting inherit it. Without a user dictionary nothing changes. The classes are @lucene.experimental.
• What it takes. A user noun whose line ends with the noun itself (the usual one-word line, 학교). The id is picked from the last character of the whole line (UserDictionary.java:94), so a line that ends in whitespace or a comment (학교 , 학교 # memo) gets the fallback RIGHT_ID 3533, which in 2.1.1 happens to be the no-coda noun id: on such lines vowel-final nouns are read correctly (학교가 gives [학교], 깃허브가 있다 gives [깃허브, 있]), while consonant-final nouns stay affected (학생 : 이/VCP and [학생, 이]). That line-end behaviour is the sibling LFV-654.
• New words. Adding words the bundled dictionary has no entry for is a common reason for a user dictionary, and they are affected the same way (깃허브가 있다). A word the bundled dictionary has under another tag is too: 루씬 is a bundled proper noun (NNP), and as a user noun 루씬이 gives 이/VCP and [루씬, 이] instead of 루씬/NNP 이/JKS and [루씬].
• How often (grammatical probes only): over 300 nouns a verifier chose (150 with and 150 without a final consonant) in 8 sentence templates each, the particle tag changes in {N}가 있다 and {N}이 있다 for 150 of 150 nouns each, and KoreanAnalyzer indexes an extra 가 in {N}가 있다 for 150 of 150; 0 on 9.12.3 and 0 with the fix. Bare {N}가 and {N}이 give 가/VV and 이/VCP plus an extra term for every noun compared (147 and 150; the verifier counts only nouns whose bundled analysis is the noun as one NNG token); 되었다 and 왔다 give JKC, 중요하다 and 필요하다 give MM (tag only). An independent reproducer's 39 grammatical probes: 15 particle tags and 11 term lists differ.
• Unaffected in those templates: 을/를, 은/는, 의, 도, 에서, 로/으로, 와/과 함께, and {N}가/이 before 없다, 많다, 좋다, 크다, 아니다. In every printed probe the user noun's own term is the same on stock and with the fix.
• Effect. Wrong PartOfSpeechAttribute tags for the particle (sometimes for the next word too: 있/VA becomes 있/VX) and, with the default stop tags, an extra one-syllable term, at index and query time alike when both use this analyzer configuration. No exception, with assertions on or off.
• Scale. No threshold: every occurrence, in an affected context, of every user noun whose line ends with the noun (and of consonant-final nouns also when the line ends in whitespace or a comment). A fix changes the terms of fields analysed with a user dictionary, so those fields need a reindex.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[학교]> but was:<[학교, 가]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.ko.TestUserNounParticle.testUserNounKeepsSubjectParticle(TestUserNounParticle.java:41)

• The contract. The comments say what each constant is meant to be: "NNG right", "NNG right with hangul and a coda on the last char", "NNG right with hangul and no coda on the last char" (UserDictionary.java:40-45), "NNG left" (UserMorphData.java:25). The code picks between them by the coda of the line's last character, for a one-word line the noun's last syllable (:94-103), and the entry reports NNG (UserMorphData.java:61-69). The default constructors use "the system and unknown dictionaries shipped with Lucene" (KoreanTokenizer.java:97, :113), so the bundled 2.1.1 matrix is what these numbers are read against.
• The oracle. The tests compare with nori's own reading of the same noun from its bundled dictionary, not with an outside linguistic judgement; on 9.x, where the ids match the dictionary, the same tests pass.
• Tests. No upstream test pins either behaviour: TestUserDictionary checks lookup and segmentation, TestKoreanTokenizer.testUserDict has no particle after a user noun. The module's 87 upstream tests pass on stock and with the fix.
• Precedent. LUCENE-8603 (#9649, 7.7.0, under "Bug fixes": "Fix the inversion of right ids for additional nouns in the Korean user dictionary", CHANGES.txt:6910-6911) fixed these constants because "the ids were not in sync with the version of the dictionary that is used by the library" (@jimczi). Its values (53b05c5e7e8 on master; 7.7.0 shipped the branch_7x copy 0d6d12a0ebb) are today's, correct for 2.0.3.
• History. LUCENE-10416 (76c9fd4e38a, CHANGES 10.0.0, CHANGES.txt:2330-2331) changed the dictionary and updated the unknown-word NGRAM left id (1798 to 1801, UnknownDictionaryBuilder.java:31), but not UserDictionary or UserMorphData. Its discussion (#11452) covers reindexing only.

The strongest case for "not a bug" (argued by the devil's-advocate verifier and by the cross-model reviews; the reviews rated it "moderate", the verifier gave no rating): no javadoc promises that a user noun connects like a bundled noun; nori is @lucene.experimental; user words get a dominant cost of -100000, so connection costs look secondary; hard-coded ids cannot be guaranteed to fit an arbitrary custom system dictionary (the experimental constructor that takes a TokenInfoDictionary and ConnectionCosts, LUCENE-9253); and LUCENE-10416 was a deliberate upgrade.

Why it fails: the cost decides whether the user word wins, while the right id decides how the next morpheme is read, and that is exactly what changes. The comments name the contexts and the code selects them by coda. "Experimental" allows API changes, not wrong output. The custom-dictionary point argues for deriving the ids, not for keeping numbers that are wrong for the bundled dictionary every default constructor uses. LUCENE-8603 already treated this exact mismatch as a bug. Verdicts: devil's advocate real (high); the cross-model reviews real (high); adjudication confirmed (high).

One fix was built and validated with gradle in a scratch worktree at a9779ef3c193: the finder's values, with comments that name the definitions. Deriving the ids from the dictionary data is the sturdier design but was not built.

UserDictionary: RIGHT_ID 3533 -> 3532 (NNG,*,*), RIGHT_ID_T 3535 -> 3534 (NNG,*,T), RIGHT_ID_F 3534 -> 3533 (NNG,*,F); UserMorphData: LEFT_ID 1781 -> 1780 (NNG,*,*). 2 files, +10/-8. The new test's id check compares with the bundled dictionary, so a renumbering like this one fails the build instead of silently shifting the ids again (it checks two nouns, so it is a guard rather than a proof).

--- a/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/UserDictionary.java
+++ b/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/UserDictionary.java
-  // NNG right
-  private static final short RIGHT_ID = 3533;
-  // NNG right with hangul and a coda on the last char
-  private static final short RIGHT_ID_T = 3535;
-  // NNG right with hangul and no coda on the last char
-  private static final short RIGHT_ID_F = 3534;
+  // Right context ids of a plain NNG in the bundled dictionary (right-id.def of
+  // mecab-ko-dic-2.1.1-20180720); update them together with the dictionary.
+  // NNG right (NNG,*,*)
+  private static final short RIGHT_ID = 3532;
+  // NNG right with hangul and a coda on the last char (NNG,*,T)
+  private static final short RIGHT_ID_T = 3534;
+  // NNG right with hangul and no coda on the last char (NNG,*,F)
+  private static final short RIGHT_ID_F = 3533;
--- a/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/UserMorphData.java
+++ b/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/UserMorphData.java
-  // NNG left
-  private static final short LEFT_ID = 1781;
+  // NNG left (NNG,*,* in left-id.def of mecab-ko-dic-2.1.1-20180720, the bundled dictionary)
+  private static final short LEFT_ID = 1780;

Trade-offs

• Changes 10.x tokenization after user nouns: fields indexed with a user dictionary keep the old terms until reindexed; worth a CHANGES line and a decision on which release gets it (the 9.x backport of the dictionary upgrade was reverted over reindexing).
• The ids stay hard-coded: the fix moves them from the 2.0.3 layout to the bundled 2.1.1 one, so a user of the experimental custom-dictionary constructor whose dictionary keeps the 2.0.3 numbering gets fitting ids today and would not after the fix; a dictionary with yet another numbering fits neither.
• It also moves the fallback RIGHT_ID (lines that end in a Latin letter, a digit, whitespace or a comment) from 3533 to the generic 3532; from 3532 both 가 and 이 read as JKS (probe COST lines), but no test targets those lines.
• Cost: none (constants).
• Ship it with, or link it to, the sibling fix for the line-end coda bug (LFV-654), which chooses among these same three right ids.

What the checks validated

• Gradle, JDK 25, scratch worktree at a9779ef3c193: the new test fails 3 of 3 on stock (assertions on and off) and passes 3 of 3 with the fix; the new test with TestUserDictionary, TestKoreanTokenizer, TestKoreanAnalyzer and TestKoreanTokenizerFactory: 37 tests, 2 nightly skipped, 0 failed; the whole module, 14 suites and 90 tests (87 upstream + 3 new; 4 skipped: 3 nightly and TestKoreanNumberFilter.testLargeData), passes with assertions on and off; the nightly run of TestKoreanTokenizer, TestKoreanAnalyzer and the new test passes 27 of 27; :lucene:analysis:nori:check passes (checkGoogleJavaFormat, ecjLint, forbiddenApis, checkLicenses, tests); tidy changes nothing.
• Standalone runner with the fix: the new test 3 of 3 (assertions on and off), the finder's test 2 of 2, the independent reproducer's 4 of 4; every row of the token dump gives the bundled reading.
• Earlier, with the same values: 4 upstream nori suites run by the finder (34 tests started, 32 passed, 2 nightly aborted, 0 failed), and every verifier's tests (adjudication: 3 of 3 pass).
• Re-run after the fact check (the test helper's javadoc corrected, a comment-only change): tidy leaves the file unchanged; stock 3 of 3 fail with assertions on and off, the fix 3 of 3 pass, :lucene:analysis:nori:check with the fix passes (90 tests, 4 skipped); the standalone runner 3 of 3 fail on main and 3 of 3 pass with the fix.

Have the dictionary build read the plain-noun ids (NNG,*,*, NNG,*,T, NNG,*,F) from left-id.def/right-id.def and store them with the bundled dictionary, so the user dictionary uses the ids of whatever dictionary was built. An independent C++ re-implementation of nori looks the ids up this way; LUCENE-8816 (#9860) notes that MeCab can derive user-entry ids.

Trade-offs

Sturdier against the next upgrade, but a build or resource-format change. UserDictionary.open(Reader) is independent of the system dictionary, so serving custom system dictionaries would also need an API that ties the two.

The test's id check fails the build while the constants are stale.

Trade-offs

Reports the problem instead of fixing it; option A already includes it as a regression guard.

Method: differential probe plus a data-definition cross-check (the hunt's "semi-formal" tier), during a hunt unit that checked, bounded-exhaustively on the stock jar, that the CJK tokenizers' Viterbi output is the minimum-cost path of their own lattice (for nori that check found 0 mismatches), 2026-10-27, main a9779ef3c193.

• While modelling nori's user-dictionary costs, a differential probe (the same entry with and without trailing whitespace) gave different particle analyses. That led to the sibling bug LFV-654 (the coda read from the end of the line) and to this one.
• The finder dumped the context ids of bundled nouns on main and on 9.0.0 (main: 학교 1780/3533; 9.0.0: 1781/3534) and read left-id.def/right-id.def of mecab-ko-dic 2.1.1: the user constants match the 9.x numbering, not the bundled one.
• A public-API JUnit then compared the analysis of each noun from the bundled dictionary with the same noun as a one-line user entry. No solver or model checker produced the signal.
• Independent AI tools reproduced and judged it (see the checks).