nori gives the parts of a decompounded word offsets counted from the end of the word, so parts that do not cover the whole word point at the wrong characters

KoreanTokenizer (nori) splits a COMPOUND dictionary entry into parts in DISCARD mode (the default, used by KoreanAnalyzer) and MIXED mode. Each part's text is cut from the start of the word using the stored part lengths (UserMorphData.java:85, TokenInfoMorphData.java:124), but its offsets are counted back from the end of the word (Viterbi.java:264, :267-292). The two agree only when the parts add up to the whole word. When they add up to less, every part's offsets move right by the uncovered length, onto other characters of the same word: the user-dictionary entry 세종시 세종 gives 세종 [1,3) = '종시', and the bundled 픽토그램 (parts 픽토 + 그) gives 픽토 [1,3) = '토그' and 그 [3,4) = '램'. Lucene accepts both inputs: UserDictionary rejects only a segmentation longer than the word (UserDictionary.java:114), and 7 of the 160,029 bundled compounds have parts that do not add up to the word. Highlights, snippets and stored offsets mark the wrong characters; terms and positions are unchanged, and so were matches and scores in the tested setups. Present since nori's first commit: it fails on every release tested from 7.4.0 through 10.6.0 and on main. Counting the offsets from the start of the word fixes it; rejecting such user entries, as the Japanese tokenizer does, and repairing the 7 bundled rows are complementary.

// Viterbi.java, backtrace: the whole word is [backWordPos, backWordPos + length)   (:238-247)
int endOffset = backWordPos + length;                       // :264  end of the WHOLE word
for (int i = morphemes.length - 1; i >= 0; i--) {           // :267  last part first
  ...
  if (token.getPOSType() == POS.Type.COMPOUND) {
    compoundToken = new DecompoundToken(morpheme.posTag(), morpheme.surfaceForm(),
        endOffset - morpheme.surfaceForm().length(),        // :276
        endOffset, backType);                               // :277
  } else { /* INFLECT parts: the whole token's offsets */ }
  ...
  endOffset -= morpheme.surfaceForm().length();             // :292
}

// UserMorphData.getMorphemes: part text cut from the START of the word   (:85)
morphemes[i] = new Morpheme(POS.Tag.NNG, new String(surfaceForm, off + offset, segs[i]));
// TokenInfoMorphData.getMorphemes, bundled dictionary: the same left cut  (:124)
form = new String(surfaceForm, off + surfaceOffset, formLen);

With part lengths summing to C and the word length L, the parts' offsets are shifted right by L - C against their text. For 세종시 세종, L = 3 and C = 2: 세종 gets [3 - 2, 3) = [1,3). In DISCARD mode '시' is in no token; MIXED mode emits 세종시 [0,3) and 세종 [1,3). KoreanTokenizer emits the offsets as given (KoreanTokenizer.java:236).

Nothing rejects such input: UserDictionary throws only if (offset > token.length()) (dict/UserDictionary.java:114), and the dictionary builder asserts only compoundOffset <= entry[0].length() (dict/TokenInfoDictionaryEntryWriter.java:161). The Japanese tokenizer's UserDictionary rejects any segmentation that does not concatenate to the word (ja/dict/UserDictionary.java:114, LUCENE-8933).

Found by scanning all 816,283 entries (160,029 compounds) of the bundled dictionary. Offsets are relative to the word's start, as emitted on main by the default KoreanAnalyzer inside ordinary sentences:

In the last three rows a part is missing in the middle of the stored row, so even the part texts are wrong (랑, 로, 크노, 파); only repairing the data fixes those. In a sentence the shift is the same: in '도쿄 올림픽 개막식의 픽토그램 공연이 화제가 되었다.' 픽토 is at [13,15), covering '토그'. Since 10.0.0 일로역 on its own is tokenized as 일 + 로역, but inside '일로역에 도착했다.' it still hits.

• Default settings. The 7 bundled words need no configuration. Each was hit in every sentence tested, also inside ordinary sentences with particles (all 7 on main and 10.5.2). Whether a word is hit depends on the tokenizer choosing its stored compound in that context: since 10.0.0, 일로역 on its own is split as 일 [0,1) + 로역 [1,3) and is not hit, while '일로역에 도착했다.' still hits.
• User dictionaries. Any entry whose segmentation is shorter than the word, for example to keep only the first part of a name, is accepted and misplaces the kept part.
• What goes wrong. Highlights and snippets mark characters 1-2 positions to the right (bundled words) or shifted by the uncovered length (user entries), always inside the same word. Term vectors and postings store the same shifted offsets.
• What does not change. Terms, positions and position lengths: only the offset arithmetic is wrong. In a check with 148 tokens, 88 of them misplaced (assertions on and off), all misplaced offsets stayed inside their own word, IndexWriter accepted them (they never went backwards), CheckIndex was clean and 46 phrase queries matched their documents. Each per-term span of the unified highlighter stayed inside its own word (term 픽토 gives 픽토그램); a phrase span covers the matched words, as usual, but starts one character late (phrase 픽토그램 gives 픽토그램, phrase '오늘 픽토그램 소식' gives 오늘 픽토그램 소식). Only the unified highlighter was run.
• Not affected: DecompoundMode NONE, which does not split compounds.
• How likely. Narrow: 7 of 160,029 bundled compounds, or a user entry of this shape. No remark on the misplaced offset was found since 2018, although it is visible in output posted on #9570.

The outcome does not depend on the seed: the test's only random choice is the attribute factory (newAttributeFactory() picks one of three, BaseTokenStreamTestCase.java:1543-1559), and run once with each of the three it fails with the same message, assertions on and off. It belongs in lucene/analysis/nori/src/test/org/apache/lucene/analysis/ko/ and passes :lucene:analysis:nori:check -x test (formatting and static checks; that task runs no tests). It uses the user-dictionary trigger because that one does not depend on the bundled data; the bundled words fail the same way (픽토그램 gives 픽토 [1,3) and 그 [3,4)), but their correct parts are themselves in question.

Regression test: this same test. It pins the recommended fix. If maintainers choose to reject such entries instead, UserDictionary.open throws and the test becomes an expectThrows(IllegalArgumentException.class, ...).

TestCompoundPartOffsets > testSegmentationShorterThanWord FAILED
    java.lang.AssertionError: startOffset 0 term=세종 expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:7B056883CFD98131]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.test_framework@12.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:264)
        ...
        at org.apache.lucene.analysis.ko.TestCompoundPartOffsets.testSegmentationShorterThanWord(TestCompoundPartOffsets.java:35)
:lucene:analysis:nori:test (FAILURE): 1 test, 1  failure

Assertions off (-Ptests.asserts=false): the same failure. Release jars 10.6.0 and 9.12.3: the same failure.

• Contract. OffsetAttribute.startOffset(): "the position of the first character corresponding to this token in the source text"; endOffset(): "one greater than the position of the last character corresponding to this token" (OffsetAttribute.java:23-50). For COMPOUND parts nori computes a separate sub-span per part, and the characters corresponding to a part are the ones its text was cut from.
• Tests. testPartOfSpeechWithCompound expects 가락지 [0,3) and 나물 [3,5) (TestKoreanTokenizer.java:246-252). TestTokenInfoDictionary.java:168-183 checks that each COMPOUND part equals the next left-to-right substring of the word.
• Javadoc. UserDictionary presents an entry as a split of the word: "custom nouns (세종) or compounds (세종시 세종 시)" (UserDictionary.java:33-34).
• History. Right-anchored offsets, left-cut text and the longer-only check all came with nori's first commit (LUCENE-8231). The saved design thread (#9278) gives no rationale for counting part offsets from the word's end. There Jim Ferenczi wrote "For compound nouns I serialize the segmentations with the term but we could just use offset from the surface form", which treats the parts as pieces of the word; a later patch there "fixes an issue with offsets of the compound nouns", without detail. The introducing commit's test userdict.txt has the entry 세종시 세종 시, which covers the word.
• Precedent. For the Japanese tokenizer, Tomoko Uchida wrote on #9976 (LUCENE-8933) that "the sum of length of segments are expected to be exactly same to the length of corresponding surface form", and that such entries should be rejected "even if this does not cause any exceptions". That thread checked nori only with an entry without segmentation.
• The right contract to state. "A COMPOUND part's offsets must cover the characters its text was cut from", not "a token's text equals its offset slice": INFLECT parts carry the whole word's offsets by design (TestKoreanTokenizer.java:311-317: 감싸이 [0,3) and 어 [0,3) for 감싸여).

The strongest case for "intended" or "misuse": nori tolerates parts that do not cover the word on purpose (the <= in TestTokenInfoDictionary.java:183 and in the dictionary builder since 2018); the 7 bundled rows are errors in the source data (Jim Ferenczi called #9570's 그레이맨 case one of the "bugs due to invalid rules in the mecab-ko-dic"); a shorter user segmentation is arguably a misconfiguration; KoreanTokenizer is @lucene.experimental; and it is excluded from TestRandomChains with @IgnoreRandomChains(reason = "LUCENE-10359: fails with incorrect offsets") (KoreanTokenizer.java:54), which the cross-model intended lens called "the strongest documented warning supporting a works-as-designed defense".

Why it fails: the tolerance confirms that parts are left-anchored text, yet nothing documents or tests offsets counted from the word's end; the annotation points to #11395, a different defect (its failing chain had no Korean text, and Robert Muir and Tomoko Uchida traced it to KoreanNumberFilter on that thread); Lucene accepts both inputs without a warning; the bundled case needs only default settings. Even if dropping a suffix were deliberate, the kept part's offsets must still point at its own text. Verdict: a defect. The adjudication kept medium confidence because maintainers may prefer repairing the data and rejecting such entries over changing Viterbi.

Validated with gradle in a clean scratch worktree of 5adff78bb168 (OpenJDK 25.0.3, seed DEADBEEF, 2026-11-05): the repro fails stock with assertions on and off and passes with fix A, on and off; with fix A the whole nori test suite passes (88 tests including the repro, 4 skipped) and so does :lucene:analysis:nori:check -x test. With fix B the nori suite passes (87 tests without the repro, 4 skipped) and the repro fails at UserDictionary.open, as B intends.

A fixes the offsets for every input. B and C are hardening: B stops new bad user entries, C repairs the data, which is the only way to fix the wrong part texts of 안지랑역, 일로역 and 테크노파크역. Maintainers may prefer B + C alone. The cross-model fact check would lead with B + C, with A as a consistency repair, because A keeps the 3 wrong part texts; both readings are defensible. A touches the same loop as the open PR #14969 (label Stale, its four commits from 2025-07-20), which keeps the offset formula.

Sum the part lengths and end the last part at the end of the covered prefix instead of the end of the word. Offsets of parts that cover the word are unchanged (the sum equals length); INFLECT parts keep the whole token's offsets.

--- a/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/Viterbi.java
+++ b/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/Viterbi.java
@@ -261,7 +261,13 @@ final class Viterbi
               System.out.println("    add token=" + pending.get(pending.size() - 1));
             }
           } else {
-            int endOffset = backWordPos + length;
+            // The parts' text is cut from the start of the word (getMorphemes), so their offsets
+            // must be counted from the start too, also when the parts do not cover the whole word.
+            int coveredLength = 0;
+            for (KoMorphData.Morpheme morpheme : morphemes) {
+              coveredLength += morpheme.surfaceForm().length();
+            }
+            int endOffset = backWordPos + Math.min(length, coveredLength);
             int posLen = 0;
             // decompose the compound
             for (int i = morphemes.length - 1; i >= 0; i--) {

Trade-offs

+7/-1 lines (two of the added lines are a comment), no format or API change. In DISCARD mode the uncovered tail of the word stays in no token. A makes each part's offsets match the text nori emits, not a linguistically correct split: for 안지랑역, 일로역 and 테크노파크역 the part texts themselves are wrong (랑, 로, 크노, 파) and only C fixes them. A does not guard against parts longer than the word (then min keeps the old end anchor); such data cannot be loaded today (UserDictionary.java:114 rejects it, TokenInfoDictionaryEntryWriter.java:161 asserts against it) and the bundled scan found none.

What the checks validated

Yes: repro on and off, the whole nori suite (88 tests, 4 skipped) and :lucene:analysis:nori:check -x test pass (gradle, 5adff78bb168, 2026-11-05).

As the Japanese UserDictionary does since LUCENE-8933: throw when the part lengths do not add up to the word, not only when they exceed it (UserDictionary.java:114).

--- a/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/UserDictionary.java
+++ b/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/UserDictionary.java
@@ -111,11 +111,11 @@ public final class UserDictionary implements Dictionary<UserMorphData> {
           length[i - 1] = splits[i].length();
           offset += splits[i].length();
         }
-        if (offset > token.length()) {
+        if (offset != token.length()) {
           throw new IllegalArgumentException(
               "Illegal user dictionary entry "
                   + entry
-                  + " - the segmentation is bigger than the surface form ("
+                  + " - the segmentation does not match the surface form ("
                   + token
                   + ")");
         }

Trade-offs

Existing user dictionaries with such entries stop loading: a behaviour change that needs a CHANGES entry. Does nothing for the bundled words.

What the checks validated

Partly: the nori suite passes (87 tests without the repro, 4 skipped); the repro then throws IllegalArgumentException: Illegal user dictionary entry 세종시 세종 - the segmentation does not match the surface form (세종시).

Fix the stored parts so they add up to the word, or have the dictionary builder store no parts for a COMPOUND whose parts do not cover the word; then make the builder's assertion (TokenInfoDictionaryEntryWriter.java:161) and TestTokenInfoDictionary.java:183 require equality.

Trade-offs

Needs regenerating the bundled dictionary. The only option that also fixes the wrong part texts.

What the checks validated

Not run.

Method: semi-formal (bounded-exhaustive translation validation of the stock analysis jars with a twin-run provenance oracle), on main a9779ef3c193, 2026-10-26.

• The sweep ran every input of up to 6 symbols per component over 136 component and char-filter configurations, 74,769,564 checks. Its check that a tokenizer's token slices its own text failed only for the Korean user-dictionary configuration with a shorter segmentation.
• An exhaustive scan of all 816,283 bundled nori entries found the 7 COMPOUND entries, of 160,029, whose parts do not cover the word, and a script traced them to their source rows.
• No solver or model checker produced the signal. An independent reproducer, reviews by another model family (OpenAI GPT-5.6), a devil's-advocate verifier and an adjudication then checked the claim.