Nori's DictionaryBuilder only asserts that rows sharing a left id have one part of speech, so a custom dictionary silently gives words another row's tag

Nori's compiled system dictionary stores a word's left part of speech in a table indexed by left context id (posDict), not per word: TokenInfoMorphData.getLeftPOS returns posDict[getLeftId(morphId)] (TokenInfoMorphData.java:73-76). When the public DictionaryBuilder writes a row, TokenInfoDictionaryEntryWriter.putEntry compares the row's part of speech and semantic class with what the slot already holds only through assert existing == null || existing.equals(fullPOSData) (TokenInfoDictionaryEntryWriter.java:100) and then overwrites the slot (:101). With assertions off, the JVM default and how the project's own regenerate task runs the builder, a source in which two rows share a left id but have different parts of speech builds without error, and every word on that id reports the part of speech of the row that sorts last by surface. In the repro the noun 가 (NNG) shares left id 1 with the topic particle 는 (JX): 가 reports JX, and KoreanPartOfSpeechStopFilter with DEFAULT_STOP_TAGS drops it (0 tokens instead of [가]); with -ea the build stops with a bare AssertionError. Kuromoji's builder throws IllegalArgumentException("Multiple entries found for leftID=") for the same case since LUCENE-8863; LUCENE-8904, "the Nori version" of that change, converted nori's neighbouring builder asserts but left this one. Only custom system dictionaries are affected (loaded through the @lucene.experimental custom-dictionary KoreanTokenizer constructor since 8.5.0; the release sweep reached older releases by putting the compiled files ahead of the bundled dictionary on the classpath): the bundled mecab-ko-dic 2.1.1 has no colliding left id, so the default KoreanTokenizer and KoreanAnalyzer are not. Every release from 7.4.0 to 10.6.0 (10.6.0 by reading) and main has it. The recommended fix replaces the assert with an IllegalArgumentException that compares only the part of speech, the one value the format stores.

• System dictionary source: MeCab-format CSV rows, 12 columns: surface, left id, right id, cost, part of speech, semantic class, coda flag, reading, type (*, Compound, Inflect, Preanalysis), first and last part of speech, expression (TokenInfoDictionaryEntryWriter.java:39-60). DictionaryBuilder.build(inputDir, outputDir, encoding, normalize) compiles it (DictionaryBuilder.java:23-39).
• Left context id: the id that the connection-cost matrix uses for the word's left side. Many words share one id.
• posDict: the compiled table that maps a left id to one part of speech. getLeftPOS reads it (TokenInfoMorphData.java:73-76), and getRightPOS returns the same value for plain and compound words and words with a single part of speech (:78-87).
• KoreanPartOfSpeechStopFilter: removes a token whose left part of speech is a stop tag (KoreanPartOfSpeechStopFilter.java:91-95); DEFAULT_STOP_TAGS contains the particles (JX among them, :52) and not the nouns.

1. The builder sorts all rows by surface with a stable sort (TokenInfoDictionaryBuilder.java:89-90) and writes them in that order. 가 (U+AC00) comes before 는 (U+B294).
2. For each row, putEntry builds fullPOSData = leftPOS.name() + "," + semantic class (TokenInfoDictionaryEntryWriter.java:98), checks it against the slot with assert existing == null || existing.equals(fullPOSData) (:100) and sets the slot (:101). Row 가 sets slot 1 to NNG,*; row 는 overwrites it with JX,*.
3. writePosDict writes one byte per left id, the part of speech only (:174-189).
4. At analysis time 가 reports JX, and the default stop tags remove it. The same tag also feeds the tokenizer's space penalty (Viterbi.java:344-346; read, not run).
5. With -ea, the assert fires inside DictionaryBuilder.build with no message. Lucene's gradle test task turns assertions on for about 75% of seeds unless -Ptests.asserts is given, so pass -Ptests.asserts=false to see the silent result.

• Who is exposed. Code that calls DictionaryBuilder (in the jar since 8.3.0) and loads the result through the Path constructors and the custom-dictionary KoreanTokenizer constructor (8.5.0+, @lucene.experimental), whose javadoc says it "provides an entry point for users that want to construct custom language models that can be used as input to {@link DictionaryBuilder}" (KoreanTokenizer.java:138-152). Before 8.5.0 there is no such constructor; the release sweep loaded the compiled files by putting them ahead of the bundled dictionary on the classpath. KoreanTokenizerFactory takes only a user dictionary, so factory-configured chains cannot load a custom system dictionary.
• Not exposed. The bundled mecab-ko-dic 2.1.1: 0 left ids with more than one part of speech over all 816,283 rows, so the default KoreanTokenizer and KoreanAnalyzer are clean. The user dictionary uses a fixed left id and tag (UserMorphData.java:25-26, :62-63).
• What the damage looks like. Silent with assertions off: every word on the shared id reports the tag of the row that sorts last by surface. With the default stop tags a noun disappears from the index (the repro) or a particle is indexed (the reverse case). With -ea, a bare AssertionError that names neither the id nor the rows.
• Why assertions are usually off. The JVM default, and the project's own regenerate task runs the builder through javaexec with only -Xmx1G (lucene/analysis/nori/build.gradle:34-42).
• How likely. Rare. In mecab-ko-dic's own tooling the left context is the part of speech and semantic class (its rewrite.def and left-id.def), so a collision usually needs hand-assigned or copied ids; what other dictionary compilers emit was not surveyed. One path through its conventions exists: Preanalysis rows whose first-POS column is * with semantic class 지명 map to the wildcard left id 1, while nori takes their tag from column 4, which can differ (two such rows, NNG and NNP: the NNG word reports NNP). In the bundled data all 7 such rows are NNP. No public report was found in the documented searches.
• Scale. Two rows reproduce it; deterministic; no size or timing factor.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError
       org.apache.lucene.analysis.ko.dict.TokenInfoDictionaryEntryWriter.putEntry(TokenInfoDictionaryEntryWriter.java:100)
       org.apache.lucene.analysis.morph.BinaryDictionaryWriter.put(BinaryDictionaryWriter.java:48)
       org.apache.lucene.analysis.ko.dict.TokenInfoDictionaryBuilder.buildDictionary(TokenInfoDictionaryBuilder.java:105)
       org.apache.lucene.analysis.ko.dict.TokenInfoDictionaryBuilder.build(TokenInfoDictionaryBuilder.java:56)
       org.apache.lucene.analysis.ko.dict.DictionaryBuilder.build(DictionaryBuilder.java:31)
       org.apache.lucene.analysis.ko.dict.TestNoriBuilderLeftIdPosCollision.testEachWordKeepsItsPartOfSpeech(TestNoriBuilderLeftIdPosCollision.java:44)

• Contract. No javadoc states the one-part-of-speech-per-left-id rule or what the builder does with a source that breaks it (DictionaryBuilder has a one-line javadoc, "Tool to build dictionaries."). The assert shows that the authors treat the collision as an error, not as something to merge. KoMorphData.getLeftPOS is documented to return the left part of speech "of specified word" (KoMorphData.java:34-39); here it returns another word's.
• Neighbouring checks are exceptions. In the same method the left-id limit and the POS-type checks throw IllegalArgumentException (TokenInfoDictionaryEntryWriter.java:130-135); kuromoji's writer throws for this very case (ja/dict/TokenInfoDictionaryEntryWriter.java:129-133, still marked // TODO: test me).
• History. LUCENE-8863 (4502065f036, 2019, #9906, 8.2.0) turned kuromoji's identical assert into the exception. LUCENE-8904 (2ac5fb668a8, 2019, #9947, 8.3.0): "It is the Nori version of @msokolov's #9906. ... 1) Improve exception handling ... Overall, it is the same as #9906." Its diff converted the left-id limit, POS-type, word-order, source-id-order and posDict-format asserts; this one stayed without discussion (other putEntry input asserts stayed too, so it was not singled out).
• Maintainers on builder asserts. In #9860 (LUCENE-8816, about kuromoji), @rmuir: "the various asserts in the current tooling should be fixed to be real if statements+exception messages" and "The use of assert was laziness on my part: we should treat it as technical debt and fix it"; @mocobeta asked to "change the all assertions to some Exceptions so that users can figure out the problem with their customized dictionary".
• Supported path. The custom-dictionary KoreanTokenizer constructor (KoreanTokenizer.java:138-152, @lucene.experimental) names DictionaryBuilder output as its input.
• Tests. No nori or kuromoji test builds a left-id collision; TestTokenInfoDictionary.testPutException covers only too few columns and too large an id (:92-99).

The strongest "intended" case (rated moderate by the cross-model reviews and the adjudication; the devil's-advocate verifier built it too and rejected it, without a rating): one part of speech per left id is a deliberate format choice. In LUCENE-8231 (#9278) @jimczi wrote "The semantic class + POS is unique per leftId so this could also save 1 byte in the binary dictionary" and then "added a POS dict that maps the left id and the part of speech tag". Read that way, the assert is a developer precondition on mecab-ko-dic-shaped data and a colliding custom source is garbage in, garbage out; DictionaryBuilder states no input contract, the custom-dictionary constructor is experimental, and the open kuromoji UniDic PR #12517 would relax kuromoji's check rather than enforce it.

Why it fails: the format choice is fine, but silently overwriting the slot is documented nowhere; the project turned the identical kuromoji line into an exception, set out to do the same for nori, and its maintainers called builder asserts technical debt; KoreanTokenizer documents custom dictionaries as an entry point. Relaxing the rule, as #12517 does for kuromoji, would mean storing the tag per entry, not overwriting it. Verdict: a defect (adjudication: confirmed, medium). What the counter-case gets right: the reach is narrow and the impact low, so it reads best as a missing input check, a hardening fix like LUCENE-8904.

Option A keeps the format and turns the silent overwrite into a clear build error; option B lifts the restriction instead and is the larger change. Do not copy kuromoji's check literally (option C): it also compares the semantic class, which the format drops. Whatever is chosen, extend TestTokenInfoDictionary.testPutException with the collision, as the validated diff does.

+11/-1 in TokenInfoDictionaryEntryWriter.java (the finder's logic, formatted with applyGoogleJavaFormat, its one-line comment rewritten as two lines), plus two tests in the existing TestTokenInfoDictionary: testPutException gains the 가 NNG / 는 JX collision with expectThrows(IllegalArgumentException.class, ...) and a check that the message names leftId=1; the new testPutSameLeftIdDifferentSemanticClass builds 서울 NNP,지명 and 철수 NNP,인명 on left id 1 and asserts NNP for both.

     String existing = posDict.get(leftId);
-    assert existing == null || existing.equals(fullPOSData);
+    // only the part of speech is stored per left id (see writePosDict), so all entries that share
+    // a left id must have the same part of speech; the semantic class may differ
+    if (existing != null && existing.startsWith(leftPOS.name() + ",") == false) {
+      throw new IllegalArgumentException(
+          "Multiple parts of speech found for leftId="
+              + leftId
+              + ": "
+              + existing
+              + " vs "
+              + fullPOSData);
+    }
     posDict.set(leftId, fullPOSData);

Trade-offs

• A custom source that built "successfully" before, with wrong tags, now fails to build with a message naming the id and both values: the intended behaviour change, worth a CHANGES entry.
• Keeps the one-part-of-speech-per-left-id restriction rather than lifting it; no format change, nothing on the analysis hot path (the check runs only while building).
• startsWith(leftPOS.name() + ",") is exact because tag names contain no comma; a reviewer may prefer to keep the POS.Tag per left id in a separate list.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: on stock, the repro and the two test changes (7 tests) fail 4 of 7 with assertions on (the 2 repro tests, the extended testPutException and the new testPutSameLeftIdDifferentSemanticClass, each a bare AssertionError at :100) and 3 of 7 with assertions off (the semantic-class test passes there); with the fix, 7 of 7 pass both ways.
• With the fix: the whole :lucene:analysis:nori:test, 14 suites, 90 tests, 0 failures, 4 skipped (3 @Nightly, 1 @Ignore), with assertions on and off; :lucene:analysis:nori:checkGoogleJavaFormat passes.
• The bundled mecab-ko-dic 2.1.1 rebuilt with the fix, -ea and -da: all 9 resource files byte-identical to the shipped ones.
• Earlier, with the finder's copy of the same logic: the independent reproducer's test 5 of 5, the cross-model reviewers' tests, upstream TestTokenInfoDictionary and TestUnknownDictionary 6 of 6; the release-sweep driver rejects the colliding source and still builds the control.

Write the left part of speech into each entry instead of the per-left-id table, so rows that share a left id may differ. Not run.

Trade-offs

Lifts the restriction, the direction the open kuromoji UniDic PR #12517 takes for kuromoji. But it changes the binary format (a format version bump and a rebuild of every dictionary), adds a byte to entries and touches TokenInfoMorphData, and no known nori dictionary needs it. Bigger than a bug fix; could follow option A.

if (existing != null && existing.equals(fullPOSData) == false) throw new IllegalArgumentException(...).

Trade-offs

Compares the semantic class too, which writePosDict discards, so it would reject rows with one tag and different semantic classes that the format stores correctly. Today's assert already does that with -ea: two NNP rows with semantic classes 지명 and 인명 fail with AssertionError under -ea and build correctly under -da, and testPutSameLeftIdDifferentSemanticClass fails on stock with assertions on.

A javadoc note on DictionaryBuilder that rows sharing a left id must share a part of speech.

Trade-offs

Changes nothing for a build without -ea, which stays silent.

Method: bounded-exhaustive translation validation, then a targeted JUnit (the hunt's "semi-formal" tier). A hunt unit, "CJK compiled-dictionary source round trip", lead 9.1 of the unit plan (builder asserts as the only guard on the source data), 2026-10-27.

• The finder rebuilt the bundled mecab-ko-dic 2.1.1 with -ea: no builder assert fired and the resources were byte-identical to the shipped ones.
• A round-trip harness compared the shipped nori resources with the source over all 816,283 lexicon rows on the stock jars: 0 left ids with more than one part of speech, so the bundled dictionary is unaffected.
• Reading how posDict is written showed that the only guard against such a collision is the assert at TokenInfoDictionaryEntryWriter.java:100; the finder then wrote the two-row JUnit (found 2026-10-27).
• Independent AI tools and reviews by another model family then reproduced, challenged and judged it (see the checks).