Nori's DictionaryBuilder rejects a left context id of 8192 or more but silently accepts such a right id, so a custom dictionary changes segmentation or crashes KoreanTokenizer

Nori's DictionaryBuilder compiles a custom Korean system dictionary from MeCab-format sources; the custom-dictionary KoreanTokenizer constructor is documented as the entry point for such dictionaries (@lucene.experimental). Each entry stores its left and its right context id shifted left by 2 in a 16-bit field (TokenInfoDictionaryEntryWriter.java:136-137), so each id gets a 14-bit field; the reader decodes it from a sign-extended short (TokenInfoMorphData.java:54, :59), so only 0 to 8191 round-trip. The builder rejects leftId >= 8192 with IllegalArgumentException (:130-132) but writes the right id unchecked (:137). Right ids 16384 to 24575 come back as id - 16384 (16385 becomes 1), which silently changes connection costs and segmentation (가나다 becomes [가나, 다] instead of [가, 나다] in the repro); right ids 8192 to 16383 and 24576 to 32767 come back as about 1.07 billion, and tokenizing text with the word throws IndexOutOfBoundsException from ConnectionCosts.get; a negative id has one of the two outcomes (-1 throws, -8193 becomes 8191). Word rows and unk.def rows go through the same writer. The bundled dictionary (3,822 right ids) and the default analyzer are not affected. Present since nori's first commit (7.4.0) and reproduced on every release tried up to 10.5.2 and main; a three-line IllegalArgumentException next to the left-id check fixes it, and the bundled dictionary rebuilds byte-identically with it.

• System dictionary: compiled by DictionaryBuilder.build(Path, Path, String, boolean) (ko/dict/DictionaryBuilder.java:24-28, public) from MeCab-format word rows (*.csv), unk.def, char.def and matrix.def; loaded with the Path constructors of TokenInfoDictionary, UnknownDictionary and ConnectionCosts and passed to the custom-dictionary KoreanTokenizer constructor, "an entry point for users that want to construct custom language models that can be used as input to {@link DictionaryBuilder}" (ko/KoreanTokenizer.java:138-152, @lucene.experimental).
• Context ids: matrix.def gives the cost of each (right id of a word, left id of the next word) pair. The tokenizer adds costs.get(lastRightID, leftID) between words (morph/Viterbi.java:471) and costs.get(lastRightID, 0) at the end of input (:389), and keeps the cheapest path.

1. putEntry parses both ids with Short.parseShort (TokenInfoDictionaryEntryWriter.java:63-64), so 32768 and above already fail (with NumberFormatException).
2. if (leftId >= ID_LIMIT) throw new IllegalArgumentException(...) (:130-132, ID_LIMIT = 8192 at :33); there is no such check for the right id.
3. buffer.putShort((short) (rightId << 2 | flags)) (:137): bits 0-13 of the id survive; bit 13 (8192) lands in the sign bit and bit 14 (16384) is cut off.
4. getRightId is buffer.getShort(morphId + 2) >>> 2 (TokenInfoMorphData.java:58-60) on the sign-extended short: 16385 reads back as 1, 8192 as 1,073,733,632, -1 as 1,073,741,823.
5. So 0 to 8191 round-trip; 8192 to 16383 are written intact and only the signed read loses them (an unsigned read, (getShort(...) & 0xffff) >>> 2, as kuromoji's reader does with its 3 flag bits at ja/dict/TokenInfoMorphData.java:72, :77, returns them exactly); 16384 and above lose bit 14 on write. A negative id behaves like its value modulo 16384 (-8193 like 8191, -1 like 16383).
6. ConnectionCosts.get reads buffer.getShort((backwardId * forwardSize + forwardId) * 2) (morph/ConnectionCosts.java:61-64) from a bounds-checked direct buffer: a wrapped id reads another id's costs; a huge id throws IndexOutOfBoundsException.
7. TokenInfoDictionaryWriter (:30) and UnknownDictionaryWriter (:35) both use this entry writer, so unk.def rows are affected too.

• Who is exposed. Builders of custom or retrained nori system dictionaries. Before 8.5.0 such a dictionary could only be used by putting its files on the classpath ahead of the nori jar; from 8.5.0 the custom-dictionary constructor is the public path. User dictionaries do not go through this writer (UserDictionary uses fixed right ids, UserDictionary.java:41-45).
• Default vs opt-in. Opt-in. The bundled dictionary has 3,822 right ids and 2,693 left ids (read from its ConnectionCosts.dat header), so the default KoreanTokenizer and KoreanAnalyzer never reach the bug.
• What the damage looks like. Right ids 16384 to 24575, and negative ids from -16384 to -8193 or from -32768 to -24577: the dictionary builds and loads, and every connection cost after the word comes from another id's row, so segmentation (and the indexed terms) can change with no error. Right ids 8192 to 16383 and 24576 to 32767, and negative ids from -8192 to -1 or from -24576 to -16385: the build still succeeds, and the failure comes later, wherever the tokenizer runs, as a bare IndexOutOfBoundsException that does not name the dictionary row.
• How likely. Rare, and contrived for the silent effect alone. MeCab context ids are dense (0 to N-1). A model with 8,193 to 16,384 right ids only crashes; a model big enough to reach the silent band also has ids in the crash band, so its user most likely sees the exception first. Larger MeCab models exist: an open kuromoji UniDic PR (#12517) shows Japanese context ids above 15,000. The documented searches found no prior report (2026-10-27, best effort).
• Scale and modes. One row reproduces it; assertions on and off behave the same (there is no assert either). The fix runs only at build time.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<16385> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.analysis.ko.dict.TestNoriBuilderRightIdLimit.testRightId16385(TestNoriBuilderRightIdLimit.java:48)

• The builder's own rule. Both ids are written as (short) (id << 2 | bits) (TokenInfoDictionaryEntryWriter.java:136-137) and decoded as getShort(...) >>> 2 (TokenInfoMorphData.java:54, :59); with that signed read only 0 to 8191 round-trip, and the builder enforces that limit (ID_LIMIT, :33) for the left id (:130-132). No javadoc, CHANGES or MIGRATE entry states an id range, and nothing says an id may wrap.
• Upstream tests. TestTokenInfoDictionary.testPutException (:92-99): // id too large expects newDictionary("HANGUL,8192,8192,1,NNG,*,*,*,*,*,*,*") to throw; with both ids at 8192 only the left check is exercised. testPut (:45-64) uses ids 5000 with a 1 1 matrix, so the tests rely on there being no check against matrix.def.
• What the committers said. #9860 (LUCENE-8816, kuromoji), @msokolov: "we restrict incoming leftID (and rightID) to be <4096 ... change the asserts there to be IllegalArgumentException so they are raised whenever the tool is run, since we would get garbage if this limit is exceeded, and (I think) nothing else will catch it"; @rmuir: "The use of assert was laziness on my part: we should treat it as technical debt and fix it"; @mocobeta: "change the all assertions to some Exceptions so that users can figure out the problem with their customized dictionary". #9947 (LUCENE-8904), which added nori's left-id exception, notes "left context ID == right context ID / Nori : can be different / Kuromoji : always same".
• History. Kuromoji's writer had assert leftId == rightId; assert leftId < 4096;, so its bound covered both ids. Nori's first commit (e851b89cbeb, LUCENE-8231) dropped the equality, because nori's ids differ, and kept only assert leftId < 8192; LUCENE-8904 (2ac5fb668a8 on master; backport 70854dc1efb in 8.3.0) turned that into today's exception and added nothing for the right id. Kuromoji today checks leftId != rightId before its limit (ja/dict/TokenInfoDictionaryEntryWriter.java:117-122).

The case for intended behaviour (the intended-behaviour check and the adjudication: weak to moderate; the cross-model fact check: moderate): DictionaryBuilder is a lightly validating, @lucene.experimental build tool for trusted input; it never checks ids against matrix.def; and in #9860 @mocobeta wrote "There is another possibility that users give large values to leftIDs (and rightIDs) in their customized dictionary by hand, however I don't think we should take care about that."

Why it fails: that remark is about whether to support larger ids, and the same comment asks for exceptions so users can find the problem in their dictionary. The limit here is the builder's own ID_LIMIT, the range its format reads back correctly, which only the builder can enforce and which it already enforces for the other id stored the same way. The outcome is not a clear error but a dictionary that silently means something else, or crashes later. What the case gets right: the reach is narrow. Also, 8192 is a limit of the signed read, not of the stored bits, so maintainers may prefer to support 8192 to 16383 with an unsigned read (the second fix option) instead of rejecting them; ids of 16384 and more and negative ids need the check either way.

Keep the fix to the builder's own limit (ID_LIMIT), or read the field unsigned (the second option). A check against the matrix.def dimensions is a separate, wider gap that the upstream testPut would trip. Side observation: a negative left id already fails the build, but with IndexOutOfBoundsException: Index -1 out of bounds for length 0 from the part-of-speech table (TokenInfoDictionaryEntryWriter.java:94-101), before the left check at :130 is reached; moving both checks right after the parse would give it a clear message too (not run).

Add the check after the left-id check in TokenInfoDictionaryEntryWriter.putEntry (:130-132), and two right-id-only cases to TestTokenInfoDictionary.testPutException. The finder's fix; the lower bound was added after the cross-model review.

    if (leftId >= ID_LIMIT) {
      throw new IllegalArgumentException("leftId >= " + ID_LIMIT + ": " + leftId);
    }
+   if (rightId < 0 || rightId >= ID_LIMIT) {
+     throw new IllegalArgumentException("rightId out of range [0, " + ID_LIMIT + "): " + rightId);
+   }

Trade-offs

• Build time only: no cost at analysis time.
• A dictionary in the silent band that seems to work today stops building; it was already compiled wrong.
• Covers word rows and unk.def rows (one writer). It does not check ids against matrix.def.
• It also rejects 8192 to 16383, which the stored field holds and only the signed read loses; the unsigned-read option would accept them instead.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: on stock the new test fails 4 of 6 (assertions on, seed 0B1D0710, and off, seed 0B1D0711) and the extended testPutException fails; with the fix, org.apache.lucene.analysis.ko.dict.*: 5 suites, 20 tests, 0 failures, assertions on and off; the whole :lucene:analysis:nori:test: 14 suites, 93 tests, 4 skipped, 0 failures, assertions on and off; checkGoogleJavaFormat passes.
• Standalone runner with the fix shadowed: 6 of 6 (-ea, -da); the probe rejects the 11 out-of-range ids from -8192 to 32767 with IllegalArgumentException and still builds 1 and 8191.
• Earlier: the bundled mecab-ko-dic 2.1.1 rebuilt with the fix is byte-identical (9 files); the finder's test 4 of 4, the independent reproduction's 7 of 7, the cross-model reviewers' tests, and the upstream TestTokenInfoDictionary + TestUnknownDictionary 6 of 6 pass.

Ids 8192 to 16383 are written intact; only the signed read loses them. Read (buffer.getShort(...) & 0xffff) >>> 2 in getLeftId and getRightId (TokenInfoMorphData.java:54, :59), as kuromoji's reader does with its 3 flag bits (ja/dict/TokenInfoMorphData.java:72, :77), and raise ID_LIMIT to 16384. Evaluated with the same Java expressions, not with Lucene: this read returns every id in [0, 16383] exactly and, below 8192, the same value as today, so existing dictionaries read back the same and the on-disk layout does not change. The first option's check is still needed with the new limit: 16384 and above lose bit 14 on write, and a negative id would read back as id mod 16384 (-1 as 16383). Not run in Lucene.

Trade-offs

Supports larger models instead of rejecting them, so it is closer to a feature. A dictionary that uses the new range is misread by older Lucene versions. The cost matrix is right ids x left ids shorts in one direct buffer (morph/ConnectionCosts.java:46-49), about 88 MB for 16,384 x 2,693. Ids beyond 16383 need a wider field and a DictionaryConstants.VERSION bump.

Not run.

Trade-offs

Catches more broken sources, but the upstream testPut (ids 5000 with a 1 1 matrix) would fail, and it is a separate, wider gap.

A bounds check in getRightId or ConnectionCosts.get. Not run.

Trade-offs

Adds work on the analysis hot path and cannot detect the silent band, where the decoded id (16385 -> 1) is valid.

Method: bounded-exhaustive translation validation on the stock jars, then code reading (the study's "semi-formal" tier). A hunt unit, "CJK compiled-dictionary source round trip", 2026-10-27.

• The harness compared every field of every entry of the shipped nori dictionary with its mecab-ko-dic 2.1.1 source row, with assertions on and off: 12,723,657 checks, clean apart from compound rows of another hunt bug and reading rows that are harmless by construction.
• Reading the encoding of each checked field showed the asymmetry: an explicit 8192 limit for the left id and none for the right id, packed the same way.
• The hypothesis was turned into a public-API JUnit that builds a tiny custom dictionary with DictionaryBuilder; independent AI tools then reproduced, reviewed and judged it (see the checks).