JapaneseNumberFilter and KoreanNumberFilter merge a keyword-protected word into the number before it

JapaneseNumberFilter and KoreanNumberFilter turn numbers written in kanji or hangul into digits by joining consecutive numeral tokens into one number. Their javadoc warns that they may also normalize words that only look like numbers, such as the name 京一, and says the filter "respects the KeywordAttribute, which can be used to prevent specific normalizations from happening". It checks the flag only on the token that starts a number: a protected word met while a number is being composed (typically right after a numeral token) is appended to the number. With 京一 protected, 社員の三京一 gives 30000000000000001 instead of 3, 京一; with 경일 protected, the Korean name 오경일 gives 50000000000000001. The terms and offsets are the same as without the marker, no error is raised, a term query or delete-by-term on the protected word misses the document, and a term query for the made-up number finds it. Both filters are opt-in; both have had the defect since they were added (Japanese 6.0.0, Korean 8.2.0), through 10.5.2 and main. The recommended fix: a keyword token ends the number, and the number keeps the attributes of its own tokens (which also fixes the known #10130).

• Number filter: JapaneseNumberFilter (kuromoji) and KoreanNumberFilter (nori) rewrite numbers written with kanji or hangul numerals into Arabic digits: 三千2百２十三 gives 3223 (a javadoc example). A number may span several tokens; the filter reads ahead and joins them. They do not use context or part of speech, so words made of numeral characters are also rewritten: the name 京一 (京 = 10^16, 一 = 1) becomes 10000000000000001, the noun 구조 ("structure"; 구 = 9, 조 = 10^12) becomes 9000000000000.
• Keyword marker: SetKeywordMarkerFilter (or another filter that sets KeywordAttribute) flags chosen words so that later keyword-aware filters leave them alone. Both number filters declare themselves keyword-aware for exactly the case above.

For the tokens 三, 京一 with 京一 protected:

1. JapaneseNumberFilter.java:159-161: if (keywordAttr.isKeyword()) { return true; } runs once, on 三, the token that would start a number. It is not a keyword, so a number starts (:177-179: numeralTerm = isNumeral(term)).
2. The loop (:181) reads the next token, 京一, which carries keyword=true (:189), and appends 三 to the buffer (:203).
3. :205-207 decide whether 京一 continues the number: numeralTerm = isNumeral(term) || isNumeralPunctuation(term);. 京一 is all numeral characters (isNumeral, :458-475), so it is appended too; a decimal point or thousands separator would continue the number the same way. The keyword flag is not read.
4. :211-225: 三京一 is normalized to 30000000000000001 and emitted with the offsets of both tokens; 京一 is gone.

KoreanNumberFilter has the same code here: keyword check :148-150, predicate :196. The number is emitted with the attributes left by the last read, and only the term and offsets are overwritten (:220-222 / :209-211): the look-ahead token's when another token follows, the last token's at the end of the stream. In 社員の三京一 the stream ends at 京一, so stock emits 30000000000000001 with 京一's keyword=true; in 이 구조는 튼튼하다 the look-ahead is 는, so the number has no flag and 는's part of speech.

• Who is exposed. Chains that use either number filter (SPI names japaneseNumber, koreanNumber) with a keyword marker in front of it. JapaneseAnalyzer and KoreanAnalyzer do not use the number filters.
• What it takes. A protected word met while a number is being composed: right after a numeral token, or after a decimal point or thousands separator inside a number (a keyword-marked separator is absorbed the same way). Executed examples: the name 오경일 (surname 오, given name 경일, split by the tokenizer without a user dictionary), 이 구조는 튼튼하다 and 社員の三京一 (a constructed phrase). The spaced Korean example needs the space token removed (by the SP stop filter of the upstream test chain, or by discardPunctuation=true); with punctuation kept and no stop filter the space ends the number. The marker matches whole tokens: protecting 千万 does nothing where the tokenizer emits 千 and 万. Protected words in other positions are unaffected.
• What the user sees. Nothing fails: the index and the queries get the terms the filter produces with no marker. TermQuery(body:京一) misses a document whose body is 社員の三京一 and finds one with 田中京一; deleteDocuments(body:京一) leaves the first one live; TermQuery(body:30000000000000001) finds it, although with the protection applied it would hold 3 and 京一. A search for the numeral (3) misses it too: the index holds no 3. An analyzed query goes through the same chain, so query text with the same numeral before the word still matches (inference). Other fields, other values and other documents are unaffected; CheckIndex is clean.
• How likely. Rare: it needs two opt-in pieces and a protected all-numeral word right after a numeral; nobody has reported it since 2015. Within that configuration it is deterministic and one document reproduces it.
• Downstream (read, not run). A downstream search server's documentation recommends the same nori chain as the upstream tests, and its keyword-marker filter can precede it.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<3[]> but was:<3[0000000000000001]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:689)
       org.apache.lucene.analysis.ja.TestJapaneseNumberFilterKeywordAfterNumeral.testKeywordTokenAfterNumeralIsKept(TestJapaneseNumberFilterKeywordAfterNumeral.java:33)

• Class javadoc (JapaneseNumberFilter.java:69-72, KoreanNumberFilter.java:69-72): "This filter may in some cases normalize tokens that are not numbers in their context. For example, is 田中京一 is a name and means Tanaka Kyōichi, but 京一 (Kyōichi) out of context can strictly speaking also represent the number 10000000000000001. This filter respects the KeywordAttribute, which can be used to prevent specific normalizations from happening."
• Core analysis docs (lucene/core/.../analysis/package-info.java:487-490): "Keyword-aware TokenStreams/-Filters skip modification of tokens that return true from this attribute's isKeyword() method."
• The documented limits do not include it. The javadoc lists 15,7 becomes 157 (be aware of this weakness) (:60), the punctuation requirement (:40-47), stacked tokens left untouched (:63-64) and attributes taken from the last composing token (:74-82), but not "keyword protection only for the first token of a number".
• Tests and history. testName (TestJapaneseNumberFilter.java:161-195, TestKoreanNumberFilter.java:178-219) protects 京一 / 경일 only after a non-numeral word. The keyword check and the predicate are unchanged since each filter's first commit (LUCENE-3922, LUCENE-8812); neither issue, no CHANGES entry and no comment limits the protection to the first token.
• Inconsistent stock behaviour, read per token. With 천만 protected, 천만 원 keeps 천만, while 삼천만 명 gives 30000000. (Read per run of numeral tokens, the devil's-advocate reading below, this is consistent.)

The strongest "intended" reading, built by the intended-behaviour checker and rated moderate by the cross-model reviewers: KeywordAttribute only says keyword-aware streams "can decide to modify a token based on the return value of isKeyword()" (KeywordAttribute.java:22-27); the filter composes logical numbers, so the whole run started by an unprotected token could be read as the unit of normalization, and keeping a standalone protected 천만 while composing 삼 + 천만 is then consistent first-token behaviour; and the literal reading costs something: with 천만 protected, stock turns 삼천만 명 into the right amount 30000000, the fix into 3, 천만.

Why it fails: once the filter declares that it respects the attribute, the package contract says keyword-aware filters skip modification of keyword tokens, which is a per-token rule; the "numeric context" or run reading is in no javadoc, comment, test, commit or issue, and the filter "does not use any part-of-speech information" (JapaneseNumberFilter.java:66-67); read per token, stock keeps the same protected word in one position and drops it in another; and the 천만 cost is about the fix direction, not evidence of intent. What the case gets right: a fix changes long-standing output for some real numbers, so documenting the limit is a fair alternative to offer; and it does not settle which attributes the split-off number should carry (see Suggested fix).

A+C, A and B were run through gradle in a scratch copy at a9779ef3c193 (A+C and A: both modules' check; B: the new tests); D and E are design alternatives that were not run. C on its own is the fix for the already reported #10130.

fixAC.diff, +12/-1 in each filter: fix A (below) plus the #10130 fix of the sibling candidate. It remembers the state of the last token appended to the number and the first token's position increment, and restores them before the term and offsets are set, so the split-off number carries its own part of speech, reading and position increment instead of the protected token's.

    State lastNumeralState = null;
    final int firstPositionIncrement = posIncrAttr.getPositionIncrement();
    ...
      lastNumeralState = captureState();
      moreTokens = input.incrementToken();
    ...
        numeralTerm = !keywordAttr.isKeyword() && (isNumeral(term) || isNumeralPunctuation(term));
    ...
      restoreState(lastNumeralState);
      posIncrAttr.setPositionIncrement(firstPositionIncrement);
    ...
      keywordAttr.setKeyword(false); // from fix A; redundant once the state is restored

Trade-offs

• Intended change: as for A: a protected word right after a numeral stays a word, and the numeral before it is normalized alone; the 천만 cost below applies too.
• C's own change: every composed number that is followed by another token now carries the part of speech, reading and other attributes of its own last token, and its first token's position increment, instead of the next token's. With no marker, stock gives the number in 이 구조는 튼튼하다 the part of speech of 는 (JX), A+C that of 구조 (NNG). That is the separate, already reported #10130; maintainers may prefer it in its own PR.
• Index-time output changes, so existing indexes keep the old terms until reindexed.

What the checks validated

• Gradle, scratch copy at a9779ef3c193 with the doc's tests added, JDK 25: :lucene:analysis:kuromoji:check and :lucene:analysis:nori:check pass (google-java-format, ecjLint, forbiddenApis and the tests: kuromoji 31 suites, 197 tests; nori 14 suites, 88 tests; 0 failed, 11 skipped by their own annotations; the doc's 3 tests among them).
• Standalone harness with the fix: the doc's tests 3/3 with -ea and -da.
• Token dumps: 이 구조는 튼튼하다 with 구조 protected gives 2 (+1) and 구조 (+2, keyword), the input's increments; 社員の三京一 gives 3 with 名詞-数 / サン and 京一 (keyword).
• Not run for A+C: the 10-iteration and assertions-off gradle runs and the finder's and reproducer's suites that A had.

fixA.diff, +4/-1 in each filter (two code lines and two comments). The first line stops the composition at a keyword token, which is then emitted unchanged on the next call (it is already the captured look-ahead token). The second line is safe because a number starts only at a non-keyword token (:159), and with the first line every continuation token is non-keyword too, so a composed number is never a keyword.

      if (moreTokens) {
        term = termAttr.toString();
        // a keyword token is not part of the number: it ends the number and is emitted unchanged
        numeralTerm = !keywordAttr.isKeyword() && (isNumeral(term) || isNumeralPunctuation(term));
      }
    ...
      offsetAttr.setOffset(startOffset, endOffset);
      // the number is made of non-keyword tokens only; do not keep the look-ahead token's flag
      keywordAttr.setKeyword(false);

Trade-offs

• Not fixed by A: the split-off number keeps the protected token's other attributes (the #10130 mechanism; A overwrites only term, offsets and the flag), among them its position increment, part of speech and reading: 이 구조는 튼튼하다 gives 2 with position increment 2 instead of 1 (increments 2, 2 instead of 1, 2, so the positions after it shift by one), and 社員の三京一 gives 3 with 京一's part of speech and reading キョウイチ. The doc's tests pass with A because they do not assert these attributes.
• Intended change: a protected word right after a numeral stays a word, and the numeral before it is normalized alone (오경일 gives 5, 경일; 이 구조는 gives 2, 구조, ...).
• Cost: a protected word that is also a real part of a number splits that number: with 천만 protected, 삼천만 명 gives 3, 천만 instead of 30000000, and 일천만 원 gives 1, 천만. The user asked for 천만 not to be normalized, and stock already keeps it in 천만 원.
• Index-time output changes, so existing indexes keep the old terms until reindexed.
• One extra boolean check per continuation token.

What the checks validated

• Gradle, scratch worktree at a9779ef3c193, JDK 25: the 3 new tests fail on stock and pass with the fix.
• :lucene:analysis:kuromoji:check and :lucene:analysis:nori:check pass (google-java-format, ecjLint, forbiddenApis and the tests: kuromoji 31 suites, 197 tests; nori 14 suites, 88 tests; 0 failed, 11 skipped by their own annotations).
• The number-filter suites also pass with -Ptests.asserts=false (27 + 24 tests) and with -Ptests.iters=10 (270 + 240 tests).
• Standalone harness with the fix: the doc's tests 3/3 (-ea and -da), the finder's 2/2, the independent reproducer's 8/8.

The first line of fix A, without clearing the flag.

        numeralTerm = !keywordAttr.isKeyword() && (isNumeral(term) || isNumeralPunctuation(term));

Trade-offs

Terms and offsets are right, but the number before the protected word inherits keyword=true from it, and its part of speech and position increment (the look-ahead state, the mechanism of the known #10130), so a later keyword-aware filter would skip the number. Fine only together with a #10130 fix, which is A+C.

What the checks validated

The finder's 2/2 and the reproducer's 8/8 pass, and the upstream TestJapaneseNumberFilter and TestKoreanNumberFilter pass 800 of 800 (20 iterations, in the hunt); the doc's tests fail 3 of 3 on the flag: keywordAtt 2 term=3 expected:<false> but was:<true> (gradle). The finder's model check with this change finds 0 violations of its keyword invariant in 531,440 streams per filter; that invariant checks terms and offsets, not the flag.

Say in the javadoc that KeywordAttribute is honoured only on a token that starts a number.

Trade-offs

No output change, but the javadoc's own use case, protecting a name, stays broken after a numeral (오경일). Not run (no code).

三京一 would stay 三, 京一.

Trade-offs

Needs buffering of the run and changes more output; mentioned by the reproducer as the other possible semantics. Not run.

Method: bounded-exhaustive model checking of the real classes (a hunt unit, CJK number filters, 2026-10-26, main ea57360912ed; the hunt's "semi-formal" tier: exhaustive checking of the real jar against an exact oracle, no model checker or prover).

• The finder, an AI tool, wrote a checker (TokenStreamModelCheck.java) that builds every token stream of length 1 to 6 over 9 abstract token kinds (a digit, an exponent, a decimal point, a word, a stacked synonym, a keyword-marked numeral, a numeral after a removed token, and two decompounded forms) that does not start with the stacked synonym (531,440 streams), and runs it as a canned stream through the stock JapaneseNumberFilter and KoreanNumberFilter.
• Six invariants, each tied in the unit's model notes to a javadoc statement: no input token vanishes, a number holds no text from outside its span, its attributes come from inside its span, positions, position length 1, and KW: a keyword-marked token is emitted with its own term and offsets (from the class javadoc's keyword sentence). KW failed on 103,089 of 531,440 streams per filter, shortest dk (a digit, then a keyword-marked numeral); with the one-line predicate change, KW fails on 0.
• The other invariant failures belong to two already reported defects found in the same unit (#10130; #11398 / #11397 / #11395); each of the three fixes clears exactly its own invariants, and all three together satisfy all six.
• The finder reduced the KW counterexample to the upstream testName analyzer and wrote the JUnit pair; the reproducer later found the natural Korean name 오경일, and the devil's advocate 이 구조는 튼튼하다.