KatakanaRomanizer.romanize(CharsRef) ignores CharsRef.offset and romanizes the start of the backing array

KatakanaRomanizer.getInstance().romanize(CharsRef input) turns katakana into the romaji a user would type, for Japanese auto-completion. A CharsRef is "a slice (offset + length) into an existing char[]". romanize walks a position from 0 to input.length but uses it as an absolute index into input.chars, without adding input.offset, in the keystroke lookup and in the copy of the unmatched rest. So a slice that does not start at index 0 is romanized from chars[0, length) instead of chars[offset, offset + length): the slice トウキョウ at offset 1 of アトウキョウ gives [atoukyo] instead of [toukyou], with no exception, assertions on or off.

The bug is latent inside Lucene: the only in-tree production caller, JapaneseCompletionFilter, passes a CharsRefBuilder.get(), whose offset is always 0, so the filter, JapaneseCompletionAnalyzer and indexes are unaffected. Only code that calls the public romanizer directly with a window into a larger array (a reused buffer, or CharsRef.subSequence) gets wrong romaji. It has been there since the class was created (LUCENE-10102, first released in 9.0.0) and affects every release to 10.6.0 and main; adding input.offset at the two sites fixes it.

1. The input is a slice. CharsRef "Represents char[], as a slice (offset + length) into an existing char[]" (CharsRef.java:23). The public constructor CharsRef(char[] chars, int offset, int length) (:51-56) and subSequence (:146-150, new CharsRef(chars, offset + start, end - start)) make slices with a non-zero offset.
2. The entry check reads the slice correctly. romanize starts with assert CharSequenceUtils.isKatakanaOrHWAlphabets(input) (KatakanaRomanizer.java:104), which reads through CharSequence.charAt (CharSequenceUtils.java:45-53), and CharsRef.charAt returns chars[offset + index] (CharsRef.java:139-143).
3. The body reads from index 0. pos starts at 0 (:107) and runs to input.length (:108). Each step calls longestKeystrokeMatch(input, pos) (:112), which builds the lookup key as new CharsRef(input.chars, inputOffset, len) (:173) instead of input.offset + inputOffset. When no keystroke matches, the loop stops (:113-115) and the rest is copied as output.append(input.chars, pos, input.length - pos) (:165) instead of input.offset + pos.
4. Result. For offset 0 both readings agree; for any other offset the method processes chars[0, length). Every index stays inside the array (a valid CharsRef has offset + length <= chars.length), so there is no exception and no out-of-bounds read. The method is inconsistent with itself: the assert checks one range and the body reads another, and :150 does honour the offset of the map's own refs.
5. Why Lucene never sees it. The only in-tree production caller is JapaneseCompletionFilter.java:228, romanize(pdgReading.get()), where pdgReading is a CharsRefBuilder (:152) and CharsRefBuilder.get() asserts ref.offset == 0 (CharsRefBuilder.java:132-135). The other in-tree calls, in TestKatakanaRomanizer, all pass new CharsRef(String).

In the first four rows the output is exactly what romanize gives for the first length characters of the array; in the last, :165 copies the unmatched A from outside the slice, even with assertions on, because the entry assert checks only the slice. The same slice content at offset 0 gives the expected value. subSequence is declared to return CharSequence (CharsRef.java:146), so the executed tests cast it to CharsRef. In a bounded-exhaustive check over every katakana string of length 1 to 3 (U+30A0 to U+30FF) with the suffixes "", "k" and "sy", 2,682,141 of 2,682,144 inputs romanize differently when sliced at offset 1 of an array padded with ア (the 3 others are ア, アア and アアア, which the padding matches); over random buffers and slices, 79% to 90% differ.

• Silent. Romaji of the wrong characters, or of fewer characters, with no exception, assertions on or off. Characters from before the slice can be copied verbatim: the slice スシ of トAスシ gives [toA].
• Who is exposed. Code that calls the public romanizer directly with a CharsRef whose offset is not 0: new CharsRef(buf, off, len) over a reused buffer, or CharsRef.subSequence(start, end) with start > 0. Every such call reads the wrong window, so the result is usually wrong (2,682,141 of 2,682,144 sliced inputs in the bounded-exhaustive check, which pads with ア; 79% to 90% of random slices).
• Not affected. JapaneseCompletionFilter, JapaneseCompletionAnalyzer, JapaneseCompletionFilterFactory and any analysis chain built on them: the filter's call always has offset 0, so analysis output and indexes are unchanged in every release. Stock uses of the filter, in Lucene or in downstream engines, go through that same offset-0 call; no public direct caller with another offset was found.
• Likelihood: rare. The package javadoc reads "Utilities for JapaneseCompletionFilter" (package-info.java:18); a code search found no outside direct caller other than two test harnesses that pass new CharsRef(String), and nothing has been reported since 9.0.0 (2021).

The test sits in org.apache.lucene.analysis.ja.completion only to match the upstream test layout; it uses only public classes and methods (CharsRef is tagged @lucene.internal). It has no randomness and fails the same way with seed DEADBEEF, assertions on and off, through gradle and the standalone runner.

The same file fails unchanged on the 10.6.0, 10.5.2 and 9.12.3 jars (assertions on). It does not compile on 9.0.0, where LuceneTestCase was still in org.apache.lucene.util (it moved in 9.1.0); 9.0.0 is covered by the history probe, a main() program on the public API.

Regression test: this same test, as its own class or as one more method in the existing TestKatakanaRomanizer.

TestKatakanaRomanizerCharsRefOffset > testRomanizeSliceAtNonZeroOffset FAILED
    java.lang.AssertionError: expected:<[toukyou]> but was:<[atoukyo]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:E60AB4631A1BC6B9]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.analysis.ja.completion.TestKatakanaRomanizerCharsRefOffset.testRomanizeSliceAtNonZeroOffset(TestKatakanaRomanizerCharsRefOffset.java:28)

• The contract is a slice. CharsRef.java:23 documents (offset + length). The romanize javadoc ("Translates a sequence of katakana to romaji. An input can produce multiple outputs ...", KatakanaRomanizer.java:99-102) states no offset-0 precondition, and nothing asserts one.
• The API is public and exported. KatakanaRomanizer (:39), getInstance() (:90) and romanize (:103) are public, with no @lucene.internal or @lucene.experimental tag, in a package that module-info.java:24 exports. The export dates from 9.1.0: 9.0.0 has no module-info.java, though the class was already public there. The input type CharsRef itself is tagged @lucene.internal (CharsRef.java:26; see the strongest counter-case below).
• The method contradicts itself. Its assert (:104) reads the slice through charAt, which adds the offset; :150 honours cref.offset; only :165 and :173 drop input.offset.
• Precedent. Lucene treats an ignored *Ref.offset as a bug even when no in-tree caller is affected: 1b671cfba3a (2010, "fix nasty bug (doesnt affect any codecs today): respect bytesref offset"), LUCENE-4423 (7145779e4dc, "DocumentStoredFieldVisitor.binaryField ignores offset and length"), LUCENE-5339 (d815a3608b2, "fix 'ignores IntsRef.offset bug' in TaxoFacetCounts"), LUCENE-10052 (88588e3deaa, #258, "FSTTermsReader.IntersectEnum was illegally ignoring BytesRef.offset").
• Nothing relies on offset 0. TestKatakanaRomanizer uses only new CharsRef(String); the review of #297 discusses token start/end offsets and CharSequence signatures, never CharsRef.offset.

The strongest case for "intended" (the refute reviewer, the cross-model review and the intended-behaviour check): CharsRef is @lucene.internal (CharsRef.java:26); the package is documented as "Utilities for JapaneseCompletionFilter"; the class was package-private in the PR's first commit and became public on the PR branch, which the filter in another package needed anyway, so public visibility alone does not prove it was meant for end users; and the only production caller passes offset 0. So the class may be filter plumbing that never needs other offsets.

Why it fails: @lucene.internal warns about API stability, it does not let a method ignore the documented slice fields; being plumbing does not make a wrong result right when the method takes a CharsRef, documents no restriction and reads the slice correctly in its own assert; the trigger needs only Lucene's own CharsRef.subSequence (plus the cast its declared CharSequence return type requires); and Lucene has fixed latent offset bugs before. Verdict: a defect, minor; the residual doubt is priority, not whether it is a bug, and a maintainer may still prefer to document an offset-0 rule (option C). Not security: wrong romaji for the caller's own data, never outside the array.

One fix was built and validated through gradle on 645d4fe14a20; the other two options are listed for discussion and were not built.

2 lines changed in KatakanaRomanizer.java. pos and inputOffset stay slice-relative, so the loop bounds (:108, :162, :172) are unchanged.

      for (CharsRefBuilder output : pendingOutputs) {
        output.append(input.chars, input.offset + pos, input.length - pos);   // :165
      }
  ...
    for (int len = Math.min(input.length - inputOffset, keystrokes.length); len > 0; len--) {
      CharsRef ref = new CharsRef(input.chars, input.offset + inputOffset, len);   // :173

Trade-offs

None known: for offset 0, which is every call Lucene makes, the behaviour is identical, with no extra work or allocation.

Note for review: :136 appends cref.chars, 0, cref.length without cref.offset. Those refs come from the romaji map, built with new CharsRef(String) (:63), so their offset is always 0 and this is harmless; using cref.offset there for consistency with :150 is optional and was not validated.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: the repro fails on stock and passes with the fix, assertions on and off.
• The whole :lucene:analysis:kuromoji:test (31 suites, including TestKatakanaRomanizer, TestJapaneseCompletionFilter, TestJapaneseCompletionFilterFactory, TestJapaneseCompletionAnalyzer and the repro): 196 tests, 7 skipped, 0 failures.
• :lucene:analysis:kuromoji:check -x test (checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest): passes, with the test file in the tree.
• Earlier, on main ea57360912ed, the same two lines made the finder's tests, the independent reproduction's three tests and the history probe pass.

if (input.offset != 0) input = CharsRef.deepCopyOf(input); at the top of romanize (CharsRef.deepCopyOf returns a copy with offset 0, CharsRef.java:157). Not built.

Trade-offs

Keeps the body as it is, but costs a copy for every sliced input and leaves the index arithmetic inconsistent with the assert.

Add "the input must have offset 0" to the javadoc and assert input.offset == 0. Not built; not recommended.

Trade-offs

A maintainer may reasonably prefer this for filter plumbing. But with assertions off, as in production, callers that miss the javadoc still get wrong romaji silently, and romanize would accept less than the general CharsRef slice that Lucene's own subSequence returns, so such callers must copy first.

Method: semi-formal (a hunt unit, the Japanese completion filter).

• The finder, an AI tool, wrote a reference romanizer for a translation-validation harness (JcfTranslationValidation, section S3) and noticed from reading the code that romanize never adds input.offset.
• The harness then checked it exhaustively within a bound: every katakana string of length 1 to 3 over U+30A0 to U+30FF, with the suffixes "", "k" and "sy" (2,682,144 inputs), romanized once as a whole CharsRef and once as the same content sliced at offset 1 of a padded array. 2,682,141 differ (class OFFSET_SLICE_DIFF), on main and 10.5.2; the 3 equal ones are ア, アア and アアア, which the padding character ア happens to match.
• The finder then wrote a public-API JUnit on the stock jars, and independent verifiers wrote their own tests from the claim. The one-test repro was cut from them on 2026-10-29.
• It is recorded as semi-formal: the signal came from a bounded-exhaustive comparison, not from a model checker, solver or proof, and the hypothesis came from code reading.