Bundled commons-codec upgrades changed phonetic filter tokens (RefinedSoundex in 10.2.0; Metaphone, ColognePhonetic and Daitch-Mokotoff on main), and no release note says to re-index

Lucene's phonetic filters compute no codes themselves: PhoneticFilter calls a commons-codec Encoder and DaitchMokotoffSoundexFilter calls commons-codec's DaitchMokotoffSoundex, so the indexed codes are whatever the bundled commons-codec computes. Lucene bundled 1.13 up to 10.1.0, 1.17.2 from 10.2.0 to 10.6.0, and 1.23.0 on main and branch_11x. Bug fixes in those releases changed the codes of ordinary words: RefinedSoundex Köln gets K378 since 10.2.0 (10.1.0 indexed only the word), and on main Metaphone chair is XR instead of KR, ColognePhonetic quick is 4 instead of 44 and Daitch-Mokotoff it's is 040000 instead of 034000. Neither CHANGES.txt nor MIGRATE.md mentions any of these upgrades or asks for re-indexing, although Lucene announced the same kind of change in 2014 (LUCENE-6058, "any index using it will need to be rebuilt"), gives re-index advice for the 11.0 Snowball upgrade, and tells 11.0 users to "Simply open the index with the new version". An index written by the older version and searched by the newer one silently changes its sound-alike matches in that field until it is re-indexed. The new codes are correct; the fix is a migration note.

// PhoneticFilter.java:70-78 (verbatim, indentation removed)
String value = termAtt.toString();
String phonetic = null;
try {
  String v = encoder.encode(value).toString();
  if (v.length() > 0 && !value.equals(v)) phonetic = v;
} catch (Exception _) {
} // just use the direct text

if (phonetic == null) return true;

Line 73 is where commons-codec computes the code; if the encoder throws, lines 75-76 keep only the word.

• Delegation: PhoneticFilter encodes every token with the configured Encoder (PhoneticFilter.java:73); DaitchMokotoffSoundexFilter calls encoder.soundex(term) on a DaitchMokotoffSoundex (DaitchMokotoffSoundexFilter.java:38, :79).
• inject: with inject=false the code replaces the word (PhoneticFilter.java:80-84); with inject=true, the factory default (PhoneticFilterFactory.java:52-53, :106), the code is emitted and the word is kept at the same position (:86-96). So with inject=true exact-word matches survive an upgrade and only sound-alike matches change.
• Why 10.1.0 kept only the word for Köln: commons-codec 1.13's RefinedSoundex throws ArrayIndexOutOfBoundsException for letters outside A-Z (Müller: "Index 155 out of bounds for length 26"), and PhoneticFilter.java:75-76 swallows it. From 1.16.1 (CODEC-311) the encoder skips those letters and codes the rest of the word (Köln -> K378, the code of Kln). With inject=false the code now replaces the word; with inject=true the word stays and the code is added.
• The pin: gradle/libs.versions.toml:8 is commons-codec = "1.23.0" on main; the release POMs declare 1.17.2 for 10.2.0 to 10.6.0 and 1.13 for 9.0.0 to 10.1.0.

commons-codec's changelog lists two fixes under CODEC-331; the JIRA issue's own title is the Beider-Morse one.

Lucene's phonetic tests pin no input that changed: the Metaphone assertions use "aaa bbb ccc easgasg" (TestPhoneticFilter.java:36-44), and TestDaitchMokotoffSoundexFilter uses the same words. None of the bumps touched phonetic code, phonetic tests or a note about commons-codec: the commons-codec part of each changed only the version catalog, the lock file and license checksums, plus one test module-info line (requires org.apache.commons.codec;) in #14129 and #14177, and #14177 also carried the OpenNLP upgrade with its CHANGES line.

• Who: applications that put PhoneticFilter (with Metaphone, ColognePhonetic or RefinedSoundex) or DaitchMokotoffSoundexFilter into their own analyzer and keep the index across a Lucene upgrade. analysis-phonetic is an opt-in module with no analyzer of its own; default constructors and ordinary words suffice.
• Released today: RefinedSoundex, for words with letters outside A-Z, when upgrading from 10.1 or earlier to 10.2.0 or later: 19.0% of the 2,901 names in Lucene's world-cities-points.txt test resource, 0% of the ASCII-only Reuters words.
• With 11.0.0 (now on branch_11x), distinct words whose code changes from 10.5.2 to main, inject=false:

• Effect: no exception and no index damage. In that field old documents and new queries stop matching, or match other sound-alike documents, until the field is re-indexed. An independent cross-version index test found 11 of 14 queries missing their document when main searched a 10.5.2 index, two of them inject=true sound-alike queries (chorous, Kexe).
• A normal path: keeping an index across a minor upgrade is ordinary, and 11.0's migration guide allows upgrades across majors "without reindexing when no format breaks occur" (MIGRATE.md:138). That policy is about index formats and promises nothing about analysis output, which is why an analysis change needs its own release note (as Snowball got, MIGRATE.md:196-202).
• Not measured (not claimed): DoubleMetaphone, Soundex, Caverphone2 and Nysiis were unchanged on a 53-word probe and BeiderMorseFilter (GENERIC/APPROX) on 6,901 words; the Beider-Morse changes in the commons-codec 1.19.0 and 1.23.0 notes and the icu4j and morfologik bumps on main were not measured otherwise.

• Smallest input: one word, one encoder, inject=false, public API only; no index is needed, because the mismatch is between the token an old index holds and the token a new query gets.
• Read it honestly: the failure demonstrates the change. It is not a regression test for the fix: the recommended fix is a migration note, after which this test still fails. The suggested regression test is this same test with KR changed to XR (fix B).
• Why Metaphone: the hunt's first example was ColognePhonetic quick, but a verifier noticed that the new Cologne output collapses equal codes separated by vowels (Kekse 448 -> 48), which may differ from the published Kölner Phonetik rules; a Metaphone example cannot be misread as a claim that the new code is wrong. That Cologne question belongs to commons-codec and is not part of this bug.
• Not covered by this test: the RefinedSoundex change that shipped in 10.2.0. The hunt's testRefinedSoundexSameAs10_1 (Köln, expected:<K[öln]> but was:<K[378]>) fails on every release from 10.2.0 to 10.6.0.
• Determinism: no random input; seed DEADBEEF and a random seed with 5 iterations fail the same way.

cp $REPRO/TestMetaphoneTokenSinceLastRelease.java \
   $LUCENE/lucene/analysis/phonetic/src/test/org/apache/lucene/analysis/phonetic/
cd $LUCENE
./gradlew :lucene:analysis:phonetic:test \
  --tests org.apache.lucene.analysis.phonetic.TestMetaphoneTokenSinceLastRelease -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same failure)
# control: the same test passes on the 10.5.2 and 10.6.0 release jars (commons-codec 1.17.2)

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on (stack frames shortened: module prefixes removed)
TestMetaphoneTokenSinceLastRelease > testChairKeepsTokenOfLastRelease FAILED
    org.junit.ComparisonFailure: term 0 expected:<[K]R> but was:<[X]R>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:76A30ABE1062EA6B]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
        ...
        at org.apache.lucene.analysis.phonetic.TestMetaphoneTokenSinceLastRelease.testChairKeepsTokenOfLastRelease(TestMetaphoneTokenSinceLastRelease.java:30)
1 test completed, 1 failed
BUILD FAILED
# gradle, assertions off (-Ptests.asserts=false): the same ComparisonFailure
# random seed, 5 iterations: 5 of 5 fail
# 10.5.2 and 10.6.0 release jars (commons-codec 1.17.2): PASS, 1 test, 0 failures

• Same module, same dependency: CHANGES.txt:11240-11243, LUCENE-6058: "With the upgrade to Apache commons codec 1.10, the experimental BeiderMorseFilter has changed its behavior, so any index using it will need to be rebuilt." On that issue Michael McCandless wrote: "It's marked experimental so it's allowed to change between releases... but I'll updated the CHANGES.txt entry to advertise this." (#7120).
• Current practice for analysis-output changes: Snowball in 11.0 (MIGRATE.md:196-202: "Existing indexes that used Snowball stemming should be reindexed so search-time analysis matches indexed tokens"), Thai in 10.6 (:757-764: "may therefore stop matching queries analyzed with 10.6 until it is reindexed"), Romanian in 10.0 (:871-873: "you should reindex Romanian documents").
• The released bump was hand-written: #14129 ("Upgrade commons-codec from 1.13.0 to 1.17.2", msfroh, merged by msokolov) is not a dependabot PR. Its backport #14177 also upgraded OpenNLP, which got a 10.2.0 CHANGES line (GITHUB#14130, CHANGES.txt:2026-2027); commons-codec got none.
• The new upgrade policy raises the stakes: 11.0 invites in-place upgrades across majors (MIGRATE.md:136-154), exactly where phonetic matches change.
• Nothing points users to the codec notes: no Lucene documentation (javadoc, the module overview, CHANGES.txt or MIGRATE.md) says which commons-codec version is bundled; only the build files and the published POM do, and the module overview names Apache Commons Codec without a version (lucene/analysis/phonetic/src/java/overview.html:28-31). By contrast the ICU module's overview has a "Backwards Compatibility" section about the Unicode version it tracks (lucene/analysis/icu/src/java/overview.html:343-350).

The intended-behaviour verifier and the cross-model lenses built the strongest case for "intended": no Lucene document promises stable phonetic codes (the javadoc says only "Create tokens for phonetic matches" and links to Commons Codec), DaitchMokotoffSoundexFilter is @lucene.experimental, dependabot PRs carry the skip-changelog label in every ecosystem, including the gradle bumps (.github/dependabot.yml:53; also :13 GitHub Actions and :35 uv) and 349 of 351 dependabot commits touch no CHANGES.txt, CONTRIBUTING.md says only "You may want to add a CHANGES entry", the bump PR bodies quote the commons-codec release notes, the 10.0 Snowball upgrade got only an API note (MIGRATE.md:867-869), and the main-only part is still unreleased.

Why it fails: LUCENE-6058 is the same module, the same dependency and an experimental filter, and was announced with a rebuild note; #14661 added skip-changelog to the bot labels so the changelog-check workflow skips bot PRs, which is not a decision about migration notes for changed analysis output (that workflow has since been turned off, #16366); the released 10.2.0 change came from a hand-written PR; and no Lucene documentation tells a user which codec notes to read. These arguments are why the bug is minor and documentation-only, not why the omission would be intended.

The defect is a missing note, so the main fix is text. Two small additions make it last; reverting the codec is not an option. Options A, B and C were run in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3).

Add to "Migration from Lucene 10.x to Lucene 11.0" on main and branch_11x (a draft):

### Phonetic analysis: commons-codec upgrade changes some phonetic codes

The analysis-phonetic module gets its encoders from the bundled commons-codec, which
Lucene 11 upgrades to 1.23.0 (Lucene 10.2 to 10.6 bundle 1.17.2). Fixes in commons-codec
change the codes of some ordinary words:

- `Metaphone`: "CH" follows the basic Metaphone rules, for example `chair` is `XR`
  instead of `KR` (CODEC-249).
- `ColognePhonetic`: repeated codes are collapsed, for example `quick` is `4` instead
  of `44` (CODEC-317).
- `DaitchMokotoffSoundexFilter`: punctuation is ignored, for example `it's` is `040000`
  instead of `034000` (commons-codec 1.19.0).

Indexes upgraded from Lucene 10.1 or earlier (commons-codec 1.13) also see a
`RefinedSoundex` change from Lucene 10.2: words with letters outside A-Z, such as `Köln`,
now get a code (`K378`) where only the original word was indexed before (CODEC-311).

An existing index whose fields use these encoders may stop matching queries analyzed
with Lucene 11 until it is reindexed. With `inject=true` (the `PhoneticFilterFactory`
default) the original words are indexed too, so exact-word matches still work, but
sound-alike matches change. Reindex such fields.

Trade-offs

• Docs only; no behaviour change. It reaches users who read the migration guide before upgrading.
• The example codes are executed facts for these words only; the draft names no Beider-Morse change because none was measured.

What the checks validated

• ./gradlew :lucene:documentation:markdownToHtml succeeds; the rendered MIGRATE.html has the new phonetic-analysis-commons-codec-upgrade-changes-some-phonetic-codes section.
• ./gradlew :lucene:analysis:phonetic:test with the note (and B's test): 9 classes, 42 tests, 0 failures.
• The minimal repro still fails with expected:<[K]R> but was:<[X]R>, as it must: the note documents the change and does not undo it.

The minimal test with KR changed to XR, kept permanently (for example as one method in TestPhoneticFilter).

assertTokenStreamContents(
    new PhoneticFilter(tokenizer, new Metaphone(), false), new String[] {"XR"});

Trade-offs

Guards one word of one encoder; it tells users nothing by itself and only makes the next change visible to the committer who bumps commons-codec.

What the checks validated

• Passes on main through gradle with assertions on and off; checkGoogleJavaFormat is clean.
• Fails on the 10.6.0 jars (commons-codec 1.17.2) with expected:<[X]R> but was:<[K]R>, so a codec that codes chair differently fails CI at the bump.

One paragraph in lucene/analysis/phonetic/src/java/overview.html, like the ICU module's "Backwards Compatibility" section: the codes come from the bundled Apache Commons Codec, an upgrade can change them, and affected fields need re-indexing.

Trade-offs

A lasting statement, but few users read module overviews before upgrading; it complements A. It depends on A: the validated draft ends with "Such changes are listed in the migration guide; reindex the affected fields", which is true only if A lands with it and later code-changing codec upgrades get a note too; without A it would end with "See the migration guide for release-specific compatibility notes."

What the checks validated

./gradlew :lucene:analysis:phonetic:renderJavadoc succeeds with the paragraph.

For example under "Changes in Runtime Behavior", pointing to the migration note. Not run.

Trade-offs

Visible in the changelog; one summary line, not a line per dependabot bump.

The new codes are upstream bug fixes, and 1.13's RefinedSoundex even throws on non-A-Z letters.

Trade-offs

An application that must keep its old codes until it re-indexes can run its own commons-codec version at its own risk (the history proxy runs ran the main jars with 1.17.2 and 1.13); Lucene should not.

• Method: a harness dumped canonical token streams of 55 analyzers and about 650 SPI filter chains on the stock jars of 9.12.3, 10.0.0, 10.3.2, 10.5.2 and main, each with its own declared dependencies, over five corpora (edge texts, 662 natural texts, 11,288 stemmer-vocabulary texts, 133,872 code points in 4 contexts, CJK with user dictionaries) and 249 generated per-parameter factory configurations.
• Oracle: the previous release's output plus the release notes; every adjacent-version difference was mapped to a CHANGES or MIGRATE entry. The phonetic differences were the only family with none.
• Cause: reading the release POMs (commons-codec 1.13, 1.17.2, 1.23.0) and the commons-codec changelog, then a four-encoder JUnit test pinning the previous release's tokens.
• Tier: non-formal (no model or proof), because the oracle is a previous release, not an exact semantics.