DoubleMetaphoneFilter drops the alternate code of a token that already equals its primary code, so "JR" gets only JR while "jr" gets JR and AR

DoubleMetaphoneFilter is documented as "supporting secondary codes": for each token it asks the commons-codec encoder for a primary and an alternate sound code and emits both, replacing the token (inject=false) or next to it (inject=true). The encoder ignores letter case, so "jr" and "JR" both have primary JR and alternate AR. The filter skips a primary code equal to the token text, because the kept token already is that term; but the alternate's condition repeats that check (DoubleMetaphoneFilter.java:90), so a token that already equals its primary code never gets its alternate. "jr" is indexed as JR and AR, "JR" only as JR, and a search on AR finds 1 of the 2 documents. Only tokens spelled exactly like their own code are hit, in practice upper-case initials and acronyms such as J, JR, JFK and SMS; no lower-case string was hit in a probe of every A-Z string of length 1-3. The primary code is always kept, nothing throws and the index is not damaged. The check came with a 2009 rewrite (SOLR-1359), before the first release that shipped the class, so Solr 1.4.0 and Lucene 3.6.0 to 10.5.2 and main are affected. Deleting only the check is not a fix (replace mode would lose the primary); keeping the token as the primary and stacking the alternate on it is.

• Primary and alternate code: Double Metaphone gives a word two codes for its two most likely pronunciations; for many words they are equal. The commons-codec encoder returns them through doubleMetaphone(v) and doubleMetaphone(v, true); it upper-cases internally, so letter case does not change the codes.
• inject: inject=true keeps the original token and adds the codes at position increment 0; inject=false replaces the token with its codes.
• maxCodeLength: codes are cut to this length (factory default 4).

1. incrementToken computes primaryPhoneticValue and alternatePhoneticValue for the token text v (DoubleMetaphoneFilter.java:68-69).
2. The primary branch runs only when the primary is non-empty and !primaryPhoneticValue.equals(v) (:75-77). That skip is right: when the token already equals its primary code, the kept token carries that term.
3. The alternate branch requires a non-empty alternate that differs from the primary, and again !primaryPhoneticValue.equals(v) (:87-90). For "JR" the primary JR equals the token, so the alternate AR is skipped although it is a different code.
4. Nothing was queued, so the token is returned unchanged (:102-104): "JR" becomes [JR] in both modes, while "jr" becomes [JR, AR] (inject=false) or [jr, JR, AR] (inject=true).
5. The term is load-bearing in replace mode: with inject=false the token is not captured before the alternate overwrites the term, so deleting only the term turns "JR" into [AR] and loses the primary (executed). A fix must keep the token as the primary and stack the alternate on it.

Which tokens qualify depends on maxCodeLength: "JFK" keeps its alternate at maxCodeLength 1 (J, A) and 2 (JF, AF) and loses it at 3 and 4; "JR" keeps it at 1 and loses it from 2 on. Mixed case is not hit ("Jr" gets JR and AR). Over every A-Z string of length 1-3, 206 of 18,278 upper-case strings lose their alternate at maxCodeLength 4 (1, 23, 206, 206, 206 at maxCodeLength 1, 2, 3, 4, 8) and 0 of 18,278 lower-case strings do; in those 365,560 token streams there was no extra term and no position or offset change.

• Who is exposed. Any chain with the public DoubleMetaphoneFilter constructor or the doubleMetaphone factory (DoubleMetaphoneFilterFactory.java:40), with the defaults (inject=true, maxCodeLength=4) or any other settings, on text that is not lower-cased first. The factory's own javadoc example has no lower-casing filter (:26-32). The filter is opt-in: no stock Lucene analyzer uses it.
• Downstream defaults (read on 2026-10-28, not run): Solr's _default configset field type phonetic_en (dynamic fields *_phon_en) is the standard tokenizer + doubleMetaphone inject="false"; the Elasticsearch analysis-phonetic plugin builds this filter for the encoder double_metaphone, with replace defaulting to true (inject=false) and max_code_len to 4; the follow-up read the same for OpenSearch. Whether a user's chain lower-cases first is up to them.
• What it takes. A token whose exact text equals its own primary code: upper-case, consonant-heavy initials and acronyms no longer than maxCodeLength (J, JR, JFK, SMS). For such a token it happens every time at index time, at any index size, with assertions on or off. A query analysed by the same chain runs the same filter code, so it loses the alternate too (reasoned from the code; no query parser was run).
• What changes for the user. Every match through the primary code still works (TermQuery JR finds both documents). Only matches that go through the alternate code alone are lost (TermQuery AR: 1 of 2). For acronyms such a match is mostly a weak phonetic one: words whose only code is AR include "air" and "year".
• What does not happen. No exception, no extra or changed term, no position or offset change, no index damage.
• Workaround. A lower-casing filter before DoubleMetaphoneFilter: no lower-case string was hit in the probe (0 of 18,278 A-Z strings of length 1-3; other characters were not probed exhaustively), and lower-case words get both codes.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[JR, AR]> but was:<[JR]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.phonetic.TestDoubleMetaphoneFilterTokenEqualsCode.testTokenEqualToPrimaryCodeGetsAlternateCode(TestDoubleMetaphoneFilterTokenEqualsCode.java:38)

• Contract (implicit). Class javadoc: "Filter for DoubleMetaphone (supporting secondary codes)" (DoubleMetaphoneFilter.java:28). SOLR-813 (2008), which added the filter, says the existing PhoneticFilter "ignores the alternate encodings that the encoder provides for some words"; its contributor wrote that the new filter "checks to see if there is an alternate encoding for the token, and adds that to the stream as well, if it's different then the default encoding. This is part of the strength of the DoubleMetaphone implementation". The Solr Reference Guide says PhoneticFilter "will not provide the second (\"alternate\") encoding that is generated by the Double Metaphone Filter for some tokens". No sentence covers a token equal to its code.
• Tests. TestDoubleMetaphoneFilter.testAlternateInjectFalse expects "Kuczewski" -> KSSK, KXFS (:42-46); no upstream test has a token equal to its own primary code.
• No documented exception for upper-case or code-shaped tokens in the javadoc, the factory, CHANGES or MIGRATE; the factory's example chain does not lower-case.
• History. Before SOLR-1359 the alternate was emitted whenever it was non-empty and different from the primary: the previous file version (572d84c9c27) gives "J" -> J, A (executed). The rewrite 551e52f899a (2009-08-13, "DoubleMetaphone filter could skip or reorder tokens, fix and update to new API") added !primaryPhoneticValue.equals(v) to both branches. In the primary branch it removes a real duplicate (inject mode had given "J" -> [J, J, A]); in the alternate branch it removes a different code. The commit, the JIRA issue, the 2009 solr-dev@ thread "DoubleMetaphone bugs?" and CHANGES discuss only skipped or reordered tokens and keeping the original token "if there are no phonetic alternatives"; the commit added no test.
• A poor fit for a deliberate rule. The result flips with letter case, although the encoder ignores case, and with maxCodeLength; codes that are not fixed points of the encoder are still re-encoded ("XMT" -> SMT, "KSSK" -> KSK); in inject=true mode the term is not needed to keep the token (it is captured first anyway) and only suppresses the alternate and the state captures that emitting it takes. An equality test behaves like this by nature, so these points make a deliberate "already phonetic" rule less likely, not impossible.

The strongest case for "intended" (the cross-model reviewer's intended lens, the independent reproduction and the intended-behaviour check): a token that already is its own code is treated as already phonetic and passed through untouched. PhoneticFilter has a similar identity check (PhoneticFilter.java:74), the same committer fixed PhoneticFilter's duplicate tokens (SOLR-1360) the same day, the term is load-bearing in replace mode, the class promises only "supporting secondary codes", and alternates of acronyms are arguably noise.

Why it is weaker than it looks: the check compares the primary with the token, not the alternate, so in the alternate branch it never prevents a duplicate; PhoneticFilter's check drops only an output equal to the input; an equality test recognises only codes that are fixed points of the encoder at the chosen length, so it separates "JR" from "jr" and still re-encodes "XMT", a poor fit for an "already phonetic" rule (though it does not rule one out); no commit, issue or comment states such a rule; and the fix meets the replace-path concern without dropping the alternate. The cross-model reviewer rated the argument moderate, and so did the later cross-model fact check, which called this rebuttal partly fair but incomplete; the independent reproduction, the intended-behaviour check, the adjudication and the credibility audit rejected it. What survives is the narrow reach and the implicit contract: reasons for low priority, not for keeping the behaviour.

One fix was built and validated in a scratch worktree at a9779ef3c193; a workaround needs no code change. Removing only the condition was executed and is rejected.

The finder's fix, rebased on a9779ef3c193. When the token equals its primary code it is kept as is, and the alternate (when non-empty and different) follows at position increment 0, the same codes as the lower-case spelling. Tokens whose primary differs from their text take exactly the old path.

         saveState = true;
+      } else if (primaryPhoneticValue != null && primaryPhoneticValue.equals(v)) {
+        // the token already is its primary code: keep it, and stack the alternate on it
+        firstAlternativeIncrement = 0;
+        saveState = true;
       }
 
       if (alternatePhoneticValue != null
           && alternatePhoneticValue.length() > 0
-          && !alternatePhoneticValue.equals(primaryPhoneticValue)
-          && !primaryPhoneticValue.equals(v)) {
+          && !alternatePhoneticValue.equals(primaryPhoneticValue)) {

Trade-offs

• Only tokens that equal their primary code and have a different alternate change: they gain the alternate. Indexes built before the fix lack these terms until reindexed. Plain positive term matching on a rebuilt index only gains matches, but a negated clause (MUST_NOT) on the alternate code now excludes such documents, scores and top-k shift, and a mix of old and new segments (or an old index with the new query analysis) sees a partial change. A CHANGES entry should say so.
• Input that already is a code changes too: a stand-alone "JR" now gives JR and AR, so a pipeline that knowingly feeds codes, or a chain that runs the filter twice, gets the alternate of such a code as well.
• Cost on an affected token: two captureState() calls and one more output token (stock makes no capture there; the lower-case spelling costs as much or more, by reading the code). Performance was not measured.

What the checks validated

• Stock main: the paste-ready test fails 2 of 3, assertions on and off; the 11 upstream DoubleMetaphone tests pass; the whole module passes 8 classes, 41 tests.
• With fix A, TestDoubleMetaphone*: 3 classes, 14 tests, 0 failures, assertions on and off.
• With fix A, the whole :lucene:analysis:phonetic:test task: 9 classes, 44 tests, 0 failures, 0 skipped.
• With fix A, TestRandomChains and TestAllAnalyzersHaveFactories pass (one seed; whether its chains used this filter was not checked).
• ecjLintMain, ecjLintTest and checkGoogleJavaFormat of the module pass.
• Earlier checks: the finder's test with the upstream suites (12 of 12), the independent reproduction's (17 of 17 with the upstream suites), the adjudication's re-run, the finder's translation validation (0 mismatches in 1,641,332 checks) and the security probe (0 deviations in 365,560 token streams).

Add LowerCaseFilter (factory lowercase) before DoubleMetaphoneFilter. No lower-case token was hit in the probe: 0 of 18,278 lower-case A-Z strings of length 1-3 lose their alternate (other characters were not probed exhaustively), and the test's controls show lower-case words get both codes.

Trade-offs

With inject=true the kept original token becomes lower case; it does not help existing indexes or the Solr _default schema, and the library stays inconsistent.

A javadoc note that a token equal to its primary code is passed through without its alternate.

Trade-offs

Cheap, but it documents a dependence on letter case and code length that contradicts "supporting secondary codes".

Executed: inject=true is then right, but inject=false turns "JR" into [AR/1], "JFK" into [AFK/1], "SMS" into [XMS/1] and "J" into [A/1].

Trade-offs

The alternate overwrites the token and the primary code is lost: a worse bug (also found by the independent reproduction and the intended-behaviour check).

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier). The finder's harness compared each phonetic filter's output with the output computed from the encoder library called directly:

• filters: PhoneticFilter with 14 encoder configurations and both inject modes, DoubleMetaphoneFilter at maxCodeLength 1, 2, 4 and 8 in both modes, DaitchMokotoffSoundexFilter, and BeiderMorseFilter in 32 configurations (on a smaller term set: length up to 2 over 14 characters, 3 fixed words and 200 sampled terms);
• inputs: every token of length up to 3 over a 17-character alphabet plus fixed words, alone and in random streams of 1-4 tokens with position increments 0, 1 and 2, position lengths 1 and 2, and the keyword flag on or off;
• result on stock main: 1,641,300 checks, 16 mismatches, all DoubleMetaphoneFilter on the token "J" (expected J and A, got J); with the fix, 0 mismatches in 1,641,332 checks.

Reading the filter then located the repeated condition, and a JUnit test confirmed it on the stock main jars (found 2026-10-26).