JaroWinklerDistance.setThreshold javadoc says a negative value gives the Jaro distance, but it gives Jaro-Winkler

JaroWinklerDistance (suggest module) adds a Winkler bonus for a shared prefix to the Jaro score j, but skips it when j is below a threshold, 0.7 by default. The setThreshold javadoc says "Set to a negative value to get the Jaro distance." That is false: getDistance skips the bonus only when j < threshold, and j is never negative, so a negative threshold adds the bonus to every pair that has a matching character (pairs with none return 0 first), exactly as 0 does. setThreshold(-1) then getDistance("martha", "marhta") returns 0.96111107 (Jaro-Winkler) instead of 0.9444444 (Jaro). Only a threshold of 1 or more gives plain Jaro, meaning Lucene's own unboosted value.

The sentence and the condition were written in the same 2008 commit (LUCENE-1297), so the sentence was never true; every release from 2.4.0 to 10.6.0 and main carries it. The threshold condition follows the usual Winkler boost-threshold rule and the rest of the class's documentation agrees with it, so this is a documentation defect: the proposed fix is the javadoc, with a code change only as an option. Not security-sensitive.

• JaroWinklerDistance.java:29: private float threshold = 0.7f;
• :103: float j = ((m / s1.length() + m / s2.length() + (m - mtp[1]) / m)) / 3; Each term is in [0, 1], so j is in [0, 1]; pairs with no match return 0 earlier (:100-102).
• :104: float jw = j < getThreshold() ? j : j + Math.min(0.1f, 1f / mtp[3]) * mtp[2] * (1 - j); where mtp[2] is the common prefix length and mtp[3] the longer length (:85-93).
  • threshold 0 or less: j < threshold is never true, so the bonus is always added (it is 0 for a pair with no common prefix). NaN acts the same way, because j < NaN is false. Pairs with no matching character never reach this line: they return 0 at :100-102 for every threshold.
  • threshold 1 or more: either j < threshold (plain j), or j == 1 and the bonus is multiplied by 1 - j = 0. So the result is always plain Jaro, in the sense below.
• What "plain Jaro" means here: Lucene's own unboosted value, j from :103, or 0 for a pair with no matching character (:100-102). It is not textbook Jaro in every detail: Lucene compares UTF-16 chars and halves the transposition count with integer division (:93), so aaaabc/aaabca gives 0.9444444 at threshold 1, where textbook Jaro (1.5 transpositions) gives 0.9166667; and ("", "") returns 0 at every threshold, where textbook Jaro gives 1 (the separate #3901). The bonus is not textbook Jaro-Winkler either: the prefix is not capped at 4 and the scale is min(0.1, 1 / longer length) (:85-93, :104), so abcdefgh/abcdefgx gives 0.975 at the default, where textbook Jaro-Winkler gives 0.95. None of this changes the bug: in this class, "the Jaro distance" can only mean the unboosted value.
• :108-113, the false sentence: "Sets the threshold used to determine when Winkler bonus should be used. Set to a negative value to get the Jaro distance."
• :118-123, the getter, which matches the code: "Returns the current value of the threshold used for adding the Winkler bonus. The default value is 0.7."

Other pairs at threshold -1 (Jaro, then the returned value): dwayne/duane 0.82222 to 0.84000, jones/johnson 0.79048 to 0.83238, abcdef/abxyzw 0.55556 to 0.64444. abcdef/abxyzw gets no bonus at the default 0.7, because its Jaro score is below 0.7, so the setting the javadoc offers for plain Jaro scores it higher than the default does.

• Opt-in, rare. JaroWinklerDistance is used only when an application picks it, through SpellChecker.setStringDistance, DirectSpellChecker.setDistance or its own code; the trigger is an explicit setThreshold call with a negative value, which is what the javadoc tells someone who wants plain Jaro to do.
• No Lucene caller. Nothing in Lucene main or its tests calls setThreshold on a JaroWinklerDistance; TestJaroWinklerDistance and TestSpellChecker use the default. The study's security screen read the spell-suggestion code of Elasticsearch, OpenSearch and Solr (2026-10-26, default branches): all three use the no-argument constructor and none calls setThreshold; the fact check re-read the Elasticsearch and Solr lines on 2026-11-01. A spot check of three servers, not a survey of applications.
• Effect. Pairs with a shared prefix score higher, never lower, so a spell checker's accuracy cut-off (the score read in SpellChecker and DirectSpellChecker) can let a few more candidates through and their order can change; an application's own similarity cut-off can accept more pairs. Silent, deterministic, on every call.
• Scale. Every release since 2.4.0 (2008); no public report was found in 18 years, which fits the small, easy-to-miss effect. Impact minor, priority Low.

• Deterministic. Fixed input, no randomness; it fails the same way with seeds DEADBEEF and 5EED1234 and with assertions on and off.
• It transcribes the current javadoc. Under the recommended javadoc fix it does not pass; it becomes obsolete. As a regression test for the javadoc fix (optional for a javadoc-only change), use this same test with the value the corrected sentence names: jw.setThreshold(1) instead of -1, message "setThreshold javadoc: 1 or more gives the Jaro distance". The run copy also renames the class to TestJaroWinklerJaroThreshold and the method to testThresholdOfOneGivesJaro and rewords the comment; nothing else changes. It passes with the javadoc fix, with assertions on and off. Under the code alternative the test as written passes unchanged.
• Releases. The same file fails unchanged on the 10.6.0 release jars. For releases before 9.1, which lack the org.apache.lucene.tests package, the study used an equivalent public-API main() probe (2.4.0 to 10.5.2, see the history).

cp $REPRO/TestJaroWinklerNegativeThreshold.java $LUCENE/lucene/suggest/src/test/org/apache/lucene/search/spell/
cd $LUCENE
./gradlew :lucene:suggest:test --tests org.apache.lucene.search.spell.TestJaroWinklerNegativeThreshold -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false (same failure)

# selected exact lines of the gradle output (task lines, warnings, repeated nested frames left out)
# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true
TestJaroWinklerNegativeThreshold > testNegativeThresholdGivesJaro FAILED
    java.lang.AssertionError: setThreshold javadoc: a negative value gives the Jaro distance expected:<0.9444444> but was:<0.96111107>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:246726E85FD44D7C]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:577)
        at org.apache.lucene.search.spell.TestJaroWinklerNegativeThreshold.testNegativeThresholdGivesJaro(TestJaroWinklerNegativeThreshold.java:27)
:lucene:suggest:test (FAILURE): 1 test, 1  failure
# separate recorded runs, summarized (not console output):
# -Ptests.asserts=false: the same failure
# 10.6.0 release jars (standalone): the same failure
# javadoc fix (option A): changes no code; the same test pinned to setThreshold(1) passes
# code alternative (option B): :lucene:suggest:test (SUCCESS): 1 test

• A direct promise, inverted. The sentence says a negative value gives the Jaro distance; a negative value gives the bonus to every pair that has a matching character.
• The other documentation agrees with the code: the first sentence of the same javadoc ("the threshold used to determine when Winkler bonus should be used"), the getter ("threshold used for adding the Winkler bonus") and the constructor ("the default threshold for the Jaro Winkler bonus (0.7)").
• No coverage: no test, CHANGES or MIGRATE entry, or package doc mentions negative thresholds; the upstream tests use only the default.
• History: one 2008 commit (LUCENE-1297, #2374) added the class, the sentence and the condition. In #2374 a reviewer asked for a configurable threshold and the contributor replied the same day "Added getter/setter for JaroWinker threshold and javadoc"; none of the 16 comments on #2374 (17 on LUCENE-1297 in JIRA, the last being the 2022 migration notice) discusses negative values. Earlier patch versions are not in the repository, so whether one honoured the sentence is unknown.

The strongest case: the runtime rule is coherent (its threshold condition is the usual Winkler boost-threshold rule) and 18 years old, and users who pass a negative value today get the always-bonus mode and may rely on it; perhaps the author meant "a negative value always applies the bonus" and wrote "Jaro" by mistake.

Why it fails as a defence of the docs: under either reading the sentence is false, since it says "Jaro" and the result is Jaro-Winkler. The argument only shows which side to fix, the javadoc, which is what this doc proposes.

Two options; both were applied in a scratch worktree of main 645d4fe14a20 and run with gradle. The javadoc fix is the proposal, because the threshold condition follows the usual Winkler boost-threshold rule and the rest of the class's documentation already describes it.

Replace the false sentence with what the code does.

   /**
-   * Sets the threshold used to determine when Winkler bonus should be used. Set to a negative value
-   * to get the Jaro distance.
+   * Sets the threshold used to determine when Winkler bonus should be used: the bonus is skipped
+   * when the Jaro distance is below the threshold and added otherwise. Set to 1 or more to get the
+   * Jaro distance; a value of 0 or less always applies the Winkler bonus.
    *
    * @param threshold the new value of the threshold
    */

Trade-offs

• No behaviour change, so no existing score moves; it uses the wording of the first sentence and of the getter ("threshold used for adding the Winkler bonus").
• Callers who already pass a negative value to get Jaro keep getting Jaro-Winkler; they can use 1 instead.
• "1 or more", not "above 1": at exactly 1 the bonus is multiplied by 1 - j = 0 when j = 1, so 1 also gives plain Jaro.
• The wording "skipped when below the threshold, added otherwise" is exactly the j < getThreshold() test, so it is also right for NaN, where the bonus is added (j < NaN is false). An earlier draft said "added when the Jaro distance is at least the threshold", which is wrong for NaN.
• It keeps the class's historical word "distance"; StringDistance values are similarities (1 means identical, StringDistance.java:22-31), so "Jaro similarity" is a fine alternative. "The Jaro distance" means Lucene's unboosted value; the empty-string result is the separate #3901.

What the checks validated

• Gradle, a second scratch worktree of 645d4fe14a20 (after the fact check corrected the wording), seed DEADBEEF: :lucene:suggest:checkGoogleJavaFormat and :lucene:suggest:renderJavadoc pass, and the rendered JaroWinklerDistance.html shows the new text in the method detail (the method summary shows its first sentence). The first wording passed the same steps in the first worktree.
• The spell tests (org.apache.lucene.search.spell.*: the 31 upstream tests of the 8 spell test classes plus the minimal test pinned to the corrected sentence, setThreshold(1)) pass, 32 tests, with assertions on; the pinned test passes with assertions off too.
• The minimal test as written was not re-run with it: the fix changes only a comment, so it keeps failing as on stock main.

In getDistance, never add the bonus when the threshold is negative, which makes the current sentence true.

     float j = ((m / s1.length() + m / s2.length() + (m - mtp[1]) / m)) / 3;
-    float jw = j < getThreshold() ? j : j + Math.min(0.1f, 1f / mtp[3]) * mtp[2] * (1 - j);
+    float jw =
+        (getThreshold() < 0 || j < getThreshold())
+            ? j
+            : j + Math.min(0.1f, 1f / mtp[3]) * mtp[2] * (1 - j);
     return jw;

Trade-offs

• Changes scores for every caller that passes a negative value today (Jaro-Winkler becomes Jaro); it would likely warrant a CHANGES entry as a behaviour change.
• Makes the threshold non-monotonic: 0 and -0.0 (for which getThreshold() < 0 is false) and NaN still always add the bonus, while -0.5 never does (the study's threshold table, run with this fix). That contradicts the first sentence of the same javadoc.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: the minimal test as written passes with assertions on (inside the 32-test spell run) and off, the 31 upstream spell tests pass, and :lucene:suggest:checkGoogleJavaFormat passes.
• The study's bounded-exhaustive model, run with this fix, left only the empty pair ('', ''), which is the separate #3901.

Method: bounded-exhaustive translation validation of the StringDistance implementations on the stock jar (hunt unit on spell-checker dictionaries and word break, 2026-10-26, main ea57360912ed).

• Inputs: every ordered pair of the 121 strings of at most 4 code points over {a, b, U+1F600}, 14,641 pairs; JaroWinklerDistance at thresholds 0.7, -1, 0 and 2.
• Oracle: a separate Jaro / Jaro-Winkler model that follows Lucene's matching arithmetic (UTF-16 chars, integer-halved transpositions; Jaro of ('', '') is 1), and the setThreshold sentence. Threshold -1 differed from that Jaro on 4,681 pairs: 4,680 plus ('', ''), which is the separate known #3901 (getDistance("", "") returns 0). Threshold 2 matched Jaro except on ('', '').
• The finder then wrote a public-API JUnit test with controls, which failed on main, 10.5.2 and 9.12.3; a code-variant fix made it pass and kept the upstream TestJaroWinklerDistance green.