SweetSpotSimilarity.hyperbolicTf returns max instead of min for frequencies far below its midpoint

SweetSpotSimilarity offers hyperbolicTf, a tanh-shaped tf curve "provided as a convenience for subclasses that want to use a hyperbolic tf function": about min below the midpoint xoffset, about max above it. It computes the tanh as (base^x - base^-x) / (base^x + base^-x) with x = freq - xoffset. Once |x| passes ln(Double.MAX_VALUE) / ln(base) (for base > 1: about 308 at base 10, 2705 at the default base 1.3) one power overflows to Infinity and the ratio is NaN, on either side of the midpoint. The guard return Float.isNaN(result) ? max : result; returns max for both: right above the midpoint, wrong below it, where the formula gives min. A subclass that uses hyperbolicTf as its tf() with a midpoint far above its documents' frequencies then gives the largest tf to the rarest mentions and inverts rankings, silently, in every release since 2.0.0. The class defaults, the stock tf() and the documented example curves are unaffected. A one-branch fix (use Math.tanh when the ratio is NaN) was validated.

SweetSpotSimilarity.java:160-179:

final double x = (double) (freq - xoffset);                      // :167
final float result =
    min + (float) ((max - min) / 2.0f
        * (((Math.pow(base, x) - Math.pow(base, -x))
                / (Math.pow(base, x) + Math.pow(base, -x)))          // :174-175
            + 1.0d));
return Float.isNaN(result) ? max : result;                          // :178

1. The formula (javadoc, :150-152): tf(x)=min+(max-min)/2*(((base**(x-xoffset)-base**-(x-xoffset))/(base**(x-xoffset)+base**-(x-xoffset)))+1). The ratio is tanh((x - xoffset) * ln(base)): -1 far below the midpoint, 0 at it, +1 far above it.
2. Far above the midpoint (freq 800, xoffset 400, base 10: x = 400): Math.pow(10, 400) is Infinity, Math.pow(10, -400) is 0, the ratio is Infinity / Infinity = NaN, and the guard returns max = 2.0. The formula's limit is max too: this side is right.
3. Far below the midpoint (freq 1: x = -399): Math.pow(10, 399) is Infinity and Math.pow(10, -399) is 0, so the ratio is (0 - Infinity) / (0 + Infinity) = NaN, and the same guard returns max = 2.0. The formula's limit here is min = 0.
4. Just inside the range (freq 92: x = -308): 10^308 is still finite, the ratio is exactly -1 and the result is 0.0. So tf falls from 2.0 at freq 91 to 0.0 at freq 92.
5. Where it reaches scores. The TF-IDF scorer multiplies tf(freq) by the query weight at search time (TFIDFSimilarity.java:469). The stock SweetSpotSimilarity.tf delegates to baselineTf (:122-130), so only a subclass whose tf() returns hyperbolicTf is affected; no non-test code in Lucene calls hyperbolicTf.

For base > 1, hyperbolicTf returns max wrongly when xoffset - freq is above about 709.78 / ln(base): about 2705 at base 1.3 (the default), 1024 at base 2, 710 at e, 308 at base 10, 154 at base 100. A verifier measured the first failing xoffset at freq 1 for 14 bases (2707 at base 1.3, 1025 at 2, 711 at e, 310 at 10, 156 at 100); each matched this threshold.

Factors (0, 2, 10, 400): tf = 2.0 for freq 1 to 91, 0.0 at 92, 1.0 at 400, 2.0 at 800. The exhaustive check over 720,000 inputs found 37,209 mismatches with the formula, every one of them this fallback, and no other kind.

• Who. Opt-in only. The class javadoc says "baselineTf and hyperbolicTf functions are provided, which subclasses can choose between" (SweetSpotSimilarity.java:28-29), and the upstream test builds exactly such a subclass (TestSweetSpotSimilarity.java:207-213). Downstream, Solr's SweetSpotSimilarityFactory builds one from the hyperbolicTfMin/Max/Base/Offset schema settings, without a range check (read in Solr's source, not run).
• When. xoffset - freq above about 2705 at the default base 1.3 (a midpoint of thousands of occurrences), or above 308 at base 10 (a steep, near step-like curve with a midpoint of hundreds). The class author's advice is to pick "the value that makes sense for your data", and that for books or encyclopedia articles "the sweetspots are going to be much larger" (java-user@, 2012), so very long documents are where such a setting is plausible.
• Not affected (executed): the class defaults (0, 2, 1.3, 10), the four curves in the shipped doc-files/ss.gnuplot and the upstream test factors (3.3, 7.7, e, 5): 0 mismatches over freq 1..100000.
• Effect. Once configured it is deterministic: every document whose frequency is below the overflow point gets the largest possible tf, documents just above it get about the smallest, and rankings invert. No exception, assertions on and off. Values stay inside [min, max], so range checks do not notice.
• Nothing is written to the index. tf is applied at search time, so a fixed jar corrects existing indexes without reindexing.
• Not claimed: an effect on impact-based top-k pruning (WAND/MaxScore). It is plausible, since scores must not decrease with freq, but it was not executed.
• How likely: rare. No report in 20 years.

Public API only; the test sits in org.apache.lucene.misc next to TestSweetSpotSimilarity but needs no package access. Base 10 and midpoint 400 are small round values safely past the overflow point: at freq 1 the first failing integer midpoint is 310 at base 10 and 2707 at the default base 1.3 (so, for example, 3000 there).

The ranking inversion (two 450-term documents, 0.0953 for the single mention against 0.0477 for 400 mentions) is a consequence of this value, shown by the finder's longer test; it is not part of this repro.

Regression test: this same test; it passes with the recommended fix. It could also go into TestSweetSpotSimilarity as one more method.

cp $REPRO/TestSweetSpotHyperbolicTfFarBelowMidpoint.java $LUCENE/lucene/misc/src/test/org/apache/lucene/misc/
cd $LUCENE && ./gradlew :lucene:misc:test --tests TestSweetSpotHyperbolicTfFarBelowMidpoint -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
TestSweetSpotHyperbolicTfFarBelowMidpoint > testFarBelowMidpointGivesMin FAILED
    java.lang.AssertionError: hyperbolicTf(1), factors (0, 2, 10, 400) expected:<0.0> but was:<2.0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:82178102E9400FED]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:577)
        at org.apache.lucene.misc.TestSweetSpotHyperbolicTfFarBelowMidpoint.testFarBelowMidpointGivesMin(TestSweetSpotHyperbolicTfFarBelowMidpoint.java:27)

1 test completed, 1 failed

# with the recommended fix: BUILD SUCCESSFUL (assertions on and off)

• The formula is the spec. The method javadoc gives it exactly (SweetSpotSimilarity.java:150-152); setHyperbolicTfFactors calls xoffset "the midpoint of the hyperbolic function" (:75); the shipped doc-files/ss.gnuplot:47 plots the same formula. The class author: "The functions are pretty clearly spelled out in the javadocs" (java-user@, 2012), and of the default curve on dev@ (2006): "~2 for all values greater then 15, and ~0 for all values less then 5".
• Monotonicity is a contract. Similarity.java:207-209: "Score must not decrease when freq increases, ie. if freq1 > freq2, then score(freq1, norm) >= score(freq2, norm)". The TF-IDF scorer multiplies tf by a fixed positive query weight and norm (TFIDFSimilarity.java:469-471), so a tf that falls from 2.0 at freq 91 to 0.0 at 92 makes the score fall too. TFIDFSimilarity.java:342-344: tf implementations "usually return larger values when freq is large".
• The guard was only exercised on the correct side. TestSweetSpotSimilarity.java:205-226 uses factors (3.3, 7.7, e, 5) and freq 1..1000; with base e the power overflows only for freq >= 715, above the midpoint. It checks 3.3 <= tf <= 7.7, tf(5) = 5.5 and tf(0) = 0, nothing below the midpoint.
• History. The guard is in the first check-in (2006, LUCENE-577 / #1655), which describes hyperbolicTf as "available for subclasses to call from their own tf functions" and never discusses overflow.

The strongest case for "intended": the NaN-to-max guard is deliberate code, it keeps the value inside the documented bounds (:71-72: min and max are "the minimum/maximum tf value to ever be returned"), and the trigger factors are unusual.

Why it fails: staying inside [min, max] is necessary, not sufficient. The documented formula, the plotted curve and the author's own description all put the value near min there; returning max makes tf decrease with freq, which SimScorer#score forbids; and the only evidence of the guard's purpose, the upstream test, reaches only the upper overflow. Rarity lowers the impact; it does not make the behaviour intended. All verifiers rated the intended-behaviour argument weak.

Option A was built and run through gradle on 645d4fe14a20; option B passed the repro and the closest suite; option C was not run.

In hyperbolicTf (SweetSpotSimilarity.java:169-176), compute the ratio first and, when it is NaN, replace it with Math.tanh(x * Math.log(base)) (+8/-8 lines).

double ratio =
    (Math.pow(base, x) - Math.pow(base, -x)) / (Math.pow(base, x) + Math.pow(base, -x));
if (Double.isNaN(ratio)) {
  // base^x or base^-x overflowed to Infinity, far above or far below xoffset; the ratio is
  // tanh(x * ln(base)), which goes to +1 above the midpoint and to -1 below it (for base > 1)
  ratio = Math.tanh(x * Math.log(base));
}
final float result = min + (float) ((max - min) / 2.0f * (ratio + 1.0d));

Trade-offs

• No change where the code works today. Only the NaN branch changes, so every finite value stays bit-identical: over freq 1..100000 the stock and fixed values differ only at the overflow points (91, 294 and 76 of them in three trigger configurations) and nowhere for the defaults, the gnuplot curves and the upstream test factors.
• Upper side unchanged in effect. Above the midpoint tanh gives +1, so the result is min + (max - min) in float, the value the finite path already returns just below the overflow point; no tf above max and no decreasing step in the eight probed configurations.
• Also covers the base < 1 side case (a decreasing curve), where the stock guard returns max far above the midpoint. The final isNaN guard stays for NaN factors.
• Cost. The extra log and tanh run only on overflow; no benchmark was run.

What the checks validated

Scratch worktree of 645d4fe14a20, gradle, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test and TestSweetSpotSimilarity pass with assertions on and off; TestSweetSpotSimilarity with -Ptests.nightly=true passes 3 of 3, including the nightly testSweetSpotComputeNorm;
• the whole :lucene:misc:test passes: 317 tests, 0 failures, 9 skipped;
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest and forbiddenApisMain for misc pass;
• earlier, the finder's version of the same logic made the finder's 5-test class, an independent verifier's 5-test class and the portable history repro pass.

Replace the Math.pow ratio everywhere with Math.tanh(x * Math.log(base)), which cannot overflow (+2/-7 lines).

// (base^x - base^-x) / (base^x + base^-x) is tanh(x * ln(base)); Math.tanh does not overflow
final float result =
    min + (float) ((max - min) / 2.0f * (Math.tanh(x * Math.log(base)) + 1.0d));

Trade-offs

Simpler, and no Math.pow at all, but a few finite values change: by up to 4.4e-16 absolute, at values near min (the largest: 4.3785993E-9 becomes 4.3785997E-9), at 3, 28 and 1 points of the three trigger configurations; none for the defaults, gnuplot curves and upstream test factors over freq 1..100000. Not benchmarked.

What the checks validated

Same worktree: the minimal test and TestSweetSpotSimilarity (-Ptests.nightly=true, 3 of 3) pass, and checkGoogleJavaFormat passes. The whole module was not run with this option.

return Float.isNaN(result) ? (x > 0 ? max : min) : result;

Trade-offs

The smallest diff, but right only for base > 1: for base < 1 the limits swap, so it would need (x > 0) == (base > 1). It keeps two hand-coded limits next to a formula that already has them; A and B follow the formula for every finite base > 0. A javadoc-only limit (xoffset below ln(Double.MAX_VALUE) / ln(base), about 308 at base 10) would avoid the overflow, since freq is always positive, but it would forbid long-document midpoints and still return max silently above the limit; the code fix is one branch.

Method: semi-formal (the hunt's tier for bounded-exhaustive checks of the real jar against exact semantics). The hunt unit checked scoring components against their documented formulas.

• Translation validation. The harness evaluated hyperbolicTf on the stock jar and the javadoc formula written as min+(max-min)/2*(tanh((freq-xoffset)*ln(base))+1) over freq 1..5000 × base {1.01, 1.3, 2, e, 10, 100} × xoffset {0, 5, 10, 100, 300, 400, 1000, 3000} × 3 (min, max) pairs: 720,000 checks, 37,209 mismatches, all the fallback (got max, formula about min), 0 others.
• JUnit. A 5-test public-API class failed 4 of 5 on main (assertions on and off), 10.5.2 and 9.12.3 (assertions on): tf(1) = 2.0 for (0, 2, 1.3, 3000) and (0, 2, 10, 400), hyperbolicTf(92)=0.0 < hyperbolicTf(91)=2.0, and the ranking inversion; its control just above the overflow point passed. The minimal repro above is one assertion of that class.