FeatureField saturation and sigmoid scores cancel to 0.0 or tie when the feature value is far below the pivot

FeatureField.newSaturationQuery(field, feature, weight, pivot) documents the score weight * S / (S + pivot), which is positive and strictly increasing in the feature value S. SaturationFunction evaluates it as weight * (1 - pivot / (S + pivot)) in float, a rewrite whose code comment says it is there "to make sure it doesn't decrease with f in spite of rounding". When S is far below the pivot, S + pivot rounds to the pivot or a few ulps above it and the subtraction cancels: the score is exactly 0.0, or a small multiple of about weight * 2^-24, so documents with different values tie, while explain() prints the documented formula next to the wrong value. One document with S = 2^-17 and pivot 1000 scores 0.0 instead of 7.63e-9. newSigmoidQuery and IntervalQuery's saturation and sigmoid scoring carry the same rewrite. Every release since 7.4.0 (IntervalQuery since 8.0.0) and main are affected. The absolute error is tiny, so in the documented use, a SHOULD clause added to a text score, it is mostly hidden; it decides the order when the feature query ranks on its own or carries a large boost. Scoring as weight / (1 + pivot / S) in double removes it and keeps the scores monotone.

1. SaturationFunction.doScore returns weight * (1 - pivot / (f + pivot)), all in float (FeatureField.java:450 at 645d4fe14a20). The comment above it (:447-449): "should be f / (f + k) but we rewrite it to 1 - k / (f + k) to make sure it doesn't decrease with f in spite of rounding".
2. For S = 2^-17 and pivot 1000: floats near 1000 are 2^-14 = 6.1e-5 apart and S is 0.125 of that, so f + pivot = 1000.0000076... rounds to exactly 1000.0; pivot / (f + pivot) is 1.0, and 1 - 1.0 is 0.0.
3. A little higher, pivot / (f + pivot) lands one or a few floats below 1; floats just below 1 are 2^-24 = 6e-8 apart, so the score is a small multiple of weight * 2^-24 and whole ranges of S share one score.
4. The score, the bulk scorer and explain() all go through doScore (:454-457, :460-470, :477); FeatureQuery passes function.scorer(boost) into a TermScorer (FeatureQuery.java:135-137), so the weight and any outer boost multiply the error.

• Thresholds (weight 1; exhaustive run of the real jar over every legal stored value and 771 pivots, plus cvc5): the score is 0.0 exactly when fl(S + k) == k, always for S/k < 2^-25 (about 3e-8) and never for S/k > 2^-24; distinct stored values tie up to S/k of about 6e-5; errors above 2 ulps up to S/k of about 0.33.
• Staircase (pivot 1000, one document for each of the 2,233 values FeatureField can store between 2^-20 and 4.2e-4): 8 distinct scores; 1,281 documents (S up to 3.05e-5) score 0.0; 383 documents with S from 3.06e-5 to 9.13e-5 all score 5.9604645E-8. With the fix: 2,233 distinct scores, 0 zeros.
• Ranking (FILTER body:lucene + SHOULD saturation, pivot 1000): S = 9.54e-5 and S = 1.49e-4 both score 1.1920929E-7 and come back in doc-id order; with the fix the larger value ranks first.
• Large boost (new BoostQuery(newSaturationQuery(f, x, 1f, 1e5f), 1e5f), the linear-function workaround suggested on elastic/elasticsearch#49859): S = 0.5 and 0.50390625 both score 0.500679 (documented 0.4999975 and 0.5039037); S = 1.0 and 1.0039062 both 1.001358; every S up to 2^-8 scores 0.0 (documented about S); S = 0.0078125 scores 0.00596 (24% low) and S = 0.015625 scores 0.01788 (14% high).
• Default pivot (newSaturationQuery(field, feature), about the geometric mean of the values): on 2,000 values spread over 1e-9..1e3 (pivot 9.6e-4), 103 adjacent ties; 51 of them (69 tied pairs in all, in tie groups of up to 4 documents) are between documented scores at least 1% apart. The other 52 are near saturation, where the documented scores themselves round to the same float; with the fix 52 adjacent ties remain and none of them is 1% apart.
• Monotonicity holds everywhere (0 violations); no NaN, no negative score; the matching set never changes.

• newSigmoidQuery (FeatureField.java:527-532): (float) (weight * (1 - pivotPa / (Math.pow(f, a) + pivotPa))) in double. It gives 0.0 exactly when fl(S^a + k^a) == k^a: only when (S/k)^a <= 2^-53, and always when it is below 2^-54; the cut-off depends on the pivot (S/k up to 1.03e-4 for a = 4, 8.8e-5 at pivot 1000; up to 0.0101 for a = 8, 0.0095 at pivot 1000). For example pivot 1, exponent 8 and S = 2^-8 (documented 5.4e-20). The values lost are tiny; this half shares its lines and its fix with LFV-258 (sigmoid NaN when pivot^exp overflows).
• IntervalQuery default scoring (IntervalScoreFunction.java:72, pivot 1, IntervalQuery.java:71-73): the sloppy frequency adds 1 / (width - minExtent + 1) per interval, so gaps of 1,000,000 and 1,030,000 positions both score 9.536743E-7 (documented 9.99999E-7 and 9.708729E-7) and a 2^25-wide gap scores 0.0. Consecutive widths tie from 2,947 positions. The 4-argument constructor's sigmoid (:128) has the same rewrite.

• Who is exposed. Rankings by newSaturationQuery with a pivot set far above most values, the library's default pivot (about the geometric mean of the indexed values) on a feature spread over about 12 decades, or a large boost such as the linear-function workaround with boost and pivot 100,000; steep sigmoids (exponents 4 to 8) from 2 to 4 decades below the pivot; IntervalQuery with intervals thousands to millions of positions wide.
• Default vs opt-in. FeatureField scoring is opt-in (@lucene.experimental); saturation is IntervalQuery's default scoring. Nothing else needs changing: the default IndexWriterConfig, codec and IndexSearcher, one document, any index size.
• Silent. Different values get the same score (or 0.0) and come back in doc-id order; explain() prints the formula next to a value it does not compute. No exception, no log line.
• When the order changes. With a MUST text clause plus the SHOULD feature clause (the class javadoc's pattern), the float sum with the text score absorbs a feature score this small either way: 2,753 adjacent ties on stock against 2,704 with the exact formula (pivot 1000). With a FILTER clause plus the feature query, 2,522 ties against 0. With a large boost the error reaches 1% to 100% of the score.
• Downstream. Elasticsearch's rank_feature query calls FeatureField.newSaturationQuery with the default or a request's pivot and applies the request's boost on top.
• How likely. Rare: nobody has reported it since 2018 and the documented use masks it. Priority low, accuracy only.

ByteBuffersDirectory, the default IndexWriterConfig, a plain IndexSearcher; nothing random, one thread. 2^-17 is a positive normal float that FeatureField's 9-bit encoding stores exactly, so the stored value equals the input. The test asserts only what the javadoc supports, a positive score for a positive documented value, with no ulp tolerance.

It shows the FeatureField saturation, the strongest of the four copies. The ties, the sigmoid and IntervalQuery's scoring come from the same expression; they were executed in the study and in the tie-break and are described in the text, not added as more tests.

Regression test: this same test, as one more method of the existing TestFeatureField. It passes with the recommended fix.

cp $REPRO/TestFeatureFieldSaturationFarBelowPivot.java $LUCENE/lucene/core/src/test/org/apache/lucene/document/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestFeatureFieldSaturationFarBelowPivot -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (the same failure)

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF (assertions on, and again with -Ptests.asserts=false)
TestFeatureFieldSaturationFarBelowPivot > testSaturationScoreIsPositive FAILED
    java.lang.AssertionError: expected w*S/(S+k) = 7.62939447304234E-9 > 0, got 0.0
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:49D6D0ED47427B2D]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.document.TestFeatureFieldSaturationFarBelowPivot.testSaturationScoreIsPositive(TestFeatureFieldSaturationFarBelowPivot.java:43)
:lucene:core:test (FAILURE): 1 test, 1  failure
BUILD FAILED

# standalone runner, main 645d4fe14a20 jars, and release jars 10.5.2 and 10.6.0: -ea and -da FAIL 1/1, same message

# explain() of the same hit (stock)
0.0 = Saturation function on the features field for the popularity feature, computed as w * S / (S + k) from:
  1.0 = w, weight of this function
  1000.0 = k, pivot feature value that would give a score contribution equal to w/2
  7.6293945E-6 = S, feature value

# with the recommended fix, gradle, -Ptests.seed=DEADBEEF
:lucene:core:test (SUCCESS): 1 test                    # the minimal test (and again with -Ptests.asserts=false)
:lucene:core:test (SUCCESS): 503 tests, 26 skipped     # --tests 'org.apache.lucene.document.*'
:lucene:queries:test (SUCCESS): 118 tests              # --tests 'org.apache.lucene.queries.intervals.*'

• The contract is a formula. newSaturationQuery "will score documents as weight * S / (S + pivot)" (FeatureField.java:634); the class javadoc gives satu(S) = w * S / (S + k) and the sigmoid's w * S^a / (S^a + k^a) (:70-77); newSigmoidQuery the same (:681). The only documented limits are the weight and pivot ranges (:639-642); feature values must be positive normal floats (:162), which the repro's 2^-17 is.
• explain() says "computed as". FeatureField.java:484, :565; IntervalScoreFunction.java:82, :138.
• The rewrite's goal does not need the cancellation. Its comment (FeatureField.java:447-449, :528-530; IntervalScoreFunction.java:69-71, :125-127) names monotonicity only. Over all 65,024 legal stored values x 6 pivots, the stock rewrite has 0 monotonicity violations but 161,285 zero scores and 173,308 lost distinctions; the literal w*f/(f+k) in float has 7 violations (why a rewrite was needed); (float) (w / (1 + (double) k / f)) has 0 violations, 0 zeros and 0 lost distinctions.
• IntervalQuery says wider intervals score lower ("with wider intervals contributing lower values", IntervalQuery.java:41-43).
• The storage note promises more. Values "are stored with a relative precision of 2^-8" (FeatureField.java:58-60); the score's relative error reaches 100%.
• Tests never reach the region. TestFeatureField.java:142-195 pins the rewritten expression with delta 0 for S/k from 0.22 to 22, where it does not cancel; doTestSimScorer (:279-287) checks only finiteness and monotonicity.
• No record accepts it. No javadoc, test, CHANGES or MIGRATE entry, commit message or discussion (#9244, #9679) presents zeros or ties as intended.

The strongest case, from the refute review, the devil's-advocate verifier and the adjudication: the rewrite is deliberate; SimScorer only requires that the "Score must not decrease when freq increases" (Similarity.java:207) and 0.0 is a legal score; the errors are tiny and masked in the documented SHOULD use; newFeatureSort exists for ordering by value; FeatureField is @lucene.experimental; and upstream declined accuracy work before: #9120 (LUCENE-8072, Won't Fix: "There is a cost to log1p, I'm not sure we should do this per-hit") and #9547 (LUCENE-8501, jpountz: "In general scoring doesn't focus on accuracy").

Why it fails: those decisions accept small inaccuracies in formulas computed as documented (log(1+x) is what is documented there). Here a strictly positive documented formula returns 0.0, explain() says "computed as" that formula, distinct stored values tie, and a subtraction-free form keeps the monotonicity the comment asks for. Fixed counter-precedents exist: #12675 (MultiSimScorer should sum into a double, type:bug) and PR #16139 (IBSimilarity rounding gave negative scores, merged 2026-09-10). The class javadoc's "produces values between 0 and 1" (FeatureField.java:72) and IntervalQuery's "Scores are bounded to between 0 and 1" (IntervalQuery.java:56) put 0.0 inside the documented range, but they bound the values; the formula fixes them. Verdict: not intended, a defect, low priority.

Option A was built and run through gradle in a scratch worktree of 645d4fe14a20; it also fixes LFV-258 (sigmoid NaN), whose minimal test passes with it. B removes the zeros too but without a monotonicity guarantee, C was measured and rejected, D only documents the behaviour. Workaround without a fix: keep the pivot within a few decades of the values that must be told apart, or order by the value with FeatureField.newFeatureSort when the feature alone decides the order.

In FeatureField and IntervalScoreFunction, compute the saturation as weight / (1 + (double) pivot / f) and the sigmoid as weight / (1 + Math.pow((double) pivot / f, a)), drop the then-unused pivotPa fields and update the comments (3 files, +21/-23, formatted with ./gradlew tidy). Three zero-tolerance assertions in TestFeatureField.testBasics (:156, :190, :193) pinned the old float expression and now assert the documented formula computed in double, e.g. assertEquals((float) (6.0 * 42 / (42 + 4.5)), s.score(), 0f);.

// FeatureField.SaturationFunction.doScore (IntervalScoreFunction: the same with freq)
return (float) (weight / (1 + (double) pivot / f));

// FeatureField.SigmoidFunction.doScore (IntervalScoreFunction: the same with freq)
return (float) (weight / (1 + Math.pow((double) pivot / f, a)));

Trade-offs

• Still monotone. pivot / f is a correctly rounded division, so it does not grow when f grows; 1 + x, w / x and the float cast are monotone; Math.pow is specified as semi-monotonic. Measured: 0 violations over all 65,024 legal stored values x 6 pivots, and in the study's exhaustive run with the fix (771 pivots, 4 weights, 6 exponents).
• Accuracy. Returns the documented value within 1 ulp, correctly rounded except when the exact value is a float midpoint (1 of 2,861,056 checked cases; max relative error 5.8e-8 in the tie-break cases). Ordinary scores move by about 1 ulp: TestFeatureField.java:156 expected the stock 5.4193544, the fix gives 5.419355, and the exact value is 5.41935484. Exact scores change, so it is worth a CHANGES.txt line.
• Cost. One more division per scored document, and the saturation moves from float to double arithmetic; no benchmark was run.
• Review. No reviewer has read the diff line by line yet.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, seed DEADBEEF: the minimal test fails before the fix and passes after it, with assertions on and off.
• :lucene:core:test --tests 'org.apache.lucene.document.*': 61 classes, 503 tests, 26 skipped, 0 failures, including TestFeatureField (14 of 14, its monotonicity checks included) and the independent reproducer's TestFeatureFunctionPrecisionV1 (4 of 4; it fails on stock main).
• :lucene:queries:test --tests 'org.apache.lucene.queries.intervals.*': 8 classes, 118 tests, 0 failures, including TestIntervalQuery (34), TestIntervals (43) and TestIntervalScorePrecisionV1 (the IntervalQuery ties and zero; it fails on stock main).
• LFV-258's minimal test (sigmoid NaN) passes with this diff (assertions off).
• The staircase and ranking probe with the FeatureField half shadowed: 2,233 distinct scores, 0 zeros, and the larger value ranks first.

(float) (weight * (double) f / (f + (double) pivot)), and the sigmoid's S^a / (S^a + k^a) in double.

Trade-offs

Removes the zeros and ties (the independent reproducer's root-cause shadow did this for all four lines and its tests pass), but monotonicity is not guaranteed by construction, since numerator and denominator both grow, and for the sigmoid Math.pow(f, a) can overflow to Inf/Inf = NaN, the LFV-258 failure. Not recommended for the sigmoid.

weight / (1f + pivot / f), staying in float.

Trade-offs

Gives false zeros when pivot / f overflows float at extreme ratios: 6,333 zero scores in the tie-break's arithmetic check over all legal stored values x 6 pivots.

Say in the javadoc that scores carry an absolute error of up to about weight * 1e-7, that values far below the pivot can tie or score 0, and point to newLinearQuery or newFeatureSort for ordering by small values.

Trade-offs

Leaves the zeros, the ties and an explain() that prints a formula it does not compute.

Method: non-formal first signal, then bounded checks (the study classes it non-formal; the finder tagged it semi-formal).

• An independent model from another family (OpenAI GPT-5.6), asked for numeric-scoring leads, proposed "feature-saturation-sigmoid-cancellation"; its executed probe compared the stock main jar with the documented formulas evaluated in double on 4,000 log-uniform feature values: saturation with pivot 1e3 gave 430 exact zeros and 932 ties, the sigmoid with exponent 4 gave 621 zeros (2026-10-28).
• The finder then ran bounded-exhaustive translation validation of the real core and queries jars: 1,403,738,112 function-level checks of the real jar's score(), bulk scorer and explain() over every legal stored value, 771 pivots, 4 weights and 6 exponents (5,614,952,448 together with three re-implemented fix and oracle forms), 39,144,448 end-to-end hit scores, and every float sloppy frequency for IntervalQuery; cvc5 floating-point queries showed the saturation scores 0.0 exactly when fl(S + k) == k.
• Public-API JUnit tests were written from those thresholds; the one-test minimal repro came later.