FeatureField.newSigmoidQuery scores NaN when pivot^exp leaves the double range

FeatureField.newSigmoidQuery(field, feature, weight, pivot, exp) documents the score weight * S^a / (S^a + pivot^a) and accepts any finite pivot and exponent above 0. Its SigmoidFunction evaluates it as weight * (1 - pivot^a / (S^a + pivot^a)) with pivot^a precomputed once per query as a double. When pivot^a overflows (above about 1.8e308) every document scores Inf/Inf = NaN; when it underflows to 0 and S^a does too, the score is 0/0 = NaN. With pivot 2^20 and exponent 60, a document with S = pivot, documented to score 0.5, scores NaN, silently. Since 10.2.0 NaN hits rank ahead of correctly scored ones; older releases return placeholder hits or throw an assertion from the collector. Every release since 7.4.0 and main are affected, and IntervalScoreFunction (behind the 4-argument IntervalQuery) copies the arithmetic. Scoring as weight / (1 + (pivot / S)^a) removes it.

1. SigmoidFunction stores pivotPa = Math.pow(pivot, a) once per query (FeatureField.java:495, :500 at 645d4fe14a20).
2. Each document scores (float) (weight * (1 - pivotPa / (Math.pow(f, a) + pivotPa))) (:531), where weight is the scorer's boost: newSigmoidQuery passes a weight other than 1 through a BoostQuery (:706-709). The comment above it (:528-530) explains the rewrite of f^a / (f^a + k^a): "to make sure it doesn't decrease with f in spite of rounding".
3. Overflow. If pivot^a rounds past Double.MAX_VALUE (about 2^1024), pivotPa is +Infinity; the denominator is +Infinity for every finite f^a, so every document that has the feature scores 1 - Inf/Inf = NaN, whatever its value.
4. Underflow. If pivot^a is below about 2^-1074, pivotPa is 0; documents whose f^a also underflows score 1 - 0/0 = NaN, the others score weight.
5. An SMT model of the final expression (cvc5, floating-point theory, with the two powers as independent non-negative doubles) finds no other way to NaN. It does not model Math.pow or how FeatureField encodes values.
6. The bulk scorer (:540-550) and explain() (:556-558) use the same arithmetic, and FeatureQuery passes the scorer straight into a TermScorer (FeatureQuery.java:135-137); nothing on the way to the collector checks for NaN.
7. newSigmoidQuery rejects only pivot <= 0, exp <= 0 and non-finite values (FeatureField.java:700-705). The weight is capped at 64 (MAX_WEIGHT, :576-582); the exponent is not.

• One document, S = pivot = 2^20 = 1,048,576, weight 1, exponent 60: pivot^60 = 2^1200 overflows and the hit scores NaN instead of 0.5, with assertions on and off.
• Boundary at the same pivot: exponent 51 (2^1020, finite) gives 0.5, exponent 52 (2^1040) gives NaN.
• First failing float pivot, measured on the real scorer (S set to the pivot as FeatureField stores it; the overflow does not depend on S), close to 2^(1024/exp): exponent 9: 1.7804262e34; 20: 2.586639e15; 35: 6.416196e8; 50: 1,462,494.8; 60: 137,270.94; 100: 1,209.3365. Exponent 8 gives no NaN for any pivot up to Float.MAX_VALUE, because a float is below 2^128 and 128 * 8 = 1024, and none through underflow either: a stored feature value is at least 2^-126 (FeatureField.java:162-171), so S^8 >= 2^-1008 > 0.
• A finite wrong value right next to the overflow: where pivot^a is finite but S^a + pivot^a overflows (about 2^1023 <= pivot^a < 2^1024; at pivot 2^20, exponents from about 51.15 to just under 51.2, measured at 51.16, 51.18 and 51.19) the score at S = pivot is 1.0 instead of 0.5.
• Ranking: body:lucene as MUST plus newSigmoidQuery("f", "x", 1, 2^20, 60) as SHOULD, over doc0 "lucene apache apache" with feature value 1.0 (documented boost 1/(1+2^1200), about 0) and doc1 "lucene lucene lucene" without the feature: the result is [doc=0 score=NaN, doc=1 score=0.13022968]. With the fix doc1 ranks first.
• Underflow (study test): pivot 2^-20 with exponent 60 over 4 documents gives 3 NaN scores, and the best document (score 1.0) is ranked last.

IntervalScoreFunction.SigmoidFunction (IntervalScoreFunction.java:109-131, :117, :128), behind new IntervalQuery(field, source, pivot, exp) (IntervalQuery.java:97-99), has the same arithmetic and the same checks (IntervalScoreFunction.java:33-41). Sloppy frequencies are small, so it needs exponents in the hundreds: with pivot 4 and exponent 600, all 4 documents score NaN.

• Who is exposed. Users of the sigmoid feature function (FeatureField is @lucene.experimental) who want the boost to switch on sharply around the pivot, for example to approximate a popularity threshold; a steep exponent is the documented way to get that ("higher values make the function grow slower before 'pivot' and faster after 'pivot'", FeatureField.java:689-690).
• No special setup. One document, the default IndexWriterConfig and codec and a plain IndexSearcher are enough; no special codec or concurrency setting was needed. The score depends only on the query's parameters and the document's feature value (FeatureField.java:527-532).
• Silent. With assertions off there is no exception or warning. TopScoreDocCollector's javadoc says NaN values "are not valid scores. This collector will not properly collect hits with such scores" (TopScoreDocCollector.java:30-31).
• What users see depends on the release (4-document probe; the arithmetic never changed):
  • 7.4.0-10.0.0: with -ea, search() throws an AssertionError in TopScoreDocCollector; with -da, TopDocs holds placeholder hits (doc 2147483647, score -Infinity) in place of real documents;
  • 10.1.0: only 1 of 4 hits;
  • 10.2.0 to main: every hit, with NaN scores ranked ahead of finite ones (overflow: all NaN, in doc-id order);
  • explain() reports NaN on every release.
• Downstream. Elasticsearch's and OpenSearch's rank_feature sigmoid and Solr's {!rank function=sigm} pass the request's pivot and exponent to newSigmoidQuery unchanged, so a search client can hit it with one query parameter. Their intervals queries use the 2-argument IntervalQuery, so the interval copy needs an application that calls the 4-argument constructor itself.
• How likely. Rare: the exponents involved are far above typical ones (Lucene's tests use 0.6 and 2; Elasticsearch documents the exponent as "typically in [0.5, 1]"), and nobody has reported it since 2018. Priority low.

ByteBuffersDirectory, the default IndexWriterConfig, a plain IndexSearcher; nothing random, one thread. 2^20 is a positive normal float that FeatureField stores exactly, and the fixed formula gives exactly 0.5 for it. The test asserts the documented value (weight/2 at S = pivot) rather than just "not NaN", so the failure reads expected:<0.5> but was:<NaN>.

It covers the overflow case of the FeatureField copy, the common one. The same root cause also gives NaN through underflow (tiny pivots with tiny feature values) and through the 4-argument IntervalQuery (for example pivot 4 with exponent 600); those were executed by the study and are described in the text, not added as more tests.

Regression test: this same test. It passes with the recommended fix and fits as one more method of the existing TestFeatureField.

cp $REPRO/TestFeatureFieldSigmoidOverflow.java $LUCENE/lucene/core/src/test/org/apache/lucene/document/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestFeatureFieldSigmoidOverflow -Ptests.seed=DEADBEEF -Ptests.asserts=false
# assertions on (drop -Ptests.asserts=false): the same failure

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF (assertions on, and again with -Ptests.asserts=false)
TestFeatureFieldSigmoidOverflow > testScoreAtPivotWithSteepExponent FAILED
    java.lang.AssertionError: expected:<0.5> but was:<NaN>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:7964E766F43B5E25]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:577)
        at org.junit.Assert.assertEquals(Assert.java:701)
        at org.apache.lucene.document.TestFeatureFieldSigmoidOverflow.testScoreAtPivotWithSteepExponent(TestFeatureFieldSigmoidOverflow.java:41)
BUILD FAILED

# standalone runner, main 645d4fe14a20 jars, seed DEADBEEF: -ea and -da both tests_run=1 failures=1
# release jars 10.5.2 and 10.6.0, -ea and -da: FAIL 1/1, same message

# with the recommended fix, gradle, -Ptests.seed=DEADBEEF
:lucene:core:test (SUCCESS): 1 test                    # the minimal test, -Ptests.asserts=false
:lucene:core:test (SUCCESS): 499 tests, 26 skipped     # --tests 'org.apache.lucene.document.*'
:lucene:queries:test (SUCCESS): 117 tests              # --tests 'org.apache.lucene.queries.intervals.*'

• The contract is exact. newSigmoidQuery: the score is "weight * S^a / (S^a + pivot^a)", pivot is the "feature value that would give a score contribution equal to weight/2, must be in (0, +Infinity)", exp "must be in (0, +Infinity)" (FeatureField.java:680-693). Class javadoc: "values are in the 0..1 range and 0.5 is obtained when S and k are equal" (:76-77); with a weight the formula's range is [0, weight], so the 0..1 wording holds for weight 1, as in the repro. IntervalQuery: "Scores are bounded to between 0 and 1" (IntervalQuery.java:56) and pivot is "the sloppy frequency value at which the score will be 0.5" (:92).
• Lucene treats NaN as invalid. TopScoreDocCollector.java:30-31; the test framework's AssertingScorer asserts !Float.isNaN(score) (AssertingScorer.java:129).
• Upstream's own oracle. TestFeatureField.doTestSimScorer (:279-287) asserts Float.isFinite(maxScore) // used to compute max scores and that the score never decreases with the value. It is run for SigmoidFunction(20f, 0.6f) (:275-277); run on (2^20, 60) and (1e16, 20) it fails with max score NaN.
• Robustness was a goal. The code comment (FeatureField.java:528-530) says the formula was rewritten so the score does not decrease "in spite of rounding".
• No limit or NaN in the recorded and archived material. No javadoc, test, CHANGES or MIGRATE entry or commit message at 645d4fe14a20, and no discussion the searches found, limits the exponent or presents NaN as intended. In the archived LUCENE-8197 review (#9244) reviewers asked for documented and validated parameter ranges, and the committed code has the (0, +Infinity) checks; no archived comment discusses pivot^exp overflowing.
• Tests never reach the region. Every upstream sigmoid test uses exponent 0.6 (TestFeatureField) or 2 (TestIntervalQuery.java:439).

The strongest case, built by the cross-model reviews and by the devil's-advocate verifier: FeatureField is @lucene.experimental, its constants "typically need training" (FeatureField.java:84), and exponents of 50 to 150 are unrealistic, so a committer could say the function was never meant for them.

Why it fails: @lucene.experimental is about API stability ("might change in incompatible ways in the next release"), not about returning invalid scores; the domain is documented as (0, +Infinity) and validated, so these inputs are legal; the authors capped the one parameter they found unreasonable (MAX_WEIGHT) but not the exponent; the javadoc describes high exponents as the way to make the function steeper; and an algebraically equal one-line rewrite removes the failure, at the price of one more division per document. The realistic-exponent point lowers the priority, not the reality, and maintainers may reasonably prefer validating the parameters (option B) to changing the formula. Verdict: not intended, a defect.

Option A was built and run through gradle in both copies. B (validate the exponent) is a possible addition that turns NaN into an error, C (document a range) leaves NaN, and D (guard in the collector) treats the symptom. Workaround without a fix: use exponents up to 8, which never fail. With a larger exponent keep pivot^exp well inside the double range on both sides: no overflow (exp * log2(pivot) well below 1024) and, for a pivot below 1, no underflow (pivot 2^-14 with exponent 100 already underflows).

Drop pivotPa from FeatureField.SigmoidFunction and IntervalScoreFunction.SigmoidFunction and compute the same function as weight / (1 + (pivot / S)^a) (2 files, +7/-10). For positive finite inputs pivot / S lies in [0, +Inf], so does its power, 1 + x lies in [1, +Inf] and the result in [0, weight]: no NaN. At S = pivot it is exactly weight / 2. It stays non-decreasing in S, the property the old comment protects: the division is monotone, Math.pow is specified as semi-monotonic, and 1 + x and w / x are monotone.

// FeatureField.SigmoidFunction.doScore; the same line in IntervalScoreFunction.SigmoidFunction
return (float) (weight / (1 + Math.pow((double) pivot / f, a)));

Trade-offs

• Ordinary scores barely change, but not bit for bit. A copy of the old and new expressions in plain double arithmetic (not the real scorer) on 1,320,480 points (7 pivots from 0.01 to 1e6, exponents 0.5 to 8, weights 1 and 3, S from Float.MIN_NORMAL to 1e30 in 1% steps): 73% bit-identical; where the old score is at least 1e-6 * weight, at most 1 ulp apart. The largest absolute difference, 9.1e-13 (2^-40), is a 1-ulp difference at a score of about 1e-5 (weight 1, pivot 4.5, exponent 2, S 0.014), and every score above about 1e-5 is bit-identical. Where the old form rounds a tiny score to 0 by cancellation, the new form keeps it (at most 3.3e-16); that gives the largest ulp distance. The new form gave no NaN, nothing outside [0, weight] and no decreasing step on the grid. A ranking that depends on exact score values can still change, so it is worth a CHANGES.txt line.
• Cost. One more division per scored document (two instead of one) and no precomputed constant; no benchmark was run.
• Review. No reviewer has read the diff line by line yet.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, seed DEADBEEF: the minimal test fails before the fix (assertions on and off) and passes after it (-Ptests.asserts=false).
• :lucene:core:test --tests 'org.apache.lucene.document.*': 60 classes, 499 tests, 26 skipped, 0 failures, including the upstream TestFeatureField (14 of 14).
• :lucene:queries:test --tests 'org.apache.lucene.queries.intervals.*': 7 classes, 117 tests, 0 failures, including TestIntervalQuery (34) and TestIntervals (43); the study's interval test (pivot 4, exponent 600), which fails on unfixed main, passes.
• Earlier checks of the same diff: the study's and the independent reproducer's tests pass with it; the exhaustive sweep of the score functions (6.8e9 checks) shows no sigmoid violation; the tie-break's ranking probe and boundary tests pass.

In newSigmoidQuery and IntervalScoreFunction.sigmoidFunction, throw IllegalArgumentException when Math.pow(pivot, exp) is +Infinity or 0, or reject exponents above a documented bound.

Trade-offs

Turns silent NaN into an error that downstream engines would surface to the client. Leaves the finite band where S^a + pivot^a overflows (score weight instead of weight / 2) unless the bound is tighter, and rejects parameters option A handles. An exponent bound is a judgement call; none is documented today. Not run.

Say in the javadoc which exponents are safe for a given pivot.

Trade-offs

Leaves NaN scores for anyone who misses the note, and contradicts the documented (0, +Infinity) domain unless that changes too.

Make top-k collection tolerate NaN scores.

Trade-offs

TopScoreDocCollector already documents that it does not handle NaN; the related collector-side issue #3347 (LUCENE-2271) has been open since 2010. Fixing the producer is local and cheap.

Method: bounded-exhaustive translation validation of the real jar (the study's "semi-formal" tier: the first signal came from executing the stock code against exact properties).

• A harness drove every FeatureField score function on the stock main jar over every integer frequency from 1 to 65,535 plus Integer.MAX_VALUE and Float.MAX_VALUE (for the sigmoid: 287 pivots covering every float binade, 21 exponents, 2 weights), about 6.8e9 checks against seven properties: no NaN or negative score, non-decreasing in the value, within the max-score bound, a finite max score, bulk scoring equals single scoring, explanation equals score, at most the weight. Only the sigmoid produced NaN (as a score and as its max score). The only other failures were +Infinity max scores of the linear and log functions at a feature value of Float.MAX_VALUE, which were not reported.
• cvc5 queries over the final score expression (floating-point theory, the two powers modelled as independent non-negative doubles) then showed NaN is possible only if pivot^a = +Inf, or pivot^a = 0 and S^a = 0 (the "only if" query unsat, both vacuity variants sat), and a region probe mapped which accepted (pivot, exponent) pairs fail.
• The JUnit repro was written by hand from those parameters; the one-test minimal repro came later.