A valid LatLon distance query that stops just short of a pole can fail with "invalid longitude NaN"

Before a LatLon distance query runs, Rectangle.fromPointDistance draws a latitude/longitude box around the circle. When the circle, padded by 7 cm, reaches neither pole, it computes the box's half-width in longitude as asin(sin(radDistance) / cos(radLat)) (Rectangle.java:157) with no clamp. The quotient is at most 1 in exact arithmetic, but sin, cos and asin are SloppyMath's fast versions, and SloppyMath.cos is not monotone where it switches table entries. For the circle centred at (0.43945312499999994, 0) with radius 9,958,692.080819247 m, which stops 7 cm short of the north pole, the quotient comes out 1.0000000000000002 (StrictMath: 0.9999999999999999), asin returns NaN, and the Rectangle constructor throws IllegalArgumentException: invalid longitude NaN; must be between -180.0 and 180.0. LatLonPoint.newDistanceQuery therefore fails at search time, and LatLonDocValuesField.newSlowDistanceQuery when built, although their javadoc accepts every valid centre and finite, non-negative radius. No wrong hits and nothing corrupted; the inputs are contrived. It dates from LUCENE-6908 (5.5.0, 2015). Fix: asin(Math.min(1, ...)), the clamp SloppyMath.haversinMeters already uses.

1. LatLonPoint.newDistanceQuery accepts the query: the constructor checks only a non-null field, a finite non-negative radius and the centre (LatLonPointDistanceQuery.java:57-71). At search, createWeight calls Rectangle.fromPointDistance(latitude, longitude, radiusMeters) (:83).
2. fromPointDistance pads the radius by 7 cm, radDistance = (radiusMeters + 7E-2) / EARTH_MEAN_RADIUS_METERS (Rectangle.java:150), and takes the non-pole branch when minLat > MIN_LAT_RADIANS && maxLat < MAX_LAT_RADIANS (:156). For the repro the padded top edge is 7.3e-15 rad (47 nm) below the pole, so this branch runs.
3. double deltaLon = asin(sin(radDistance) / cos(radLat)); (:157), with asin, sin and cos imported from SloppyMath (:31-33). SloppyMath.sin(a) is cos(a - PIO2) (SloppyMath.java:203-205), so the ratio is cos(0.007669903939435452) / cos(0.007669903939428205). The numerator's angle is 7.2e-15 rad larger, so exactly the ratio is below 1.
4. SloppyMath.cos (SloppyMath.java:92-114) rounds the angle to the nearest table point (a step of 2pi/2048; 2049 entries), index = (int) (a * SIN_COS_INDEXER + 0.5) (:100), and adds a 4th-order Taylor correction. The two angles straddle the switch between points 2 and 3, at 2.5 x 2pi/2048 = 0.007669903939428206 rad (0.439453125 degrees). The correction from point 3 lands one ulp higher: 0.9999705864309741 against 0.999970586430974. The documented error, "around 1E-15" (:80), still holds; monotonicity is not promised.
5. The ratio is 1.0000000000000002; SloppyMath.asin returns NaN above 1, as documented (:124-125, code :171-176). minLon and maxLon become NaN, and new Rectangle(...) (Rectangle.java:174-178) rejects them in GeoUtils.checkLongitude (Rectangle.java:88, GeoUtils.java:98-100).

Neighbours pass: radius 1 m smaller gives longitudes -89.997 to 89.997; 1 m larger crosses the pole (-180 to 180); the exact knot latitude 0.439453125 gives -90 to 90; a centre 1e-9 degree north or south also gives a box.

LatLonPoint.newGeometryQuery(INTERSECTS, Circle) becomes a distance query (LatLonPoint.java:333-334) and fails at search. LatLonDocValuesField.newSlowDistanceQuery and LatLonShape.newGeometryQuery with a Circle fail when built (Circle2D.java:417). Read only, not executed: the same call sits behind GeoEncodingUtils.createDistancePredicate (GeoEncodingUtils.java:174), distance sort (LatLonPointDistanceComparator.java:94), NearestNeighbor (NearestNeighbor.java:131) and distance-feature pruning (LatLonPointDistanceFeatureQuery.java:375), there with computed radii.

• Who is hit. Applications that run a LatLon circle query on a user-supplied centre and radius: LatLonPoint.newDistanceQuery, LatLonPoint.newGeometryQuery(INTERSECTS, Circle), LatLonDocValuesField.newSlowDistanceQuery and LatLonShape.newGeometryQuery with a Circle. Default codec, writer and searcher.
• What it takes. A centre latitude within about 1e-14 to 1e-13 degree below one of five latitudes, k x 360/4096 degrees for k = 5, 9, 23, 29, 139 (0.44, 0.79, 2.02, 2.55, 12.22 degrees), together with a radius within about 1e-7 m of "distance to the pole minus 7 cm". The repro's centre is one double (5.6e-17 degree) below 0.439453125; in a scan of the 4,097 doubles at and below each latitude the throwing centres reach at most 5.6e-14, 1.45e-13, 9.7e-14, 9.3e-15 and 1.07e-14 degree below. The southern mirrors fail too (the repro's centre negated throws the same). 0 of 2,000,000 random legal circles throw, drawn as TestGeoUtils does (latitude in [-90, 90], longitude in [-180, 180], radius in [0, 50,000 km)); an independent run with radius in [0, 20,000 km) also found 0 of 2,000,000. The five latitudes lie on LatLonPoint's 32-bit encoding grid (step 4.2e-8 degree) and the exact knot passes, so a centre decoded from an indexed LatLonPoint cannot trigger it (the repro's indexed point decodes to 0.43945308309048414, which gives a box); the repro itself passes the raw double as the query centre.
• What happens. That one query fails with an IllegalArgumentException that blames a longitude the caller never passed, before any index data is read.
• What does not happen. No wrong hits, no corruption, no hang, no effect on other searches or on the index (a larger query that contains such a clause, for example a BooleanQuery, fails as a whole).
• Why fix it anyway. It is a one-line guard, and the unclamped ratio is fragile in general: with Math.sin/Math.cos instead of SloppyMath the same formula goes above 1 in 450 of 1,280,000 near-pole test cases (OpenJDK 25, Linux aarch64, JIT intrinsics).

Public API only: IndexWriter with the default IndexWriterConfig, ByteBuffersDirectory, LatLonPoint, IndexSearcher.count. The inputs are exact doubles (bits 0x3fdc1fffffffffff and 0x4162fea48296123f); the seed does not matter (10 of 10 random-seed iterations fail). The code is pure Java double arithmetic; a run in the interpreter with SVE off (-Xint -XX:UseSVE=0) fails the same way; x86 was not run.

As the regression test: this same test, for example as a method in TestLatLonPoint.

cp TestDistanceQueryNearPoleNaN.java $LUCENE/lucene/core/src/test/org/apache/lucene/document/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.document.TestDistanceQueryNearPoleNaN -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

TestDistanceQueryNearPoleNaN > testLegalCircleEndingJustShortOfPole FAILED
    java.lang.AssertionError: expected 1 hit for a legal circle, got java.lang.IllegalArgumentException: invalid longitude NaN; must be between -180.0 and 180.0
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:B3F9BEAC2208C74F]:0)
        at org.apache.lucene.document.TestDistanceQueryNearPoleNaN.testLegalCircleEndingJustShortOfPole(TestDistanceQueryNearPoleNaN.java:44)

        Caused by:
        java.lang.IllegalArgumentException: invalid longitude NaN; must be between -180.0 and 180.0
            at org.apache.lucene.geo.GeoUtils.checkLongitude(GeoUtils.java:100)
            at org.apache.lucene.geo.Rectangle.<init>(Rectangle.java:88)
            at org.apache.lucene.geo.Rectangle.fromPointDistance(Rectangle.java:178)
            at org.apache.lucene.document.LatLonPointDistanceQuery.createWeight(LatLonPointDistanceQuery.java:83)
            at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:978)
            at org.apache.lucene.search.ConstantScoreQuery.createWeight(ConstantScoreQuery.java:146)
            at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:978)
            at org.apache.lucene.search.IndexSearcher.count(IndexSearcher.java:527)

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF, assertions on; identical with -Ptests.asserts=false

# the arithmetic (stock 645d4fe14a20 core jar)
SloppyMath.sin(rd) = 0.9999705864309741
SloppyMath.cos(radLat) = 0.999970586430974
sloppy quotient = 1.0000000000000002  asin -> NaN
StrictMath quotient = 0.9999999999999999  asin(deg) -> 89.99999914622636

• Contract. LatLonPoint.newDistanceQuery: "@param radiusMeters maximum distance from the center in meters: must be non-negative and finite", "@throws IllegalArgumentException if {@code field} is null, location has invalid coordinates, or radius is invalid." (LatLonPoint.java:290-294); the same for LatLonDocValuesField.newSlowDistanceQuery (LatLonDocValuesField.java:204-208). fromPointDistance checks only the centre (Rectangle.java:145-146): "Compute Bounding Box for a circle using WGS-84 parameters" (:142).
• The guard shows the intent. The non-pole branch (:156) appears intended to keep asin's argument at most 1 (no comment says so, but in exact arithmetic that is what the guard ensures); rounding breaks that at the boundary.
• Lucene clamps the same hazard elsewhere. SloppyMath.haversinMeters: asin(Math.min(1, Math.sqrt(sortKey * 0.5))) (SloppyMath.java:58). spatial3d Tools.safeAcos: "Java acos yields a NAN if you take an arc-cos of an angle that's just a tiny bit greater than 1.0, so here's a more resilient version" (Tools.java:27-38). XYRectangle.fromPointDistance rounds its box up "to avoid numerical errors" (LUCENE-9243, XYRectangle.java:79-81). The 7 cm pad in this method (Rectangle.java:149-150, comment "LUCENE-7143") is the same kind of guard.
• Tests. TestGeoUtils.testRandomCircleToBBox calls fromPointDistance on random legal circles with radii up to 50,000 km and expects a box every time (TestGeoUtils.java:34-51); it even carries "TODO: randomly quantize radius too, to provoke exact math errors?" (:49). No test expects an exception for a valid circle.

The strongest case for "intended" (the study's intended-behaviour verifier and cross-model reviewers): SloppyMath trades accuracy for speed on purpose ("This is performance sensitive, check SloppySinBenchmark and RectangleBenchmark", SloppyMath.java:183); its asin is specified to return NaN above 1 and TestSloppyMath asserts asin(2) is NaN (TestSloppyMath.java:95); Circle is @lucene.experimental and says "For very large circles or circles near poles, consider using polygons instead" (Circle.java:33); the Rectangle class notes that rectangles near poles are distorted (Rectangle.java:63-64); and the witness needs nanometre-scale tuning.

Why it fails: SloppyMath keeps its own contract (one ulp is within "around 1E-15"), which is exactly why the caller must keep asin's argument in range, as haversinMeters does. The pole notes come from a docs-only change (#14534) and are accuracy advice, not a precondition that permits an exception. LatLonPoint.newDistanceQuery is not experimental and documents no such case. A Math.min leaves the speed trade-off untouched. The argument supports low priority, not intended behaviour; every verifier rated it weak. Verdict: not intended, a defect. Not security: a clean exception on one query.

Fix A was validated through gradle in a scratch worktree of 645d4fe14a20. B is an alternative that was not run; C is not recommended.

Same idiom as SloppyMath.haversinMeters (SloppyMath.java:58). It covers every caller of fromPointDistance: distance and Circle queries, the distance predicate, distance sort, nearest-neighbour and distance-feature pruning.

--- a/lucene/core/src/java/org/apache/lucene/geo/Rectangle.java
+++ b/lucene/core/src/java/org/apache/lucene/geo/Rectangle.java
@@ -154,7 +154,9 @@ public class Rectangle extends LatLonGeometry {
     double maxLon;
 
     if (minLat > MIN_LAT_RADIANS && maxLat < MAX_LAT_RADIANS) {
-      double deltaLon = asin(sin(radDistance) / cos(radLat));
+      // the quotient is <= 1 in exact arithmetic, but rounding can push it just above 1 (asin would
+      // return NaN); clamp it, as SloppyMath.haversinMeters does
+      double deltaLon = asin(Math.min(1, sin(radDistance) / cos(radLat)));
       minLon = radLon - deltaLon;
       if (minLon < MIN_LON_RADIANS) {
         minLon += 2d * PI;

Trade-offs

• Conservative: SloppyMath.asin(1) is exactly pi/2 (SloppyMath.java:172-173), at least the exact half-width (89.99999914622636 degrees here, 90.0 with the clamp), so the box grows by a hair and still contains the circle.
• The box is only a pre-filter; LatLonPointDistanceQuery then checks each candidate's distance (LatLonPointDistanceQuery.java:115, :187), so hits do not change.
• Cost: one Math.min per box (not measured). Low-risk rather than proven risk-free: no regression in the validation suites listed under "What the checks validated", which are not the full test suite.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF, 2026-11-01: without the fix the repro fails with assertions on and off; with it, it passes with assertions on and off.
• ./gradlew :lucene:core:test --tests 'org.apache.lucene.geo.*' --tests 'org.apache.lucene.document.TestLatLon*' --tests 'org.apache.lucene.search.TestLatLon*' --tests org.apache.lucene.search.TestNearest: 39 classes (all 17 geo test classes including TestGeoUtils, TestCircle, TestCircle2D; 19 TestLatLon* document classes; TestLatLonPointQueries, TestLatLonDocValuesQueries, TestNearest), 433 tests, 0 failures, 26 skipped.
• ./gradlew :lucene:core:check -x test with the fix and the repro as a new test file: passes (google-java-format, ecj lint, forbidden APIs). The full Lucene test suite was not run.
• Earlier: with the clamp shadowed into the stock jars, the finder's and the independent verifier's tests (including a 49M-pair scan) pass, the history probe finds 0 failures, and eight upstream geo test classes pass as without the fix (116 tests).

Compute q = sin(radDistance) / cos(radLat); if q >= 1, use the full longitude range as the pole branch does (Rectangle.java:166-172).

Trade-offs

• Also safe, but the box spans every longitude instead of a half-width of 90 degrees, so a few more candidates are checked.
• Slightly more code than A.

Replace the SloppyMath calls with StrictMath or Math.

Trade-offs

• Math.sin/Math.cos also push the same ratio above 1 (450 of 1,280,000 near-pole cases on Linux aarch64), so a clamp is still needed.
• The sloppy versions are kept here for speed (SloppyMath.java:181-183).

Method: bounded-exhaustive translation validation of the stock main jar (study unit on SloppyMath and geo distance error bounds, main a9779ef3c193, 2026-10-27).

• Step 1: a scan of every SloppyMath.cos table switch in [0, pi] for the two doubles on either side, looking for a cosine that goes up as the angle grows: 12 such switches.
• Step 2: around each of the 512 switches in [0, pi/2], an exhaustive window scan of the ratio sin(radDistance) / cos(radLat) at Rectangle.java:157 for circles that stay in the non-pole branch, evaluated with the same public SloppyMath.sin/SloppyMath.cos calls; any ratio above 1 is a witness, and the first one at each switch was replayed through Rectangle.fromPointDistance. The south side was not scanned (it is the mirror image).
• SMT encodings of the same branch (QF_FP, with Z3 4.12.2 and cvc5 1.4.2) returned unknown or timed out, so the witness comes from the executed scans, not from a solver model.
• The finder then wrote a public-API JUnit with controls and the clamp check; an independent verifier found its own witness with its own scan (487 failing legal pairs in 49M). The one-test minimal repro was reduced from them.