On WGS84, the geo3d outside distance to a small circle is +Infinity in most directions, and newOutsideDistanceSort sorts those documents last

GeoCircle.computeOutsideDistance (ARC or LINEAR style) asks the circle's plane for the distance from a point to the circle's edge, Plane.arcDistance or Plane.linearDistance. Those methods look for the nearest edge point on one line: where the circle's plane meets the plane through the origin, the plane's normal and the point. On a sphere that line always passes through the centre of the circle's ring (on the radius through the circle's centre), so it always crosses the edge. On WGS84, or any non-spherical model, the circle plane's normal follows the surface normal, tilted 0.19° from the centre direction at latitude 40, so the line always runs through a point 21 km north of the centre. For a circle smaller than that offset the line misses the edge in most directions, the intersection search finds no point, and the method returns its start value, +Infinity: for 12 points 3 km from a 1 km circle at 40°N 74°W, one every 30°, only due north and due south are finite, the other 10 are +Infinity. At 5° steps 56 of 72 are +Infinity; within about 15° of north and 20° of south the line still crosses the circle, and the values are finite but wrong (up to 2,430 m where about 2,000 m is expected). Geo3DDocValuesField.newOutsideDistanceSort uses this distance, so 10 of 12 documents 21 km from the centre get +Infinity, the value its javadoc reserves for documents without the field, and sort last. Larger circles get finite but wrong values (at latitude 40, points 2 km outside: up to 9.9% off at 50 km, 2.3% at 100 km, 0.6% at 200 km). Which documents match is not affected. The code dates from LUCENE-6578 (5.3.0, 2015; the wrong value is Double.MAX_VALUE before 6.1.0); the sort is affected since it was added in 6.1.0.

• Outside distance: GeoOutsideDistance.computeOutsideDistance(style, point) is 0.0 inside a shape and "the distance from a point to the closest outside edge" outside it. DistanceStyle.ARC measures the angle, LINEAR the straight-line (chord) distance, both in planet units (multiply by the mean radius for meters). ARC is the angle at the planet's centre, not a geodesic surface distance.
• Circle plane: a geo3d circle (GeoStandardCircle, from GeoCircleFactory.makeGeoCircle) is the part of the planet surface on one side of a plane; its edge is where the plane cuts the surface. On WGS84 that cut is a slightly elliptical ring.
• F: the point of the circle plane nearest the planet's centre. It lies on the plane's normal through the origin.

1. GeoStandardCircle.outsideDistance is one call: distanceStyle.computeDistance(planetModel, circlePlane, x, y, z) (GeoStandardCircle.java:168-172), which ArcDistance and LinearDistance forward to Plane.arcDistance / Plane.linearDistance (ArcDistance.java:52-60, LinearDistance.java:52-60).
2. Plane.arcDistance (Plane.java:600-636) builds perpPlane = new Plane(this.y * z - this.z * y, this.z * x - this.x * z, this.x * y - this.y * x, 0.0) (:612-614), the plane through the origin, the normal n and the point p; intersects it with the circle plane on the surface (findIntersections, :622); starts from double minDistance = Double.POSITIVE_INFINITY; (:625); and returns the minimum over the intersection points (:627-635). linearDistance is the same (:727-761).
3. The two planes meet in a line through F. On a sphere F is the centre of the circle's ring, on the radius through the circle's centre, so every such line crosses the circle.
4. On WGS84 the circle plane goes through the two meridian points at latitude ± the radius (GeoStandardCircle.java:71-102), so its normal follows the surface normal. At latitude 40 the normal is tilted 0.19° and F is 21,078 m north of the centre (7.3 km at latitude 10, 13.7 km at 70, 0.7 km at 89, 0 at the equator).
5. Due north and south the point lies in the meridian plane, which holds n, F and the centre, so the line crosses the circle and the answer is right. At the other ten 30° bearings the line from F toward a point 3 km from the centre misses a 1 km circle, findIntersections returns no point, and +Infinity is returned. Bearings within about 15° of north or 20° of south still cross the circle, at the wrong points (finite, up to 2,430 m where about 2,000 m is expected, at 5° steps).
6. A circle larger than the offset contains F, so the line always crosses it, but at the wrong points: finite, wrong values.
7. Geo3DDocValuesField.newOutsideDistanceSort builds the circle with Geo3DUtil.fromDistance (Geo3DDocValuesField.java:249-258), and its comparator calls computeOutsideDistance(DistanceStyle.ARC, ...) per doc value (Geo3DPointOutsideDistanceComparator.java:101, :143-147) and returns it times the mean radius (:125). +Infinity stays +Infinity.
8. GeoExactCircle uses the same Plane.arcDistance per slice (GeoExactCircle.java:245-258): small exact circles fail the same way. Its remaining +Infinity at the slice joints due north and/or due south (at most radii tested; on stock the 1 km exact circle's due-north value is finite, 1,998.6 m) is a separate bug (LFV-737).

ARC distances of points 2 km outside the circle, 36 bearings, against a reference edge from bisecting the circle's own isWithin along 1,440 bearings (a high-resolution sampled reference; main a9779ef3c193):

CLARKE_1866 is the same (28, 18, 8, 0). 1 km circle, 72 bearings: 56 of 72 +Infinity at latitudes 10, 40 and 70; none at the equator or at latitude 89 (up to 12% off there). These are sampled points: the antipode of the circle's centre gives +Infinity on the sphere too (a degenerate search plane, see Suggested fix).

• Who is hit. Code that calls GeoCircle.computeOutsideDistance with ARC or LINEAR (LINEAR_SQUARED squares the same linearDistance, read) on WGS84, CLARKE_1866 or any non-spherical PlanetModel, and applications that sort with Geo3DDocValuesField.newOutsideDistanceSort on such a model. In the sampled cases +Infinity appears when the radius is smaller than F's offset (0 at the equator, 7.3 km at latitude 10, 21 km at 40, 13.7 km at 70). The Geo3DPoint constructors without a PlanetModel argument use WGS84; newOutsideDistanceSort takes the model as an argument since 8.5.0 and used WGS84 unconditionally in 6.1.0-8.4.x. Small GeoExactCircles too.
• Effect. Silent: +Infinity, or a wrong finite value, and no error. In a sort, documents a few km away rank after farther ones due north or south, tied with documents that have no location; with a small page size they fall off the first page.
• Not affected. Which documents match: Geo3DPoint.newDistanceQuery decides membership with isWithin, and in an executed probe a filter on the same circle returned exactly the inside documents while adding the sort left the hit set unchanged. Nothing is written to the index; it is query-time only, for any index size. On PlanetModel.SPHERE and at the equator the sampled GeoStandardCircle points are correct (at the equator at most 0.2% off, the same as with the fix; GeoExactCircle keeps its separate joint +Infinity there, LFV-737); that is sampled evidence, not a proof.
• Scale. No special data: one circle and one point.
• How common. spatial3d is an opt-in module and outside-distance sorts are a rarely used API: no upstream test calls newOutsideDistanceSort, and a read-only code search found no use in Elasticsearch, OpenSearch or Solr. No report of it was found (see Prior reports).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<2000.0> but was:<Infinity>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:555)
       org.junit.Assert.assertEquals(Assert.java:685)
       org.apache.lucene.spatial3d.geom.TestGeoCircleOutsideDistanceWGS84.testOutsideDistanceToSmallCircle(TestGeoCircleOutsideDistanceWGS84.java:30)

• GeoOutsideDistance (GeoOutsideDistance.java:20-21): "Implemented by Geo3D shapes that can compute the distance from a point to the closest outside edge."; :32-33: "A return value of 0.0 should be returned for points inside of the shape."
• Plane.linearDistance (Plane.java:703-704, :717-718): "This is defined as the distance from the given point to the nearest intersection of this plane with the planet surface." The circle's boundary points lie on the plane (max |evaluate| 1.0e-12), yet it returns Infinity at 6 of 8 sampled bearings (executed on a9779ef3c193).
• Geo3DDocValuesField.newOutsideDistanceSort (:232-237, not marked experimental): "This sort orders documents by ascending outside distance from the circle. Points within the circle have distance 0.0. The value returned in FieldDoc for the hits contains a Double instance with the distance in meters." and "If a document is missing the field, then by default it is treated as having Double#POSITIVE_INFINITY distance (missing values sort last)."
• Offered, approximately, on ellipsoids. The circle factory and the sort take any PlanetModel, and the sampled WGS84 circles of 100 km to 2,000 km and the sampled equatorial circles come out within 2.3%: approximate, not undefined. In #8955 (2017-07-25) the geo3d maintainer wrote: "arcDistance is defined well for ellipsoids but surfaceDistance requires an iterative convergent loop. [...] Since surfaceDistance is prohibitively expensive to calculate for all but a sphere, an approximation is required either way, and the GeoCircle is fitted specially to the ellipsoid in order to be a good approximation for WGS84." That contrasts the arc-distance metric with true surface distance, not Plane.arcDistance's edge search, but it calls the WGS84 circle a good approximation, which +Infinity is not.
• Tests. The only circle outside-distance asserts are on PlanetModel.SPHERE with a circle on the equator (TestGeoCircle.java:67-79), where the offset is 0; no test calls newOutsideDistanceSort. On #8267 (the sort's feature issue, still open) the last maintainer comment lists tests as still needed.
• History. Plane.arcDistance has kept the plane through the origin since LUCENE-6578 (2015); later commits changed only the sentinel (LUCENE-7288: Double.MAX_VALUE to +Infinity) and the formatting.

The strongest case for "intended": the geom API is @lucene.experimental; the circle's constructor says "In an ellipsoidal world, cutoff distances make no sense, unfortunately. Only membership can be used to make in/out determination." (GeoStandardCircle.java:66-67); the factory calls the shape "a circle in spherical planets, otherwise ... an ellipse"; open #9623 says the standard circle "is only accurate when the PlanetModel is a sphere"; in #9019 (2017) the maintainer wrote "GeoOutsideDistance we can't really do at the moment" about the exact circle; and, most directly, in #8955 (2017-07-21) a contributor proposed deciding whether a shape intersects a circle by comparing the shape's outside distance to the circle's centre with the radius, and the maintainer replied: "You are welcome to try to implement GeoArea for GeoCircles but remember that your distance computation does not work in WGS84, only in SPHERE." and "The 'GeoCircle' is by definition the intersection of a specific plane with the unit ellipsoid, so it's in fact an ellipse and distance calculations are exorbitantly expensive."; the contributor answered "Ok, I did not know that distance calculation is only support for unit sphere."

Why it fails: "experimental" allows API changes, not +Infinity for a point 2 km away, and the user-facing sort is not experimental. The constructor comment came with LUCENE-6487 (560f3a1a010, 2015), before outside distance existed, next to deleted membership-cutoff fields: it is about membership. #9623 is about small membership errors, and its remedy, GeoExactCircle, fails here too. The #9019 remark preceded the exact circle's outside distance, committed two days later, and concerns another shape. The #8955 exchange is about using the distance from the circle's centre as an intersection test, which cannot work on an ellipse because its edge is not at one distance from the centre; it does not say that outside distances may be +Infinity, and four days later the same maintainer called the WGS84 circle "a good approximation". The contracts, the 100 km and equatorial results and the sort's WGS84 default (hard-coded in 6.1.0-8.4.x) show that outside distance is offered on WGS84. An ellipse still has a nearest edge point. The verifiers and the adjudication rejected the intended reading; the cross-model reviews (OpenAI GPT-5.6; the models could not run code, a separate run re-ran the repro, which failed) and the cross-model fact check of this doc rated it moderate and agreed with the root cause. Expect a maintainer to raise the #8955 remark.

Two fixes were built and validated with gradle in a fresh scratch worktree at a9779ef3c193 (removed afterwards). Option B fixes the shared method; option A only the standard circle. Either way, add a CHANGES entry under Bug Fixes and WGS84 tests of computeOutsideDistance and newOutsideDistanceSort. The exact circle's slice-joint bug (LFV-737) needs its own fix.

Replace the plane through the origin with the plane that contains the normal n and the point p and passes through c, the centre of the ellipse in which the plane cuts the planet: the point of the plane where the surface normal is parallel to n, c = t * (xyScaling² A, xyScaling² B, zScaling² C) with t = -D / (n · M⁻¹ n). Its line in the plane goes through c, so it crosses the ellipse whenever the plane meets the surface, with one degenerate exception: when p - c is parallel to n, the normal n × (p - c) is zero, findIntersections finds no point and +Infinity remains. For the 1 km circle that is a single point on the far side of the planet, 19,973 km from the centre. The current code has the same kind of point where p is parallel to n: on WGS84 a point 21,084 m due north of the centre (the fix gives 20,083.8 m there), and on the sphere the antipode, which stays +Infinity with the fix too. A final patch should handle this case explicitly. The edge point on the line from c toward the point's projection is the nearest one when the section is a circle (on the sphere); on an ellipse it is an approximation (under 0.005% off in the probe). For D == 0 it returns the old plane unchanged.

private Plane perpendicularPlane(
    final PlanetModel planetModel, final double x, final double y, final double z) {
  if (D == 0.0) {
    return new Plane(
        this.y * z - this.z * y, this.z * x - this.x * z, this.x * y - this.y * x, 0.0);
  }
  final double xyScalingSquared = planetModel.xyScaling * planetModel.xyScaling;
  final double zScalingSquared = planetModel.zScaling * planetModel.zScaling;
  final double mx = this.x * xyScalingSquared;
  final double my = this.y * xyScalingSquared;
  final double mz = this.z * zScalingSquared;
  final double t = -D / (this.x * mx + this.y * my + this.z * mz);
  final double cx = mx * t, cy = my * t, cz = mz * t; // centre of the section
  final double px = x - cx, py = y - cy, pz = z - cz;
  final double a = this.y * pz - this.z * py; // n x (p - c)
  final double b = this.z * px - this.x * pz;
  final double c = this.x * py - this.y * px;
  return new Plane(a, b, c, -(a * cx + b * cy + c * cz));
}
// arcDistance and linearDistance: final Plane perpPlane = perpendicularPlane(planetModel, x, y, z);

Trade-offs

• Plane is shared by every outside-distance shape. Planes through the origin (polygon edges, great circles, longitude planes) keep the old code path; latitude planes give the same plane mathematically (read). Only tilted planes with D != 0 change: circle planes, exact-circle slices and the offset planes of path segments. On 6,724 points around 1 km and 20 km wide WGS84 and SPHERE paths, path outside distances changed by at most 0.4 mm.
• A few extra multiplications per call; no performance measurement. Not run on x86 or another JDK. The 3 @Nightly tests were not run: two are query tests (TestGeo3DPoint.java:601, :606) and one a shape-relationship test (TestRandomGeoShapeRelationship.java:188); none calls these methods.
• Still open: handle the degenerate case; add targeted tests for the changed D != 0 users (path sides, exact-circle slices: only the module suite and the 6,724-point path probe cover them); say in the javadoc that the result is an approximation on an ellipsoid. Recommended because it is the only option tried that also fixes small GeoExactCircles and keeps the Plane.linearDistance contract for every tilted plane; the GeoStandardCircle-only option has the smaller blast radius.

What the checks validated

• Fresh scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the repro passes 3 of 3 with assertions on and off (stock: 2 of 3 fail; again in the fact check's fresh extraction); the whole :lucene:spatial3d:test task passes (20 classes, 184 tests, 3 @Nightly skipped, 0 failures; stock with the repro: only its 2 tests fail); checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass.
• Shadowed onto the stock jars: the isWithin-bisection probe gives 0 +Infinity and under 0.005% error for WGS84, CLARKE_1866 and a flattening-3e-4 planet at 1, 5, 20 and 100 km; the latitude sweeps give 0 +Infinity for the standard circle, and for GeoExactCircle only the 2 of 72 joint bearings of LFV-737 remain (57 of 72 before); the independent 7-test check passes 6, the one failure being the exact circle at bearings 0 and 180 (LFV-737).

For ARC, LINEAR and LINEAR_SQUARED, intersect circlePlane with the plane through the origin, the circle's centre and the point; its line crosses the inside of the circle. NORMAL styles unchanged; falls back to the old computation if no intersection is found (+28/-1 in GeoStandardCircle.java, the finder's sketch re-wrapped for google-java-format).

Trade-offs

Smallest change, but GeoExactCircle stays broken (the independent check still fails 11 of 12 bearings for it; the sweep still 57 of 72), and other tilted D != 0 planes keep the spherical assumption.

What the checks validated

Same worktree: the repro passes with assertions on and off; the whole spatial3d test task passes (184 tests); checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass; the bisection probe and sweeps give the same standard-circle results as the recommended fix.

Say in the javadoc that outside distances need PlanetModel.SPHERE.

Trade-offs

Contradicts the existing contracts and the correct results for large WGS84 circles, and leaves the sort's +Infinity in place.

Method: testing in the hunt's spatial3d geometry unit (no model checker, solver or prover was involved).

• While measuring the outside-distance sort for a related candidate (decoded doc values off the surface, LFV-247), the finder saw documents far outside a small WGS84 circle come back with +Infinity.
• It isolated the effect on exact surface points with the geometry API: a bearing sweep (36 bearings) and a latitude × radius sweep (72 bearings at 5 latitudes and 4 radii).
• It confirmed the mechanism by rebuilding the plane that Plane.arcDistance uses with the public API and calling findIntersections: 2 points due north, 0 at 30°, 60° and 90°, and F 21 km from the circle's centre (an independent reproducer's diagnostic later covered all 12 bearings).
• It then wrote a public-API JUnit test on the stock jars and a narrow fix sketch; independent verifiers, cross-model reviews and an adjudication followed (see Independent checks).