A geo3d path measures its end caps to their centre point, not their edge, so newOutsidePathSort misorders documents

A geo3d path with a width (GeoPathFactory.makeGeoPath with a cutoff angle, or the path of Geo3DDocValuesField.newOutsidePathSort) is a corridor along each segment plus a cap at each end and joint, bounded by a circle plane of the half-width (round on the sphere). Its outside distance is documented as "the distance from a point to the closest outside edge", and that is what you get beside a segment. The caps are never measured to their edge: each reports the distance to its centre point. So beyond either end and on the outer side of a turn the value comes from the end point, the segment's end cutoff plane or a corner, all inside the cap or on its rim, and around a one-point path it is the distance to the centre: on the unit sphere too large by up to the half-width, exactly the half-width on the path's axis. On the unit sphere a point 0.001 rad beyond the end of a path of half-width 0.01 gets 0.011, a one-point path reports 0.011 where a GeoCircle of the same radius reports 0.001, and the value jumps from 0.0 to 0.010001 across a 2e-6 rad step at the cap edge. On WGS84, newOutsidePathSort with a 5 km half-width gives a document 1 km past the end 5,992 m and sorts it after a document about 3 km beside the path (3,067 m). The cause is BaseSegmentEndpoint.outsideDistance (GeoStandardPath.java:945-949), which the three endpoint classes with a width inherit unchanged. It dates from the outside-distance API itself (LUCENE-6578, 5.3.0, 2015) and is in every release to 10.6.0 (10.6.0 by reading) and on main. Matching is unchanged; the distances are wrong, silently, and so is the order of hits, which in a top-N sort also decides which matching documents make the top N. The recommended fix measures each cap to its circle's edge: exact on the unit sphere, close but not exact on WGS84; it passes the whole lucene/spatial3d module, and unlike the hunt's first sketch it stays finite on WGS84.

• Path with a width (GeoStandardPath, from GeoPathFactory.makeGeoPath(planet, cutoffAngle, points) or Geo3DUtil.fromPath behind newOutsidePathSort): "a path across the surface of the globe, with a specified half-width". It is built from components: a segment (a corridor bounded by two side planes and two cutoff planes) for each pair of points, and an endpoint for each point: an end cap at the first and last point, a joint where two segments meet, or a single circle for a one-point path. Each endpoint is bounded by a circle plane of the half-width (a joint by two): a round cap on the sphere, "an ellipse otherwise" (GeoStandardCircle.java:23-25).
• Outside distance (computeOutsideDistance): 0.0 for points inside the shape, else the distance "to the closest outside edge". The path takes the minimum over all its components.
• Half-width: the cutoff angle (radians); newOutsidePathSort converts its pathWidthMeters into it, so that argument also acts as the half-width.

1. Each segment measures a point to its own edge: "the minimum distance from the point to any of the segment's bounding planes", plus its four corners (GeoStandardPath.java:1783-1839).
2. Each endpoint inherits BaseSegmentEndpoint.outsideDistance (:945-949), which returns the distance to this.point, the end point at the centre of the cap. CircleSegmentEndpoint, CutoffSingleCircleSegmentEndpoint and CutoffDualCircleSegmentEndpoint do not override it, although their isWithin uses a circle of the half-width, so the caps are part of the shape.
3. Beyond an end, the segment measures to its end cutoff plane (through the end point, bounded by the side planes) and its corners, which lie inside the cap or on its rim; nothing measures the cap's circle. So the minimum is the end-point distance on the path's axis (gap + half-width), the end cutoff plane's distance a little off the axis and a corner's distance farther off (the fact check's probe: end point at 0 degrees, cutoff plane at 10 to 40, a corner from 50). On the outer side of a turn the first segment's end cutoff plane gives 0.00778 instead of 0.001. On the unit sphere the excess is at most the half-width. A one-point path has no segment, so on the unit sphere it is gap + half-width at every bearing (12 of 12 checked).
4. The formula is right only for a zero-width endpoint, where the end point is the edge: the zero-width GeoDegeneratePath gives the right ARC value beyond its end (its squared styles have a separate defect, registry LFV-741). Zero-width paths came in 2017 as that separate class (LUCENE-7955); the 2015 GeoPath rejected a cutoff angle of 0.

All five distance styles (ARC, LINEAR, LINEAR_SQUARED, NORMAL, NORMAL_SQUARED) measure the cap and the one-point path to the end point.

• Who. Users of Geo3DDocValuesField.newOutsidePathSort (public, not tagged experimental) or of GeoPath.computeOutsideDistance (@lucene.experimental) on a path made with a width. For example, ranking places by their distance from a route: places past the start or end of the route, or around a route of one point, are normal input.
• Effect. Silent wrong values and order. In the repro a document 1 km past the end gets 5,992 m (997 m with the fix) and sorts after one about 3 km beside the path; with a 20 km half-width a document 5 km past the end gets 24,972 m and sorts after one 10 km beside the path (10,055 m). The order breaks between documents whose nearest path parts differ (cap or joint against a segment side). In a top-N sort the comparator's compareBottom uses the value to decide whether a document enters the top N (Geo3DPointOutsideDistanceComparator.java:75-101), so a document past an end can drop out of the top N. Around a one-point path on the unit sphere the order is kept and every value is too large by the half-width; on WGS84 the excess varies a little with the bearing. The sort value returned in FieldDoc is that wrong number in meters.
• Not affected. Which documents match (no query uses outside distance to match; the only callers outside the geometry package are the sort comparator's competitive check and its sort value); newPathSort, which uses the inside path distance; the zero-width GeoDegeneratePath (not by this bug); the index.
• Default vs opt-in. Opt-in: the lucene/spatial3d module and a path outside-distance sort or call. Within that, default settings and any planet model; SPHERE and WGS84 were run.
• Scale. None: one shape and one point, deterministic, with assertions on and off.
• How likely. Plausible: deterministic once the feature is used, but the feature is little used (no upstream test calls a path outside distance or a path outside sort), and no report was found in the searches below.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<0.001> but was:<0.010999999999988425>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:555)
       org.junit.Assert.assertEquals(Assert.java:685)
       org.apache.lucene.spatial3d.geom.TestGeoPathOutsideDistanceEndCap.testOutsideDistanceBeyondEndCap(TestGeoPathOutsideDistanceEndCap.java:29)

• The contract is the edge. GeoOutsideDistance: "Implemented by Geo3D shapes that can compute the distance from a point to the closest outside edge" (GeoOutsideDistance.java:20-21). CHANGES for LUCENE-6578: "Geo3D can now compute the distance from a point to a shape, both inner distance and to an outside edge" (lucene/CHANGES.txt:10299-10301). The issue (#7636) asked for the distance "from a GeoPoint to the nearest edge/point of a GeoShape". newOutsidePathSort: "orders documents by ascending outside distance from the described path. Points within the path are given the distance of 0.0" (Geo3DDocValuesField.java:346-347).
• The same class and its sibling measure to the edge. PathSegment.outsideDistance measures to its bounding planes (GeoStandardPath.java:1787); GeoStandardCircle.outsideDistance measures to its circle plane (GeoStandardCircle.java:168-172), as its predecessor GeoCircle did in the same 2015 commit. A one-point path is built as a "Simple circle" (GeoStandardPath.java:152-153) yet on the unit sphere differs from the same-size GeoCircle by the half-width at all 12 bearings checked.
• Not an estimate. The value jumps from 0.0 to about the half-width at the cap edge and inverts the order within one shape (0.001 beyond the end gets 0.011, 0.005 beside gets 0.005). It shows in ARC, the exact style.
• Nothing pins it. No comment, javadoc, CHANGES or MIGRATE entry, test or issue calls the end-point distance intended, and no upstream test computes a path outside distance. The fix passes the whole spatial3d module.
• Wording nit, separate from the bug. pathWidthMeters is documented as the "width of the path in meters" (Geo3DDocValuesField.java:361) and the cutoff angle as "the width of the path" (GeoPathFactory.java:34), but both act as the half-width.

The strongest case for "works as designed" (rated moderate by the cross-model reviews, weak to moderate by the other checkers): the geometry API is @lucene.experimental and GeoStandardPath is @lucene.internal; the per-method javadoc only says "this shape's distance"; GeoDistance speaks of "distance estimates that can be computed more cheaply" (GeoDistance.java:21-22); the behaviour is 11 years old, no report of it was found, and on the sphere it is bounded by the half-width; and the endpoints were reworked twice (LUCENE-8696 in 2019, the #11965 refactor in 2022), the 2022 work editing this very line.

Why it fails: "experimental" is about API stability, not about the documented meaning; the "estimates" sentence is the javadoc of GeoDistance, the inside-distance interface ("distances from the shape to a point expected to be INSIDE the shape", GeoDistance.java:28-30), while outside distance is declared in GeoOutsideDistance, which promises the closest outside edge; the defect shows in ARC, the exact style; measuring to the circle costs about what GeoCircle already does; neither rework was about the outside distance (LUCENE-8696 reworked how endpoint circles are represented, and the 2022 commit's only change to this line was a toAggregationForm wrapper that leaves the ARC value unchanged); and a value that jumps from 0 to the half-width at the edge and disagrees with a circle of the same shape is not an estimate of the edge distance. Verdict: not intended, a defect.

Option A was applied in a scratch copy of a9779ef3c193 and validated with gradle; it was revised on 2026-10-28 after a review found that its first version returned the centre distance for a one-point path near the antipode of its centre, and every validation was re-run with the revised version. Option B is the hunt's first sketch, re-checked here and found to fail on WGS84; C was not run; D is not recommended.

+99/-0 in GeoStandardPath.java:

• a helper BaseSegmentEndpoint.circleEdgeDistance(style, circlePlane, x, y, z, bounds...): for ARC and the LINEAR styles it intersects the circle plane with the plane through the origin, the endpoint's centre and the point (normalized), keeps the intersection points that meet the bounds and returns the nearest; when the point is the centre's antipode (to within 1e-12), every plane through the centre contains it, and the helper takes the one through the coordinate axis least aligned with the centre; for the NORMAL styles it returns the plane distance, as GeoStandardCircle does;
• CircleSegmentEndpoint.outsideDistance (one-point path): the circle's edge, with the centre distance kept as an upper bound;
• CutoffSingleCircleSegmentEndpoint.outsideDistance (end cap): the part of the circle beyond the cutoff plane, and the two corners;
• CutoffDualCircleSegmentEndpoint.outsideDistance (joint): both circles between both cutoff planes, and the corners.

protected double circleEdgeDistance(DistanceStyle distanceStyle, SidedPlane circlePlane,
    double x, double y, double z, Membership... bounds) {
  if (!(distanceStyle instanceof ArcDistance || distanceStyle instanceof LinearDistance
      || distanceStyle instanceof LinearSquaredDistance)) {
    return distanceStyle.computeDistance(planetModel, circlePlane, x, y, z, bounds);
  }
  // the nearest edge point lies in the plane through the origin, the center and the point
  Vector perpNormal = new Vector(point.y * z - point.z * y, point.z * x - point.x * z,
      point.x * y - point.y * x).normalize();
  if (perpNormal == null) {
    // the point is the center's antipode: any plane through the center contains it;
    // take the one through the coordinate axis least aligned with the center
    final double ax = Math.abs(point.x), ay = Math.abs(point.y), az = Math.abs(point.z);
    if (ax <= ay && ax <= az) perpNormal = new Vector(0.0, point.z, -point.y).normalize();
    else if (ay <= az) perpNormal = new Vector(-point.z, 0.0, point.x).normalize();
    else perpNormal = new Vector(point.y, -point.x, 0.0).normalize();
  }
  final Plane perpPlane = new Plane(perpNormal, 0.0);
  final GeoPoint[] edgePoints = circlePlane.findIntersections(planetModel, perpPlane, bounds);
  if (edgePoints == null) {
    return distanceStyle.computeDistance(planetModel, circlePlane, x, y, z, bounds);
  }
  double minDistance = Double.POSITIVE_INFINITY;
  for (final GeoPoint edgePoint : edgePoints) {
    minDistance = Math.min(minDistance, distanceStyle.computeDistance(edgePoint, x, y, z));
  }
  return minDistance;
}

// end cap (CutoffSingleCircleSegmentEndpoint); the joint does the same with both circles
@Override
public double outsideDistance(DistanceStyle distanceStyle, double x, double y, double z) {
  double minDistance = circleEdgeDistance(distanceStyle, circlePlane, x, y, z, cutoffPlane);
  for (final GeoPoint notablePoint : notablePoints) {
    minDistance = Math.min(minDistance, distanceStyle.computeDistance(notablePoint, x, y, z));
  }
  return distanceStyle.toAggregationForm(minDistance);
}

Trade-offs

• Cost: for ARC and the LINEAR styles one plane intersection per circle (two per joint) per outside-distance call, with a Vector, a Plane and the intersection array allocated each time, instead of one point distance; for the NORMAL styles one plane distance per circle. The sort comparator calls it per document value; not benchmarked. GeoStandardCircle pays a similar cost for its plane distance.
• Observable change: sort values of documents near path ends change; clients that keep FieldDoc sort values, for example to page with searchAfter across an upgrade, see different values. The index is not affected.
• NORMAL styles: the fix gives what GeoCircle gives, the perpendicular distance to the circle's plane (1.05e-5 for a point 0.001 outside, against 0.001 beside a segment; stock gives 0.011). That keeps paths consistent with circles rather than with the path's own sides.
• WGS84 caps are measured as built. A separate defect from the same hunt, not reported upstream yet (internal id LFV-736), shifts WGS84 end caps toward the equator; once that is fixed, this fix follows the corrected cap.
• Shared helper possible: the same "plane through the centre" idea fixes the +Infinity that small GeoCircles return on WGS84 (another unreported defect from the same hunt, internal id LFV-274); a helper in Plane could serve both.

What the checks validated

• Revised 2026-10-28; every run below was repeated with the revised fix. The first version built the plane through the centre and the point unnormalized, which is degenerate at the centre's antipode (Plane.findIntersections then returns an empty array, not null): a one-point path got the centre distance, pi instead of pi - W, within about 1e-11 rad of the antipode on the unit sphere and 1e-9 rad on WGS84 (1 km). The antipode probe: on the unit sphere the revised fix is within 4e-14 of pi - delta - W at every offset delta from 0 to 0.01 (stock and the first version near the antipode: the full half-width too much); on WGS84 it is within about 0.1% of the half-width of a traced edge distance; multi-point paths are right with both versions.
• The new test: 5 of 5 pass in gradle and with the standalone runner, assertions on and off. The finder's, the independent reproducer's and the devil's advocate's JUnit classes: 17 run, 0 failed (13 fail on stock); the adjudication's probe: 4 of 4.
• The whole lucene/spatial3d module, its 19 suites (181 tests) plus the new test: 20 suites, 186 tests, 3 skipped, 0 failed, with assertions on (seed DEADBEEF) and off (seed 5EED0735); TestGeoPath, TestRandomGeoShapeRelationship and TestGeo3DDocValues with 20 iterations: 360 tests, 20 skipped, 0 failed; :lucene:spatial3d:check -x test passes.
• The hunt's reference harness on the unit sphere (1,231,737 checks): outside-distance mismatches 7,476 -> 0, order inversions 5,869 -> 0. Its floating-point category "outside == 0 just outside" (outside distance exactly 0.0 for points isWithin puts outside, all within 3e-8 rad, about 19 cm, of the edge: the floor of the arc distance's acos) grows from 814 to 1,668, because points just outside a cap now join those just outside a corridor side; that band predates the fix.
• A random sweep of 20,000 paths (1 to 4 points, half-widths 1e-5 to 0.05 rad, 46,140 points just outside an end or joint), against the oracle "no larger than the distance to a point that isWithin puts inside": stock breaks it in 21,390 of 25,291 sphere cases and 13,503 of 20,849 WGS84 cases, by up to 0.999 half-widths; the fix in 0 sphere cases and 95 WGS84 multi-point cases, by at most 0.30 m (half-widths of 67 to 411 m). Never +Infinity, never an exception, never larger than stock. The same counts with both versions.
• Status on ellipsoids: exact in every tested case on the unit sphere; on WGS84 a close prototype, not a proven edge distance (that the nearest edge point lies in the plane through the centre is not proved for an ellipsoid; the residuals above show the gap).

The same three overrides, each with distanceStyle.computeDistance(planetModel, circlePlane, x, y, z, bounds...), as GeoStandardCircle does. On the unit sphere it is equivalent: the hunt's tests, 32 upstream tests and the harness (7,476 -> 0) pass.

Trade-offs

Worse on WGS84. Plane.arcDistance looks for the edge along the plane through the circle plane's normal, which misses small tilted circles (the mechanism of the small-circle +Infinity above). A one-point path with a 1 km half-width at latitude 40 then returns +Infinity at 10 of 12 bearings (stock: finite, too large), so the sort would treat such documents like documents without a value, and a 1 km end cap stays at the end-point distance. In the sweep: 1,261 +Infinity results among 9,382 WGS84 one-point cases and 4,525 WGS84 cases above the bound, by up to 13 km. The new test's testOnePointPathOnWgs84 fails with it. Use it only with a general Plane.arcDistance fix.

What the checks validated

Re-run here: the new test 4 of 5 pass (testOnePointPathOnWgs84 gets Infinity); the WGS84 probe and the sweep above.

The general fix; it would also fix the small-circle +Infinity of GeoCircle.

Trade-offs

Changes a primitive that every geo3d shape uses, so it needs its own review and tests. Not run.

Not recommended. It keeps today's sort values (no change for stored FieldDoc values) and costs nothing, and a maintainer who weighs compatibility of the long-standing values highly could choose it with a javadoc note.

Trade-offs

Contradicts GeoOutsideDistance, the LUCENE-6578 CHANGES entry and GeoCircle, keeps the jump at the cap edge, and leaves the sort misordering.

Method: bounded-exhaustive translation validation on the real jar (semi-formal tier).

• A hunt unit on geo3d distance functions (2026-10-27, main a9779ef3c193) recomputed every documented distance of circles and paths with an independent BigDecimal reference (50 digits). On the unit sphere a path is exactly the half-width buffer of its polyline, so the reference outside distance is the distance to the polyline minus the half-width.
• In 1,231,737 checks over 72 paths (60 of them with a width) it flagged 7,476 outside-distance values and 5,869 order inversions, all on paths with a width; the root cause was then found by reading the endpoint classes, and with a fix shadowed in the same harness reports 0 of either.
• Then a public-API JUnit on the stock jar, an independent reproducer, an intended-behaviour devil's advocate, two executed reviews by a model from another family (OpenAI GPT-5.6), an adjudication and a credibility audit.