On WGS84, geo3d path end caps are shifted up to about 21 km toward the equator

Geo3DPoint.newPathQuery and Geo3DDocValuesField.newPathSort build a GeoStandardPath: a corridor of the given width along the path, closed at each end by a cap that is round on a sphere. Each end cap is the part of the planet cut off by one plane through the two corners of the end segment, whose normal is constrained to the plane through the corners and the earth's centre. For an end segment that runs north-south that normal is the end point's direction from the centre. On a sphere that is the surface normal, so the cap is centred on the end point; on the WGS84 ellipsoid it is the surface normal of a point about 0.19 degrees (21.4 km at 44-45N) closer to the equator, so the cap circle is centred there while still passing through the corners. With pathWidthMeters 20000 on the path (44N,20E) to (45N,20E), documents 12 and 16 km north of the last point are missed (newPathSort +Infinity) and documents 30 and 40 km south of the first point are matched (30023 and 40034 m). The caps reach 7.8 km past the poleward end and 50.5 km past the equatorward end; a 1 km path reaches 42.8 km past its start. Nothing throws. All 16 releases run, from 5.3.0 (the first with WGS84) to 10.5.2, and main are affected; the releases in between and 10.6.0 have the same construction (read).

• geo3d path (lucene-spatial3d): a GeoStandardPath, built by Geo3DPoint.newPathQuery, Geo3DDocValuesField.newPathSort / newOutsidePathSort and GeoPathFactory.makeGeoPath. It is meant to be a corridor of half-width w along the segments between the points, closed past the first and last points by a cap (on a sphere, a round cap of radius w), with joints at turns. pathWidthMeters is used as w, the distance from the path (Geo3DUtil.java:139).
• End cap: geo3d represents it as the part of the planet on one side of one plane, a plane slice, kept only beyond the end segment's cutoff plane. The plane goes through the segment's two corners, so corridor and cap meet.
• WGS84: the ellipsoid model (a = 6378137 m, b = 6356752.314245 m). On it, 'straight up' (the surface normal) and the direction from the earth's centre differ by up to 0.19 degrees, at 45 degrees latitude.

1. GeoStandardPath.done() makes the first and last points CutoffSingleCircleSegmentEndpoints, with the end segment's cutoff plane and two corners (GeoStandardPath.java:192-241).
2. The cap plane is SidedPlane.constructSidedPlaneFromTwoPoints(point, upper, lower) (GeoStandardPath.java:997-1007), that is Plane.constructPerpendicularCenterPlaneTwoPoints (Plane.java:250-260): a plane through the two corners, perpendicular to the plane through the corners and the origin.
3. For a north-south end segment the corners are mirror images east and west of the end point, so the cap plane's normal is the end point's direction from the centre.
4. A plane cuts an ellipsoid in an ellipse whose centre lies inside the earth, on the line from the earth's centre to the surface point whose surface normal is parallel to the plane's normal; that point is the cap's apex, and the cap is centred on it. For (45N, 20E) it is at 44.81N: 0.19 degrees, 21.4 km toward the equator (18.6 km at 60N, zero at the equator and the poles). That holds for oblate models such as WGS84 and Clarke 1866; on a prolate model (b > a, which PlanetModel also accepts) the same formula moves the apex poleward (not run).
5. The circle still passes through the corners, w east and west of the end point, so its radius is about sqrt(21.4^2 + w^2) km. Past a poleward end it reaches sqrt(21.4^2 + w^2) - 21.4 km; past an equatorward end 21.4 + sqrt(21.4^2 + w^2) km.

• Path (44N,20E) to (45N,20E), w = 20 km: the north cap reaches 7.8 km, the south cap 50.5 km (the flat sketch above gives 7.9 and 50.7).
• w = 5 km: 0.55 km and 43.3 km; w = 1 km: 0.02 km and 42.75 km; w = 200 km: 0.89 w and 1.11 w.
• The reversed path and the southern mirror (44S to 45S) behave the same at their poleward and equatorward ends; a north-east end segment gives 0.50 w and 2.02 w; at 60N 0.44 w and 2.29 w.
• Controls within about 1% of w, in the tested cases: a sphere (1.00 w), east-west end segments (1.01 w), the corridor sides (0.995 w), one-point paths and interior joints at 90-degree turns (0.99 w). A path from the equator to 1N gives 0.99 w at its equator end and 0.95 w at its 1N end, where the shift is already about 0.7 km.

• Who is hit. Users of the lucene-spatial3d path query, path sort and outside-path sort, or of GeoPathFactory.makeGeoPath, with PlanetModel.WGS84, CLARKE_1866 or another oblate model (a prolate model is mis-centred the other way). Also, by reading, spatial-extras Geo3dShapeFactory buffered line strings when the context factory is configured with planetModel wgs84 or clarke1866 (other spellings, clarke_1866 included, throw "Unknown planet model"), or with its public planetModel field set to PlanetModel.WGS84 or CLARKE_1866; its default is the sphere, which is not affected.
• Default vs opt-in. Opt-in by module and API (Geo3DPoint and GeoPath are @lucene.experimental). Within it the trigger is ordinary: WGS84 is what Geo3DPoint's 3-argument constructor uses, and newPathQuery / newPathSort take the PlanetModel explicitly, so a caller passes WGS84 to match points indexed that way; any route with a north-south end at mid latitudes then hits it. Index size does not matter.
• Effect. Wrong matches and wrong sort values near the path ends: documents within the width past a poleward end are dropped (sort value +Infinity, which marks "points outside of the shape"), documents far past an equatorward end match with finite sort values. For a 20 km path the poleward cap keeps 39% of its straight-out reach and the equatorward cap reaches 2.5 times the width; for narrow paths the equatorward cap reaches about 43 km whatever the width. Nothing throws, nothing is written.
• Where it is small. Near the equator and the poles (on a path from the equator to 1N the reach is 0.99 w at the equator end and already 0.95 w at the 1N end), and, in the tested cases, for east-west end segments, along the corridor sides, at 90-degree interior joints and for one-point paths. The shift is a fixed distance (about 21 km at mid latitudes), so wide paths suffer less: 0.89 w and 1.11 w at 200 km.
• How likely. The module is opt-in and only the path ends are affected; the recorded search found no report since 5.3.0 (2015), and no upstream test checks points past a path end on WGS84.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: (43.82N,20E), 20 km south of the start of the 1 km path (44N,20E)-(45N,20E): expected outside, got isWithin=true
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertFalse(Assert.java:65)
       org.apache.lucene.spatial3d.geom.TestGeoPathEndCapWGS84.testOneKmPathDoesNotContainPoint20KmBeyondItsStart(TestGeoPathEndCapWGS84.java:30)

• Contract. GeoStandardPath: "GeoShape representing a path across the surface of the globe, with a specified half-width. Path is described by a series of points. Distances are measured from the starting point along the path, and then at right angles to the path." (GeoStandardPath.java:29-32). The end components are "Endpoint that's a simple circle" (:990) and "Endpoint that's a single circle with cutoff(s)" (:1084). newPathQuery: "@param pathWidthMeters width of the path in meters." with no error statement (Geo3DPoint.java:192-213); newDistanceQuery: "if PlanetModel#WGS84 is used, the query is approximate and may have up to 0.5% error" (:112-113).
• Size. For a 20 km path 61% of the poleward cap's straight-out reach is lost and the equatorward cap reaches 2.5 times the width; a 1 km path matches documents 42.8 km away.
• History. Before 5.3.0 the cap was new SidedPlane(point, point, -originDistance), exact on the unit sphere. LUCENE-6487 replaced it with the two-corner plane so the corridor would "line up with the circles at the waypoints"; the cap's position was not discussed. LUCENE-6544 (#7602): "Circles are now always centered on their center point, no matter how significantly ellipsoid the planet is." That re-anchored circles and the one-point path endpoint, not multi-point path ends, and left this end cap as it was. LUCENE-8696's commit: "Rework how endpoint circles are represented to allow for consistency on WGS84".
• Tests. No upstream test asserts end-cap extent on WGS84 (TestGeoPath's WGS84 tests cover bounds, a degenerate-path failure, LUCENE-8696 and identical points); the random query tests compare the query with the shape's own isWithin, so a wrong shape agrees with itself.

Strongest case for "intended" (rated strong by the reviews from another model family, OpenAI GPT-5.6, and argued in full by a devil's advocate): geo3d builds shapes from plane slices and is approximate on ellipsoids; GeoStandardCircle says a one-plane circle "is a circle for spherical planets and an ellipse otherwise"; the private SegmentEndpoint javadoc says an endpoint "is not necessarily a circle" and case (2) "must be constructed to go through two supplied points and be perpendicular to a connecting plane"; constructPerpendicularCenterPlaneTwoPoints is "useful for building path endpoints on worlds that can be ellipsoids"; GeoStandardCircle notes "In an ellipsoidal world, cutoff distances make no sense, unfortunately. Only membership can be used to make in/out determination."; the classes are experimental or internal; and the geo3d author wrote in LUCENE-8696 that retrofitting paths to WGS84 "involved compromises".

Why it still fails: the private javadoc does pin this plane (a plane through both corners and perpendicular to the plane through them and the origin is unique), so the code does what it says, and fix A would change that javadoc line. But the javadoc fixes a construction, chosen in LUCENE-6487 so the corridor lines up with the end circles; where the cap then sits on an ellipsoid was not discussed. The class promises a half-width, and the corridor sides and one-point paths keep it. An ellipse is fine; GeoStandardCircle is one on WGS84, but anchored on its centre point. Its "only membership can be used" comment is about distance values; here membership itself is wrong. The "compromises" remark was about a random-test path with a 1.34 rad (77-degree) half-width and about joints. A 1 km path matching documents 42.8 km away is hard to read as an accepted approximation. Verdict: not intended, a defect, with a strong case the other way.

Two options were built and validated in a scratch worktree at a9779ef3c193 with gradle (JDK 25) and with the standalone runner against the finder's and the verifiers' repros. A passes everything; B is smaller but breaks one upstream bounds assertion on an extreme test planet. Either needs a regression test; this page's test is one.

In CutoffSingleCircleSegmentEndpoint, compute the tip (the surface point at the mean corner angle past the end point, along the outward normal of the cutoff plane) and build the cap with SidedPlane.constructNormalizedThreePointSidedPlane, the helper interior joints already use at turns; fall back to today's plane if that helper returns null (nearly collinear points, or the inside point on the plane). The finder's sketch, reworded and tidied.

super(planetModel, previous, point,
    endCapPlane(planetModel, point, cutoffPlane, topEdgePoint, bottomEdgePoint));
...
// t = unit normal of the cutoff plane, away from the segment; u = point / |point|
final double alpha =
    0.5 * (point.arcDistance(topEdgePoint) + point.arcDistance(bottomEdgePoint));
final GeoPoint tip = planetModel.createSurfacePoint(
    ux * Math.cos(alpha) + tx * Math.sin(alpha), /* y, z likewise */ ...);
final SidedPlane plane = SidedPlane.constructNormalizedThreePointSidedPlane(
    point, topEdgePoint, bottomEdgePoint, tip);
return plane != null
    ? plane
    : SidedPlane.constructSidedPlaneFromTwoPoints(point, topEdgePoint, bottomEdgePoint);

Trade-offs

• A few trig calls per path end at construction; not benchmarked.
• On a sphere the tip lies on the same circle, so the plane is today's up to rounding (the sphere control and probe give 1.00 w).
• TestGeoPath.testPathBounds passes; on its extreme test planet the bounds still contain every sampled path point (0 of 120,359 grid points outside).
• The tip is the end point's radial direction rotated by the mean corner angle and projected to the surface, not a point at exactly the half-width along a geodesic: 1.00 w in every sampled case, but no proof for all geometries. The fallback is today's plane. The private SegmentEndpoint javadoc's case (2) would need rewording.

What the checks validated

• Gradle, scratch worktree at a9779ef3c193, seed DEADBEEF: TestGeo3DPathEndCaps 2/2 with assertions on and off; the whole spatial3d module, 20 classes, 183 tests, 3 skipped, 0 failures (TestGeoPath 11/11, TestGeo3DPoint 17, TestRandomGeoShapeRelationship 6, TestGeo3DDocValues); the spatial-extras geo3d tests, 21 tests, 4 skipped, 0 failures; :lucene:spatial3d:check -x test (google-java-format, ecjLint, forbiddenApis) passes.
• Standalone runner, patched class shadowed: the finder's test 3/3, the independent reproducer's 8/8, the devil's advocate's 1/1, this page's 2/2 (assertions on and off); reach 1.00 w at every sampled end and bearing (0.99 at the equator).

Keep the plane through the two corners, but use the surface normal at the end point, (x / a^2, y / a^2, z / b^2), minus its component along the corner chord. +31/-1, no trigonometry; on a sphere it is exactly today's plane.

final double nx = point.x * planetModel.inverseXYScalingSquared; // ny likewise
final double nz = point.z * planetModel.inverseZScalingSquared;
final double dx = topEdgePoint.x - bottomEdgePoint.x;            // dy, dz likewise
final double t = (nx * dx + ny * dy + nz * dz) / (dx * dx + dy * dy + dz * dz);
final Vector normal = new Vector(nx - t * dx, ny - t * dy, nz - t * dz).normalize();
if (normal == null) {
  return SidedPlane.constructSidedPlaneFromTwoPoints(point, topEdgePoint, bottomEdgePoint);
}
return new SidedPlane(point, normal, -normal.dotProduct(topEdgePoint));

Trade-offs

• Breaks one upstream assertion: TestGeoPath.testPathBounds (TestGeoPath.java:256, assertTrue(area.isWithin(point))). On its extreme planet (a = 0.75, b = 1.25, width 0.75 rad) the test point lies outside the path on stock and with both fixes; with B the bounds no longer include it, but they still contain every sampled path point (0 of 129,751). So the assertion looks like it pins old bounds rather than correctness, but a finite grid does not prove the new bounds complete; adopting B means changing it after a bounds review.
• Reach 0.99-1.00 w, the reach of the corridor sides.

What the checks validated

• Standalone, shadowed: this page's test 2/2 (assertions on and off), the finder's 3/3, the independent reproducer's 8/8, the devil's advocate's 1/1, the cross-model reviewer's over-inclusion probe passes.
• Gradle: TestGeo3DPathEndCaps 2/2; spatial3d module 183 tests, 1 failure (testPathBounds); spatial-extras and check not run for B.

State that path end caps are approximate on ellipsoids.

Trade-offs

In our view a bound that lets a 1 km path match documents 42.8 km away is not useful.

Method: bounded-exhaustive translation validation (the semi-formal tier), in a hunt unit on geo3d distance functions against a reference.

• A harness recomputed the documented path decomposition (corridor, end caps, joints) with an independent 50-digit BigDecimal reference and compared GeoStandardPath.computeDistance with it. Check "inside membership" flagged 114 points per distance style that the reference puts robustly inside an end cap of a WGS84 path but for which the path returned +Infinity, all at the poleward end vertex.
• A targeted sweep over latitude, heading and width showed the pattern; reading the cap construction gave the cause, and a fix sketch removed it.
• Verifiers then corrected the first write-up: the equatorward cap over-includes, so it is a shift, not a truncation.