On a zero-width geo3d path the squared distance styles are not squared: path-center distance comes back as d, outside distance near the path as d^4

GeoPathFactory.makeGeoPath turns a path whose width is below Vector.MINIMUM_ANGULAR_RESOLUTION (about 3e-12 rad, for example 0) into a GeoDegeneratePath. For the squared distance styles LINEAR_SQUARED and NORMAL_SQUARED, which combine partial results in an "aggregation form" (the square root) and must convert back once (square) for the final result, this class gets two of its distance methods wrong: computePathCenterDistance returns the plain distance d instead of d^2 at every point off the path, because it never converts back, and computeOutsideDistance returns d^4 at every point off the path closer than a chord of 1 (the planet radius), because one of its pieces returns the plain d^2 into a minimum of aggregation-form values that is then squared. For a point 0.01 rad beside a sphere path that is 0.0099999583 and 9.9998e-9 where 9.9999e-5 is expected; the same path with width 1e-6 returns 9.9999e-5.

The cause is an incomplete edit in commit 839dfb5a2dc (2022-11-24, part of the #11965 spatial3d refactor; branch_9x cherry-pick f69fb6ed99b), first released in 9.5.0: it converted the pieces, not the two places that combine them. 9.4.2 returns d^2. Affected: 9.5.0 to 10.6.0 and main; not fixed upstream. ARC, LINEAR and NORMAL are unaffected, and no Lucene query, sort or comparator uses the squared styles, so the impact is limited to code that calls these GeoPath methods itself. Fix: two lines (validated with the whole lucene/spatial3d test suite).

1. The factory picks the class from the width alone. makeGeoPath returns new GeoDegeneratePath(...) when maxCutoffAngle < Vector.MINIMUM_ANGULAR_RESOLUTION (GeoPathFactory.java:40-41), else a GeoStandardPath (:43). GeoDegeneratePath is package-private and @lucene.internal (GeoDegeneratePath.java:33-35); callers only see a GeoPath.
2. The squared styles' aggregation form is the square root. toAggregationForm returns Math.sqrt(distance) and fromAggregationForm squares (LinearSquaredDistance.java:63-71, NormalSquaredDistance.java:63-71). The defaults are the identity (DistanceStyle.java:116-118, :146-148), so for ARC, LINEAR and NORMAL a missing conversion changes nothing.
3. Path-center distance: converted in, never converted out. computePathCenterDistance (GeoDegeneratePath.java:160-180) takes the minimum of PathSegment.pathCenterDistance and SegmentEndpoint.pathCenterDistance, which both return toAggregationForm(...) (:739, :522), and ends with return closestDistance; (:179). For a squared style that is sqrt(d^2) = d.
4. Outside distance: one piece left plain, then converted out. outsideDistance (:270-287) takes the minimum of SegmentEndpoint.outsideDistance (toAggregationForm(...), :534-537) and PathSegment.outsideDistance, which returns the plain value Math.min(Math.min(startDistance, endDistance), distance) (:866-874, :873), i.e. d^2, and then applies fromAggregationForm (:286). For LINEAR_SQUARED the result is min(e, d^2)^2, with d the true LINEAR distance and e the chord to the nearest path vertex. Below d = 1 (geo3d units: the sphere's radius is 1) that is always (d^2)^2 = d^4, beyond the path end too. Above 1 it is d^2 when the nearest path point is a vertex, d^4 while d^2 < e, and e^2, too large, otherwise (executed on the sphere: 1.6084 for d^2 = 1.2753 beside a 2-rad segment). NORMAL_SQUARED never exceeds 1 on the sphere, so it gets NORMAL^4 at every point off the path. computeOutsideDistance reaches it through GeoBaseMembershipShape.java:46-53 (0 for points on the path).
5. The width > 0 sibling does these two methods right. GeoStandardPath returns fromAggregationForm(...) from both (GeoStandardPath.java:278-285, :326-333). Its computeDistance has a separate defect (LFV-271).
6. A third plain return, masked. The pole branch of PathSegment.pathCenterDistance returns computeDistance(start, ...) without toAggregationForm (GeoDegeneratePath.java:712-714; same at GeoStandardPath.java:1592-1594). In the one pole probe run, the end points win the minimum, so the result is right once the two lines are fixed.
7. Scope: only these two methods. On the same zero-width path, computeDistance (a point on the path) and computeNearestDistance (a point on and a point off the path) return the square with LINEAR_SQUARED and NORMAL_SQUARED (executed), and points on the path get 0 from the two affected methods, which is right. At the pole of the path's great circle (the north pole, an equatorial path), computeOutsideDistance throws a NullPointerException from Plane.linearDistance or Plane.arcDistance with ARC, LINEAR and LINEAR_SQUARED alike, on 9.4.2 too: a separate behaviour, not part of this bug.

Lucene 9.4.2 returns the squares for the zero-width path; every release from 9.5.0 returns the values above.

• Silent, wide. With a squared style on a zero-width path, computePathCenterDistance is wrong at every point off the path, and computeOutsideDistance at every point off the path closer than a chord of 1 (with NORMAL_SQUARED on the sphere, at every point off the path) and at many farther ones. No exception, no special input; points on the path get 0, which is right.
• Who is exposed. Code that calls computePathCenterDistance or computeOutsideDistance with LINEAR_SQUARED or NORMAL_SQUARED on a path whose width is below Vector.MINIMUM_ANGULAR_RESOLUTION (for example 0). Both choices are opt-in, on the @lucene.experimental geom API (GeoPath, GeoPathFactory, DistanceStyle, GeoOutsideDistance).
• Lucene's own results are not affected. LINEAR_SQUARED and NORMAL_SQUARED appear nowhere in lucene/ outside DistanceStyle.java; nothing in Lucene calls computePathCenterDistance; the in-tree distance sorts, including the outside-path sort of Geo3DDocValuesField, pass DistanceStyle.ARC.
• Mixed widths can rank wrongly. The GeoPath javadoc says the path-center distance "is meant to allow comparisons between different paths to find the one that goes closest to a point". The class follows from the width, so a zero-width path compared with a wider path under a squared style can get the order wrong: it flips when the zero-width path's unsquared d is larger than the other path's squared value (executed; with the same two paths at q = (0.001, 0.1) the order is still right). Outside distances of a zero-width path cannot be compared with those of a GeoCircle either (d^4 against d^2).
• Likelihood: plausible. Zero-width paths are a documented, tested feature, but the squared styles have no in-tree caller and no report of the bug was found (see the prior reports).

The test sits in org.apache.lucene.spatial3d.geom only because upstream's TestGeoPath does; it uses public classes and methods only (GeoPathFactory, GeoPath, DistanceStyle, PlanetModel, GeoPoint). Nothing is randomized, so the result does not depend on the seed (it fails with DEADBEEF and 5EED0741).

On stock code it stops at the first assertion (path-center). The second assertion is the outside-distance fix site: with only the path-center line fixed it fails with expected:<9.999916666944447E-5> but was:<9.999833334583335E-9>, the same value the stock code returns. NORMAL_SQUARED goes through the same code and is left out.

The same file fails unchanged on the 10.6.0, 10.5.2, 9.12.3 and 9.5.0 release jars (run with assertions on) and passes on 9.4.2.

TestGeoDegeneratePathSquaredStyles > testZeroWidthPathLinearSquared FAILED
    java.lang.AssertionError: computePathCenterDistance(LINEAR_SQUARED) expected:<9.999916666944443E-5> but was:<0.009999958333385416>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:37E69C1AE26C7456]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:555)
        at org.apache.lucene.spatial3d.geom.TestGeoDegeneratePathSquaredStyles.testZeroWidthPathLinearSquared(TestGeoDegeneratePathSquaredStyles.java:31)

# with only the path-center line fixed, the second assertion:
    java.lang.AssertionError: computeOutsideDistance(LINEAR_SQUARED) expected:<9.999916666944447E-5> but was:<9.999833334583335E-9>

• The contract is one conversion back for a final result. DistanceStyle.java:107-148: "Distances should be converted to aggregation form before aggregation is attempted, and they should be converted back from aggregation form to yield a final result." The squared styles are the "Linear distance squared" and "Normal distance squared" calculators (:36-38, :44-46).
• Why aggregation forms exist. #8991 (LUCENE-7942, 2017, by the geo3d author): "The xxxSquaredDistance computations, for example, do not produce true squared distances but rather a distance metric that is a combination of both squared and linear." Its fix added the three conversion methods (lucene/CHANGES.txt:7864-7865).
• The same refactor converts back once in the sibling class. GeoStandardPath returns fromAggregationForm(...) from both methods (:278-285, :326-333); c9c27c755a4 ("Make sure use of aggregation form is consistent throughout ...") put those two conversions in a day before the introducing commit. (The same commit left a double conversion behind GeoStandardPath.computeDistance, LFV-271, so "consistent" holds for these two methods only.)
• 9.4.2 returned d^2: every piece returned the plain value, so the minimum was already the answer. The introducing commit converted the pieces but not the two places that combine them.
• No test pins the current values. No spatial3d test uses a squared style, nothing calls computePathCenterDistance, and computeOutsideDistance is tested only on boxes, circles and polygons, never on a path. The tests the commit changed compare paths with ARC, which cannot see the bug.

The strongest case for "intended" (the intended-behaviour check and the cross-model reviews): the geom API is @lucene.experimental and GeoDegeneratePath @lucene.internal; the path-center distance is only "meant to allow comparisons", and since sqrt is monotone, zero-width paths still rank correctly among themselves; the squared styles are second-class (they throw "Reverse mapping not implemented" from findDistancePoints and friends); and the aggregation rule speaks of sums, not minima.

Why it fails: experimental and internal allow API changes, not wrong values from the public GeoPath interface; the comparison the javadoc names is "between different paths", and a zero-width and a wider path swap order under LINEAR_SQUARED (executed); computeOutsideDistance has no comparison-only wording and returns d^4, which across the chord-1 crossover is not even one monotone function of d; the same author's design takes minima in aggregation form and converts back once (GeoStandardPath); and 9.4.2 returned the squares. Verdict: not intended, a defect (the independent reproduction; the refute-lens and intended-lens cross-model reviews; the intended-behaviour check; and the adjudication, with high confidence).

Two options were built and validated through gradle on 645d4fe14a20; a third is listed, not built. Workaround until fixed: call the method with LINEAR and square the result. NORMAL works the same way on the sphere, but has its own defect on ellipsoids such as WGS84 (LFV-740), so prefer LINEAR.

2 lines in GeoDegeneratePath.java (:179, :873), the pattern of GeoStandardPath and of SegmentEndpoint.outsideDistance. ARC, LINEAR and NORMAL are unchanged (identity conversions).

@@ computePathCenterDistance
-    return closestDistance;
+    return distanceStyle.fromAggregationForm(closestDistance);
@@ PathSegment.outsideDistance
-      return Math.min(Math.min(startDistance, endDistance), distance);
+      return distanceStyle.toAggregationForm(
+          Math.min(Math.min(startDistance, endDistance), distance));

Trade-offs

None known; local to the zero-width class. Code that worked around the wrong values (for example by squaring the path-center value itself) would have to drop that; none is known.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: the repro fails on stock and passes with A, assertions on and off.
• The whole :lucene:spatial3d:test with A and the repro (including TestGeoPath, TestRandomGeoShapeRelationship, TestGeo3DPoint, TestGeo3DDocValues): 20 suites, 182 tests, 0 failures, 3 skipped.
• :lucene:spatial3d:check -x test (checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest): passes.
• An independent reviewer's 7-test class (WGS84, both squared styles, both methods, a pole probe, two controls): 5 of 7 fail on stock 645d4fe14a20, 7 of 7 pass with A.
• Earlier, the same two-line change made every verifier's test pass: the independent reproduction 5 of 5 and the adjudication 12 of 12 on a9779ef3c193, a separate test 2 of 2 on ea57360912ed (lucene/spatial3d is the same on both).

Also return distanceStyle.toAggregationForm(distanceStyle.computeDistance(start, x, y, z)) from the pole branch of PathSegment.pathCenterDistance (:713), so that every piece returns aggregation form (3 lines in total).

Trade-offs

No executed input differs between A and B: in the pole probe the end points win the minimum. The same branch in GeoStandardPath (:1593) belongs to the other class and was not changed here.

What the checks validated

Gradle on 645d4fe14a20: the repro passes; :lucene:spatial3d:test 20 suites, 182 tests, 0 failures, 3 skipped; check -x test passes; the reviewer's class 7 of 7.

Since toAggregationForm is monotone, revert the three wrapped piece returns (:522, :536, :739) and the final fromAggregationForm (:286).

Trade-offs

More lines, and it diverges from the design the same refactor applied to GeoStandardPath; A is smaller.

Method: semi-formal (bounded-exhaustive checking of the real jar against exact semantics).

• A translation-validation harness recomputed the documented path distances with an independent BigDecimal reference (50-digit precision), the squared styles through the aggregation-form contract. On zero-width paths it flagged computePathCenterDistance: 2,487 LINEAR_SQUARED and 2,487 NORMAL_SQUARED mismatches on the sphere and 2,554 LINEAR_SQUARED on WGS84, off by ratios up to 1e9 (d returned for d^2).
• The harness checks outside distances in ARC only; the outside-distance half was found by reading the code while writing the aggregation-form part of the reference.
• The finder, an AI tool, then wrote a public-API JUnit on the stock jar. Independent reproducers wrote their own tests from the claim alone; the one-test repro was cut from the finder's test on 2026-10-29.