Geo3dDistanceCalculator.within returns the opposite answer: a point 1 degree away is not within 10 degrees

Geo3dSpatialContextFactory.newSpatialContext() installs a Geo3dDistanceCalculator as the context's distance calculator. Its within(from, toX, toY, distance) is return (distance < distance(from, toX, toY));: the parameter distance is the limit, so the line returns "limit < actual distance", the exact negation of spatial4j's documented "true if the distance between from and to is <= distance". Every answer with a finite distance and a non-NaN limit is inverted: from (0, 0), the point (1, 0) at 1.0 degree is "not within 10", the point (50, 0) at about 50 degrees is "within 10", and within(p, p, 0) is false.

The reach is narrow. No Lucene code calls DistanceCalculator.within: a counting calculator recorded 0 calls while indexing 2,000 points into eight strategies and running 63 queries and 7 distance sorts, so this bug does not change search results with shapes from the Geo3d shape factory, and no index data depends on it. (Those factory shapes do over-match on RPT over PackedQuadPrefixTree, 419 hits where 7 are inside, but through the separate Geo3dShape.relate bug LFV-673, which this fix does not change.) Applications that call ctx.getDistCalc().within(...) themselves get the opposite answer whenever neither the distance nor the limit is NaN. The one search-result path found needs application code that hand-builds spatial4j's CircleImpl with the Geo3d context: as a query on RPT over the non-default PackedQuadPrefixTree, such a circle matched 419 documents where 7 are inside.

The line has been there since the class was written (LUCENE-7951, 2017-09-20, first released in 7.1.0). Affected: 7.1.0 to 10.6.0 and main; not fixed upstream. Fix: return distance(from, toX, toY) <= distance; (one line).

1. The Geo3d context installs this calculator by default. The factory constructor sets the Geo3d shape factory and binary codec (Geo3dSpatialContextFactory.java:39-42). newSpatialContext() defaults the planet model to PlanetModel.SPHERE (:33, :46-48) and, when no calculator is configured, sets distCalc = new Geo3dDistanceCalculator(planetModel) (:49-51). The configuration path (SpatialContextFactory.makeSpatialContext with spatialContextFactory set to this class) ends in the same method; initCalculator also installs it for distCalculator=geo3d (:78-88).
2. distance() is right. distance(Point from, double toX, double toY) returns the planet model's surface distance in degrees (Geo3dDistanceCalculator.java:54-72): 1.0 for (0, 0) to (1, 0) and 49.99999999999999 for (0, 0) to (50, 0). In an independent check its distances agreed with spatial4j's geodesic calculator to about 1e-14 degrees.
3. within() compares the wrong way round (:74-77):

@Override
public boolean within(Point from, double toX, double toY, double distance) {
  return (distance < distance(from, toX, toY));
}

The parameter distance is the limit and distance(from, toX, toY) the method of step 2, so the line returns "limit < actual distance". When neither the distance nor the limit is NaN, that is the logical complement of "actual distance <= limit", the boundary actual == limit included. When either is NaN, both the stock line and the fix return false. The released bytecode (7.1.0 and main jars) is the same comparison; spatial4j's AbstractDistanceCalculator.within returns distance(from, toX, toY) <= distance.

1. Nothing in Lucene calls it. The in-tree main-source users of ctx.getDistCalc() call only distance() (SpatialStrategy.java:135-140, SpatialArgs.java:69, DistanceToShapeValueSource.java:76, ShapeFieldCacheDistanceValueSource.java:72-74, DistanceValueSource.java:75); the tests also call pointOnBearing, and nothing under lucene/, test or main, calls within on a calculator. In spatial4j 0.8 the only caller is CircleImpl.contains(double, double), used by CircleImpl.relate; the Geo3d shape factory's circle(...) returns a Geo3dCircleShape (Geo3dShapeFactory.java:176-196, :200-202), never a CircleImpl.

Changing only that line to return distance(from, toX, toY) <= distance; makes the repro pass on main, 10.5.2 and 7.1.0, so the comparison is the whole cause.

With the fix the first three rows give true, false, true, and the random and reference checks all pass.

• Silent, every call. For a direct caller every answer with a finite distance and a non-NaN limit is the opposite one, with no exception.
• Who is exposed. Applications that chose the opt-in Geo3d spatial context (Geo3dSpatialContextFactory, @lucene.experimental; the usual spatial4j context is SpatialContext.GEO) and call ctx.getDistCalc().within(...). Solr exposes the Geo3d context through its spatialContextFactory="Geo3D" alias (SOLR-11382). A read-only GitHub code search found no call of getDistCalc().within other than spatial4j's own CircleImpl and copies of it.
• This bug does not change Lucene's own search results for shapes built by the Geo3d shape factory. A counting subclass installed as the context's calculator recorded 0 within() calls while indexing 2,000 points into eight strategies (RPT over Geohash, Quad, PackedQuad and S2, PointVector, BBox, SerializedDV, Composite), and across 63 queries and 7 distance sorts. No index data depends on within(), so nothing needs reindexing after the fix.
• Not this bug: factory shapes on PackedQuad. In the same probe, factory circles on RPT over PackedQuadPrefixTree do over-match: circle(40, 40, 5) matched 419 documents where 7 are inside, and circle(-100, -30, 20) 428 where 49 are, with 0 within() calls. This fix leaves both numbers unchanged; only the fix of the sibling bug LFV-673 (Geo3dShape.relate with a plain rectangle) brings them to 7 and 51. The two bugs are separate.
• The one search-result path found (uncommon). Application code that bypasses the shape factory with new CircleImpl(center, radius, geo3dCtx) gets inverted contains/relate answers. As an Intersects query on RPT over PackedQuadPrefixTree, CircleImpl(40, 40, 5) matched 419 documents where 7 are inside (7 with the fix), and CircleImpl(-100, -30, 20) matched 428 where 49 are inside (52 with the fix, the normal grid approximation): every live document of one quarter-world cell. RPT over Geohash, Quad and S2, SerializedDV and Composite reject a CircleImpl with a clean exception, and BBox with an UnsupportedOperationException, with or without the fix; PointVector filters by distance() and is exact.
• Likelihood: rare. No report since 7.1.0 (2017).

The test sits in org.apache.lucene.spatial.spatial4j, next to the class, and uses only public methods. The result does not depend on the seed (it fails with DEADBEEF and 5EED1234). The modules it needs (spatial-extras, spatial3d, spatial4j, the test framework) are already on the module's test classpath.

The same file fails unchanged on the 10.6.0, 10.5.2 and 9.12.3 release jars (assertions on). For 9.0.0 and older the LuceneTestCase import is org.apache.lucene.util.LuceneTestCase; the history sweep ran that variant on every release from 7.1.0.

TestGeo3dDistanceCalculatorWithin > testWithin FAILED
    java.lang.AssertionError: (1, 0) is 1 degree from (0, 0), so it is within 10 degrees expected:<true> but was:<false>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:73EFDB7B71364A45]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
        at org.apache.lucene.spatial.spatial4j.TestGeo3dDistanceCalculatorWithin.testWithin(TestGeo3dDistanceCalculatorWithin.java:29)

• The contract is explicit and has not changed. spatial4j DistanceCalculator.within: "Returns true if the distance between from and to is &lt;= distance." (DistanceCalculator.java:29-30 in spatial4j 0.8). In 0.6 and 0.7, which 7.1.0 to 8.11.4 shipped with, AbstractDistanceCalculator.within has the same bytecode (checked in the binary jars; their javadoc text was not checked). AbstractDistanceCalculator.within is return distance(from, toX, toY) <= distance; (:23-25).
• The Lucene class claims that contract. Its javadoc is "Geo3d implementation of {@link DistanceCalculator}" (Geo3dDistanceCalculator.java:30-34) and gives within no other meaning. The 7.1.0 CHANGES entry for LUCENE-7951 says spatial-extras now implements "Spatial4j abstractions: SpatialContextFactory, ShapeFactory, BinaryCodec, DistanceCalculator" (lucene/CHANGES.txt:7883-7885).
• The boundary case is wrong too. within(p, p, 0) is false at distance 0.
• Nothing documents or accepts it. No test, CHANGES or MIGRATE entry mentions within on this class. The introducing review (#9000 and lucene-solr#246) discussed other methods. The methods the class does not support throw UnsupportedOperationException (:104-117); within returns a value.

The strongest case for "intended" (the intended-behaviour check and the cross-model reviews): both classes are @lucene.experimental ("This API is experimental and might change in incompatible ways in the next release"); the package says it holds "Spatial4j stuff that ideally belongs in Spatial4j (isn't related to Lucene)"; the calculator is partly unimplemented; the contributor wrote on #9000 that "the DistanceCalculator needs to be reviewed"; nothing in Lucene calls within(); and nobody has complained in nine years.

Why it fails: experimental warns that the API may change, not that a method may return the opposite of its interface's documented result; the code ships in the regular spatial-extras module and CHANGES advertises it as a DistanceCalculator; "needs review" shows the calculator was unreviewed, not that the inversion was meant; no caller and no complaints limit the impact, which is why the class is minor. Verdict: not intended, a defect (the independent reproduction, the intended-behaviour check, the refute-lens and intended-lens cross-model reviews and the adjudication; the adjudication with high confidence).

One option was built and validated through gradle on 645d4fe14a20; three alternatives are listed, not built.

1 line in Geo3dDistanceCalculator.java, the same body as spatial4j's AbstractDistanceCalculator.within.

  @Override
  public boolean within(Point from, double toX, double toY, double distance) {
-    return (distance < distance(from, toX, toY));
+    return distance(from, toX, toY) <= distance;
  }

Trade-offs

None known. No Lucene code calls within(), so nothing in Lucene changes behaviour. An application that worked around the inversion by negating the result would have to drop its negation; none is known.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: the repro fails on stock and passes with the fix, assertions on and off.
• The whole :lucene:spatial-extras:test with the fix and the repro (including TestGeo3d, TestGeo3dRpt, TestGeo3dShapeSphereModelRectRelation, TestGeo3dShapeWGS84ModelRectRelation): 27 suites, 118 tests, 5 skipped, 0 failures.
• :lucene:spatial-extras:check -x test (checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest): passes.
• Earlier, the same change made every verifier's test pass and fixed the probe's answers, and the finder's run of the full module with all six fixes from the same hunt (on main ea57360912ed, whose spatial-extras and spatial3d sources are identical to a9779ef3c193's) gave 117 tests, 0 failures, 5 skipped, as on the unpatched baseline. Shadowed into the 7.1.0 and 10.5.2 sources, it turns the repro from fail to pass.

Name the parameter maxDistance (return distance(from, toX, toY) <= maxDistance;). It removes the shadowing of the method name distance that makes the inverted line easy to misread; Java allows a different parameter name in an override.

Trade-offs

Cosmetic only.

AbstractDistanceCalculator already implements within with <=.

Trade-offs

Changes the superclass of a public, if experimental, class to fix one line; A is smaller.

For an experimental, partly unimplemented class a maintainer may prefer to make within throw UnsupportedOperationException, as area(...) and calcBoxByDistFromPt_yHorizAxisDEG already do (Geo3dDistanceCalculator.java:104-117), or to deprecate it.

Trade-offs

Throwing turns the silent wrong answer into a loud failure, but also makes a hand-built CircleImpl with this context throw from contains and relate (spatial4j's CircleImpl.contains calls within) and breaks any caller that uses within today; deprecating alone leaves the wrong answers in place. A is just as small and keeps the interface contract.

Method: non-formal (a hunt unit, a differential check of the spatial4j-to-spatial3d bridge; the finder tagged it semi-formal, the registry records it as non-formal).

• The harness checked self-consistency properties of the Geo3d bridge. Family G4a: within(from, to, lim) must equal distance(from, to) <= lim, with lim just above or just below the measured distance, on SPHERE and WGS84 (seed 5EED220): 5,000 of 5,000 failures per model on stock main, 0 with the fix. The sibling family G4b, distance(from, pointOnBearing(from, d, b)) equals d, was clean.
• The finder, an AI tool, then wrote a public-API JUnit on the stock jar. Independent reproducers wrote their own tests from the claim alone; the one-test repro was cut from the finder's test on 2026-10-29.