Geo3DPointOutsideSortField.equals casts to the sibling class, so comparing two outside-distance sorts throws ClassCastException

Geo3DDocValuesField has five public factories that sort hits by distance outside a shape, newOutside{Distance,Box,Polygon,LargePolygon,Path}Sort, each returning a package-private Geo3DPointOutsideSortField as a plain SortField. Its equals (Geo3DPointOutsideSortField.java:63-70 at 645d4fe14a20) checks getClass() != obj.getClass() (:67) and then casts the argument to the sibling inside-distance class Geo3DPointSortField (:68). The two are unrelated final subclasses of SortField, so the cast can never succeed, and because SortField.equals passes for any two outside sorts on the same field, comparing any two of them throws ClassCastException: directly, through Sort.equals, or in a HashMap keyed by Sort. The line was copied from Geo3DPointSortField.equals, where it names its own class. It dates from LUCENE-7212 (2016) and is in every release from 6.1.0 to 10.6.0 and on main. Searching and sorting never reach the cast, Lucene's own code never compares two such sorts, and equals never returns a wrong true; the fix is one line.

1. Every outside sort on a field looks the same to SortField.equals. The constructor calls super(field, SortField.Type.CUSTOM, false, Double.POSITIVE_INFINITY) (Geo3DPointOutsideSortField.java:32). SortField.equals (SortField.java:451-461) compares only field, type, reverse, comparator source and missing value, and SortField has no setter for them on main. So super.equals is true for any two outside sorts on the same field.
2. The method then checks the class and casts to the wrong one (Geo3DPointOutsideSortField.java:63-70):

public boolean equals(Object obj) {
  if (this == obj) return true;                                // :65
  if (!super.equals(obj)) return false;                        // :66
  if (getClass() != obj.getClass()) return false;              // :67  obj is a Geo3DPointOutsideSortField
  final Geo3DPointSortField other = (Geo3DPointSortField) obj; // :68  unrelated final class: always throws
  return distanceShape.equals(other.distanceShape);            // :69  never reached
}

Geo3DPointOutsideSortField (:26) and Geo3DPointSortField (Geo3DPointSortField.java:26) are both final class ... extends SortField, so no object is both. The jar's bytecode has checkcast org/apache/lucene/spatial3d/Geo3DPointSortField at line 68.

1. hashCode is value-based (:53-61, over the shape), so two identical outside sorts land in the same HashMap bucket and the lookup calls equals (a lookup with a different shape normally has a different hash code, so the map does not call equals for it). Sort.equals compares the field arrays with Arrays.equals (Sort.java:119), which calls each SortField.equals.
2. What does not throw: the same instance (:65; Arrays.equals also short-circuits on identical references), another field name or null (:66), an inside sort (:67).
3. Lucene's own code never reaches the cast. Its non-sandbox Sort comparisons are the early-termination check of sorted search, TopFieldCollector.canEarlyTerminateOnPrefix (TopFieldCollector.java:160-172, comparison at :168), which on every sorted search over an index-sorted segment compares the search sort fields with the segment's index-sort fields and so does call this equals; IndexWriter.isCongruentSort (IndexWriter.java:1232-1239) and the index-sort check in ParallelLeafReader.java:136, which compare index sorts only; and grouping's comparisons with Sort.RELEVANCE. An outside sort can never be an index sort (IndexWriterConfig.setIndexSort rejects a sort field with no IndexSorter, IndexWriterConfig.java:505-509, and SortField.getIndexSorter returns null for CUSTOM, SortField.java:598-604), so in the early-termination check equals returns false at :66 or :67, before the cast; Sort.RELEVANCE's field is null, so super.equals returns false first. An instrumented copy of the class showed it: a sorted search over an index-sorted index called this equals once (from canEarlyTerminateOnPrefix) and without an index sort not at all. An executed probe ran sorted search (10 hits returned) and searchAfter (does not throw) with outside sorts, alone and with a numeric sort, on a multi-segment index, with and without an index sort and an executor (3 of 3 pass on ea57360912ed and again on 645d4fe14a20). (Whether searchAfter pages correctly with these sorts is a separate bug, LFV-544.)

• Who is hit. Applications that sort with one of the five outside-distance sorts and whose own code compares two separately built ones on the same field: a result or request cache keyed by Sort (or by query plus sort), code that de-duplicates or compares requests, or an assertEquals in the application's tests. Every such comparison throws, deterministically; it does not depend on the data, an index or assertions.
• Default vs opt-in. Opt-in and rare. spatial3d is an optional module, and these sorts look like a niche corner of it, judging by the code inspected (real-world usage volume is unknown): no upstream test builds any Geo3D sort field, and read-only code searches of Elasticsearch, OpenSearch and Solr found no use of Geo3DDocValuesField or of these factories (not exhaustive). Within that, it needs nothing special: the default PlanetModel.WGS84, ordinary arguments, no index.
• Effect. A loud unchecked exception from equals. No wrong results, no hang, no resource use, no index effect, and never a wrong true, so a Sort-keyed cache cannot hand one request's results to another.
• Workaround. Reuse one SortField instance per distinct sort (self-comparison returns true before the cast), or key caches on the sort's parameters instead of the Sort object.

The try/catch only turns the exception into an AssertionError that states expected vs actual; the cause keeps the original stack. The test sits in org.apache.lucene.spatial3d for locality, as Lucene's own tests do, but uses no package-private member. Nothing random reaches the code path; the seed only satisfies the test framework.

Suggested regression test: this same test, as its own class or as one method in TestGeo3DDocValues.

# main 645d4fe14a20, gradle, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestGeo3DOutsideSortFieldEquals > testIdenticalOutsideDistanceSortsAreEqual FAILED
    java.lang.AssertionError: expected a.equals(b) to return true, got java.lang.ClassCastException: class org.apache.lucene.spatial3d.Geo3DPointOutsideSortField cannot be cast to class org.apache.lucene.spatial3d.Geo3DPointSortField (org.apache.lucene.spatial3d.Geo3DPointOutsideSortField and org.apache.lucene.spatial3d.Geo3DPointSortField are in unnamed module of loader 'app')
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:8165F61726058C4A]:0)
        at org.apache.lucene.spatial3d.TestGeo3DOutsideSortFieldEquals.testIdenticalOutsideDistanceSortsAreEqual(TestGeo3DOutsideSortFieldEquals.java:34)

        Caused by:
        java.lang.ClassCastException: class org.apache.lucene.spatial3d.Geo3DPointOutsideSortField cannot be cast to class org.apache.lucene.spatial3d.Geo3DPointSortField (org.apache.lucene.spatial3d.Geo3DPointOutsideSortField and org.apache.lucene.spatial3d.Geo3DPointSortField are in unnamed module of loader 'app')
            at org.apache.lucene.spatial3d.Geo3DPointOutsideSortField.equals(Geo3DPointOutsideSortField.java:68)
            at junit@4.13.2/org.junit.Assert.isEquals(Assert.java:133)
            at junit@4.13.2/org.junit.Assert.equalsRegardingNull(Assert.java:129)
            at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:112)
            at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
            at org.apache.lucene.spatial3d.TestGeo3DOutsideSortFieldEquals.testIdenticalOutsideDistanceSortsAreEqual(TestGeo3DOutsideSortFieldEquals.java:32)
:lucene:spatial3d:test (FAILURE): 1 test, 1  failure

# the same ClassCastException at Geo3DPointOutsideSortField.java:78 with the 10.6.0 (-ea, -da) and 10.5.2 (-ea) release jars
# with the one-line fix: 1 test, 0 failures (assertions on and off)

• The class defines value equality. It overrides hashCode over distanceShape (Geo3DPointOutsideSortField.java:53-61), and its equals ends by comparing distanceShape (:69). A method that can never reach its own last line for two objects of its own class is not a design.
• The contract promises a boolean. SortField.equals: "Returns true if o is equal to this" (SortField.java:447); Sort.equals: the same (Sort.java:112), comparing the field arrays element by element (:119). The Object#equals general contract describes an equivalence relation; for two instances of this class equals gives no answer at all. Nothing (javadoc, comment, CHANGES.txt, MIGRATE.md, test) documents identity-only equality or an exception for these sorts; the factory javadoc covers only ordering, missing and multiple values and argument checks. Geo3DDocValuesField is public, in an exported package, with no @lucene.experimental or @lucene.internal tag (the PlanetModel that every factory takes as an argument is tagged @lucene.experimental, PlanetModel.java:26).
• Every sibling uses the matching cast after the same guard: Geo3DPointSortField.java:68, LatLonPointSortField.java:67, XYPointSortField.java:64.
• Upstream tests treat sort equality as a contract: TestSort.testEquals (TestSort.java:73-92) asserts that separately built equal sorts are equal and different ones unequal, in both directions; TestSortedNumericSortField.testEquals and TestSortedSetSortField.testEquals do the same for single sort fields, in one direction. Sort/SortField value equality itself was added by LUCENE-1481 (#2555).
• History shows a copy slip. Geo3DPointSortField was added on 2016-05-17 (07af00d8e7bc) with a correct equals; the next day cc263413457c added Geo3DPointOutsideSortField and copied that method, cast included (git blame -C -C traces line 68 to Geo3DPointSortField.java at 07af00d8e7bc). The LUCENE-7212 discussion (#8267, 49 comments) never mentions equals, hashCode or a cast, and no later commit touched the line.
• Precedent. #12827 (merged 2023-11-21, in 9.9.0) fixed the same kind of slip, an equals that casts to a class it is not a subtype of, in PayloadFilteredTermIntervalsSource, with a one-line change.

Strongest case (from the intended-behaviour checker and the cross-model reviews): the class is package-private, no Lucene code compares two of these sorts, no test builds one, the factories take an @lucene.experimental PlanetModel, no production user has been shown, and nobody has reported it since 2016, so the method is effectively unused and harmless.

Why it fails: that is about impact, not intent. Users get the object through public, documented, untagged factories (only their PlanetModel argument type is experimental) as a public SortField, and SortField.equals and Sort.equals are public, documented and tested; a cast that cannot succeed right after a same-class guard has no intended reading. The low impact is reflected in the priority (low), not in the verdict. The cross-model lenses rated the intended argument weak.

The recommended fix was applied with git apply to a scratch worktree of 645d4fe14a20 and run through gradle; the alternative was not run.

In Geo3DPointOutsideSortField.equals (:68), cast to the class's own type, as the sibling classes do.

  @Override
  public boolean equals(Object obj) {
    if (this == obj) return true;
    if (!super.equals(obj)) return false;
    if (getClass() != obj.getClass()) return false;
    final Geo3DPointOutsideSortField other = (Geo3DPointOutsideSortField) obj;
    return distanceShape.equals(other.distanceShape);
  }

Trade-offs

No adverse trade-off was identified in the reviewed code and tests. For the tested specifications (all five factories, identical and different shapes), equals then returns true for identical specs and false for different shapes, matching its value-based hashCode and the sibling classes. No other code changes.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20 with the repro test added, seed DEADBEEF: the repro test passes with assertions on and off; TestGeo3DDocValues (the closest existing suite) passes; the whole :lucene:spatial3d:test (20 classes, 182 tests including the repro) passes with assertions on: 0 failures, 3 skipped. The same suite without the fix: 1 failure, the repro.
• :lucene:spatial3d:checkGoogleJavaFormat passes with the fix.
• Standalone runner with the fixed class shadowed: a five-factory probe passes 5 of 5 (identical specs true, different shapes false, Sort.equals true, the HashMap lookup finds the key), and an independent verifier's 9-test class passes 9 of 9.

Replace lines 67-68 with if (!(obj instanceof Geo3DPointOutsideSortField other)) return false;, the style SortField.equals itself uses (o instanceof SortField other, SortField.java:453). Not run.

Trade-offs

Equivalent here because the class is final; a two-line change instead of one, and it makes this exact slip impossible to repeat.

Method: non-formal (static analysis, then a directed JUnit), 2026-10-26, main ea57360912ed. A sweep of the non-sandbox module jars with SpotBugs 4.10.5 (-effort:max -low) reported EQ_CHECK_FOR_OPERAND_NOT_COMPATIBLE_WITH_THIS (rank 14) at Geo3DPointOutsideSortField.java:68. A custom bytecode scan of all 599 equals(Object) methods in core and the non-sandbox modules, looking for a cast to a type the declaring class is not a subtype of, found exactly one hit, this one. The finder's JUnit (identical distance sorts, different box sorts, Sort.equals over polygon sorts, an inside-sort control) then failed 3 of 4 on main with -ea and -da and on 10.5.2 and 9.12.3. A later minimisation step replaced it with the one-method test shown here.