XYDocValuesField.newDistanceSort accepts a NaN or infinite origin instead of throwing

XYDocValuesField.newDistanceSort(field, x, y) documents "@throws IllegalArgumentException if {@code field} is null or location has invalid coordinates", word for word like its lat/lon twin. It returns new XYPointSortField(field, x, y), whose constructor checks only field != null, so a NaN or infinite x or y is accepted. The XY APIs around it that take coordinates (the XYDocValuesField constructor, XYPointField.newDistanceQuery, and the XYPoint, XYCircle, XYRectangle, XYLine and XYPolygon constructors) reject exactly these values through XYEncodingUtils.checkVal, and the lat/lon sort field validates its origin.

The search then runs. A document without the field has distance +Infinity; a located one has sqrt((x - docX)^2 + (y - docY)^2), which is NaN for a NaN origin and +Infinity for an infinite one. Double.compare orders NaN after +Infinity, so with a NaN origin documents without a location come first, against the documented "missing values sort last"; with an infinite origin everything ties and comes back in doc-id order. Only requests that use this sort are affected. It came with the XY sort in 8.5.0 (LUCENE-9238, a port of the lat/lon code that kept the @throws line but not the check) and is in every release through 10.6.0 and on main. Two XYEncodingUtils.encode calls in the constructor fix it; the fix passes the repro, the closest XY suites and check.

1. No check. XYDocValuesField.newDistanceSort is return new XYPointSortField(field, x, y); (document/XYDocValuesField.java:151-152). The constructor calls super(field, Type.CUSTOM, false, Double.POSITIVE_INFINITY) (the missing value), throws only when field == null, and stores x and y as given (document/XYPointSortField.java:28-35). The twin LatLonPointSortField calls GeoUtils.checkLatitude(latitude); GeoUtils.checkLongitude(longitude); at the same place (document/LatLonPointSortField.java:34-35), and checkLatitude rejects NaN explicitly (geo/GeoUtils.java:85-86).
2. The sort key. XYPointDistanceComparator.sortKey starts from Double.POSITIVE_INFINITY, which a document without the field keeps, and takes Math.min over Math.sqrt(diffX * diffX + diffY * diffY) with diffX = x - docX (document/XYPointDistanceComparator.java:190-209). For a NaN origin the distance is NaN and Math.min(Infinity, NaN) is NaN. For an infinite origin every distance is +Infinity, equal to the missing value.
3. The order. Slots are compared with Double.compare (:77-79), which orders NaN after +Infinity; a missing document is compared as +Infinity in compareBottom (:126-128). Ascending order therefore puts missing documents first under a NaN origin and ties everything under an infinite origin (ties go by doc id).
4. No crash, no hang. setBottom builds its pruning box only when bottom < Float.MAX_VALUE (:87-97), which is false for NaN and +Infinity, so nothing throws at search time and nothing loops. The box keeps its initial full range (:55-58), so compareBottom never skips a value by the box and computes the distance of every value it sees (:122-160): possibly more work than with a finite origin. The probe saw no timing blow-up; memory was not measured.

Documents: A at (1, 1), B without a location, C at (2, 2).

Controls in the same run: new XYDocValuesField("p", NaN, 0) and XYPointField.newDistanceQuery("p", NaN, 0, 1) throw "invalid value NaN; must be between -3.4028234663852886E38 and 3.4028234663852886E38"; LatLonDocValuesField.newDistanceSort("p", NaN, 0) throws "invalid latitude NaN; must be between -90.0 and 90.0".

• Who is hit. Applications that sort with XYDocValuesField.newDistanceSort and let a non-finite origin through: a reference point computed as 0/0 (for example the centroid of an empty set), or a request parameter "NaN" or "Infinity" parsed with Float.parseFloat and passed on unchecked. Default IndexWriterConfig, codec and searcher; the missing check is in the sort field's constructor, so it does not depend on the index; query-time only.
• What goes wrong. No exception, and the results come back in a meaningless order: documents without a location first (NaN origin) or plain doc-id order (infinite origin), with NaN or Infinity as the reported distances. An application that relies on the documented IllegalArgumentException to reject bad input never sees it.
• Limits. The sort field belongs to the request that builds it, so only requests that use it are affected. In the study's probe (20,000 documents, 8 segments, 7 origins; main with assertions on and off, 10.5.2 with assertions on) the matched set and hit count equalled a filter's count, searchAfter paging returned every match exactly once, a sliced concurrent search returned the same top 50 hits as the sequential one, and search time did not blow up. These are the tested workloads, not a proof for every index. The bad sort runs only at query time and does not change the index; index-time XY fields still reject non-finite values.
• Reach. A GitHub code search found no caller in Elasticsearch, OpenSearch or Solr (default branches, 2026-10-26), only copies of Lucene's own test elsewhere. That bounds the search; it does not prove there are no callers.
• Likelihood: rare. Normal code passes finite coordinates; the recorded searches found no public report since 8.5.0 (2020).

The test has the ASF license header, so it is also the regression test unchanged; with it in place, :lucene:core:check -x test passes. It fails on stock and passes with the recommended fix.

The test covers only (NaN, 0); (0, NaN) and the infinite origins are covered by the order probe below. The recommended fix rejects all four non-finite cases.

TestXYDistanceSortNaNOrigin > testNaNOriginRejected FAILED
    junit.framework.AssertionFailedError: Expected exception IllegalArgumentException but no exception was thrown
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:E2DEC189C27FB499]:0)
        at app//org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1432)
        at app//org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
        at app//org.apache.lucene.document.TestXYDistanceSortNaNOrigin.testNaNOriginRejected(TestXYDistanceSortNaNOrigin.java:25)

:lucene:core:test (FAILURE): 1 test, 1  failure

# the same failure with -Ptests.asserts=false, and on the 10.6.0 release jars (assertions on and off)

# order probe on the same main, assertions on (identical with assertions off and on 10.6.0):
origin=(0.0, 0.0) accepted, order: A=1.4142135623730951 C=2.8284271247461903 B(no location)=Infinity
origin=(NaN, 0.0) accepted, order: B(no location)=Infinity A=NaN C=NaN
origin=(0.0, NaN) accepted, order: B(no location)=Infinity A=NaN C=NaN
origin=(Infinity, 0.0) accepted, order: A=Infinity B(no location)=Infinity C=Infinity
origin=(0.0, -Infinity) accepted, order: A=Infinity B(no location)=Infinity C=Infinity
origin=(-3.4028235E38, 0.0) accepted, order: A=3.4028234663852886E38 C=3.4028234663852886E38 B(no location)=Infinity
control new XYDocValuesField("p", NaN, 0): rejected: invalid value NaN; must be between -3.4028234663852886E38 and 3.4028234663852886E38
control XYPointField.newDistanceQuery("p", NaN, 0, 1): rejected: invalid value NaN; must be between -3.4028234663852886E38 and 3.4028234663852886E38
control LatLonDocValuesField.newDistanceSort("p", NaN, 0): rejected: invalid latitude NaN; must be between -90.0 and 90.0

• The method's own contract. "@throws IllegalArgumentException if {@code field} is null or location has invalid coordinates." (XYDocValuesField.java:149) and "missing values sort last" (:139-140).
• What "invalid" means for XY. The same class's constructor and setLocationValue: "@throws IllegalArgumentException if ... x or y are infinite or NaN" (:79, :91), enforced through XYEncodingUtils.encode (:94-95), which calls checkVal, "validates value is a number and finite" (XYEncodingUtils.java:34-41). XYPoint and XYCircle call checkVal too.
• The sibling query enforces the identical wording, and a test checks it. XYPointField.newDistanceQuery: "@throws IllegalArgumentException if {@code field} is null, location has invalid coordinates, or radius is invalid" (XYPointField.java:174-175), enforced by new XYCircle (:178). Upstream BaseXYPointTestCase.testDistanceIllegal, "distance query should not accept invalid x/y as origin", passes a NaN origin. No upstream test passes a non-finite origin to the XY sort.
• The twins validate. LatLonPointSortField has checked its origin since its first commit (LUCENE-7099, 2016), and the Geo3D distance sort goes through Geo3DUtil.fromDistance, which calls GeoUtils.checkLatitude/checkLongitude.
• A port that lost the check. LUCENE-9238 (PR apache/lucene-solr#1272, whose description reads "Adaptation of LatLonPoint to generic cartesian space.") copied the @throws line word for word but not the check. That the check was lost in the port is an inference from the identical javadoc and the validating twin, not a stated intent; the PR has no discussion of validation.
• The finite-only rule predates the sort. XYEncodingUtils.checkVal has rejected NaN and values beyond +/-Float.MAX_VALUE (so both infinities) since it was added with XYEncodingUtils in LUCENE-8632 (0c09481374c, 2019-07-08). One week before the sort, LUCENE-9218 (apache/lucene-solr#1252, merged 2020-02-14) rewrote it for floats as Float.isFinite(x) == false, "validates value is a number and finite"; in that PR the author had written Float.isNaN(x) || Float.isInfinite(x) and a reviewer suggested the single test.

The strongest case: the XY javadoc is lat/lon boilerplate (it still says the sort value is a "distance in meters", XYDocValuesField.java:137, although x/y coordinates have no unit), XYPointSortField is package-private, NaN in means garbage out, and Double.compare is a total order, so the result is at least deterministic. Finiteness is an XY-family rule, not a Lucene-wide one: the generic FloatPoint deliberately accepts NaN and the infinities (TestPointQueries.testCrazyFloats), and a caller could be using a NaN origin as a sentinel today. The cross-model reviews rated this case weak.

Why it fails: the @throws clause is exactly what the siblings enforce (the lat/lon and Geo3D sorts, the XY field, point, circle and distance-query APIs, the same class's constructor). The fix rejects only non-finite origins; the finite origins probed are unchanged (an origin at -Float.MAX_VALUE still sorts with missing last, with and without the fix). A caller that relies on a non-finite origin would now get the exception, a compatibility change. It is a good reason for low priority, not for keeping the gap.

The recommended fix was run in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF), removed afterwards. "Closest XY suites" means TestXYPointDistanceSort, TestXYDocValuesQueries, TestXYPointQueries, TestXYPoint and TestXYCircle in lucene/core.

XYEncodingUtils.checkVal is package-private in org.apache.lucene.geo; encode is public and calls it, exactly as XYDocValuesField.setLocationValue does. The error is the one new XYDocValuesField gives: "invalid value NaN; must be between -3.4028234663852886E38 and 3.4028234663852886E38". This is the finder's fix with a shorter comment. The diff shown is complete; its comment was reworded after the fact check, and it differs from the gradle-validated diff only in that comment line.

--- a/lucene/core/src/java/org/apache/lucene/document/XYPointSortField.java
+++ b/lucene/core/src/java/org/apache/lucene/document/XYPointSortField.java
@@ -16,6 +16,7 @@
  */
 package org.apache.lucene.document;
 
+import org.apache.lucene.geo.XYEncodingUtils;
 import org.apache.lucene.search.FieldComparator;
 import org.apache.lucene.search.Pruning;
 import org.apache.lucene.search.SortField;
@@ -30,6 +31,9 @@
     if (field == null) {
       throw new IllegalArgumentException("field must not be null");
     }
+    // reject NaN and infinite x/y, as XYDocValuesField does (XYEncodingUtils.checkVal)
+    XYEncodingUtils.encode(x);
+    XYEncodingUtils.encode(y);
     this.x = x;
     this.y = y;
   }

Trade-offs

• A caller that passes a non-finite origin now gets the documented exception instead of a meaningless order: a visible behaviour change worth a CHANGES entry, and a break for any caller that used a NaN or infinite origin on purpose (say, as a sentinel). The finite origins probed give the same results: an origin at -Float.MAX_VALUE still sorts, missing last. Constructor-time cost only.
• Calling encode for its side effect reads slightly odd; the alternatives below avoid that.
• Open PR #14775 ("Add the ability to inverse a Sort") also edits XYPointSortField.java, outside the constructor, so a fix PR conflicts at most trivially.

What the checks validated

• Stock 645d4fe14a20: the repro fails (1 of 1) with assertions on and off, through gradle and the standalone runner; also on the 10.6.0 jars.
• With the fix: the repro passes with assertions on and off; the 5 closest XY suites pass (90 tests, 5 skipped, 0 failures); :lucene:core:check -x test (google-java-format, ecj lint main and test, forbidden APIs, dependency license check) passes with the repro test in place. These are targeted runs, not the full Lucene test suite.
• Probe with the fixed file shadowed: (NaN, 0), (0, NaN), (Infinity, 0) and (0, -Infinity) are rejected; (0, 0) and (-Float.MAX_VALUE, 0) give the same order and distances as stock.
• The diff shown (reworded comment) applies to 645d4fe14a20 with git apply --check, is google-java-format clean, and passes the minimal repro with assertions on and off (stock still fails both).

new XYPoint(x, y); validates both values with the same checkVal and the same message.

Trade-offs

Reads as "the origin must be a valid point", at the cost of one small allocation per sort field.

if (Float.isFinite(x) == false || Float.isFinite(y) == false) throw new IllegalArgumentException(...).

Trade-offs

Clearest intent, but a new message unlike the rest of the XY family.

Would keep missing documents last for a NaN origin.

Trade-offs

Not enough: it hides the bad input instead of rejecting it, and the javadoc promises an exception.

Method: non-formal (the diff is an automated code-reading aid, not a model, a proof or a bounded-exhaustive check).

• The unit's twin-diff tool normalised the lat/lon and x/y sibling classes of the geo family and listed every member that differs beyond the expected type and name changes. In LatLonPointSortField vs XYPointSortField, the GeoUtils.checkLatitude/checkLongitude lines exist only in the lat/lon twin.
• The finder then ran a targeted public-API JUnit on the stock jar with the non-finite values (NaN, +Infinity, -Infinity): newDistanceSort accepted all of them, and a NaN origin put the document without a location first. An independent reproducer wrote its own test (all six non-finite origins accepted, controls passing), a devil's advocate and an adjudication confirmed it, and the repro was later cut to the one-line test above.