Geo3D distance sorts read a hit's points twice when a segment has a multi-valued document, so hits get another document's distance

Geo3DDocValuesField allows several points per document and promises that its sorts use the closest one (Geo3DDocValuesField.java:34-35, :173-174). TopFieldCollector can ask the comparator about the same hit two or three times: compareBottom (once the queue is full) and/or compareTop (with searchAfter), then copy. Geo3DPointDistanceComparator and Geo3DPointOutsideDistanceComparator read the document's values with SortedNumericDocValues.nextValue() in each call and never rewind, although nextValue() must not be called more than docValueCount() times per document. The default codec does not check, so a later call reads the next docValueCount() values, which belong to the following documents; copy stores that distance as the hit's sort value, and the order follows it; past the last value the search throws (RuntimeException(EOFException) on file directories on 9.12.3 and later, IndexOutOfBoundsException on 9.0.0 and on a plain heap directory; 7.x and 8.x return a wrong value). In the repro, top 2 of {1}, {2}, {9, 0.5}, {5}, {6} degrees returns doc 0 at 1° and doc 2 at 5° (doc 3's point) instead of doc 2 at 0.5° and doc 0 at 1°. It needs a segment with at least one multi-valued document and a hit that gets two or more of those calls, which a top-N smaller than the hit count or searchAfter makes possible. LUCENE-7407 (f7aa200d406, 7.0.0) introduced it when it moved doc values to iterators and, in the same commit, gave the 2D twin LatLonPointDistanceComparator a per-document value cache; the Geo3D comparators never got one. Adding the same cache fixes it.

• Geo3D doc-values sort: Geo3DDocValuesField stores each point as one encoded long in a SORTED_NUMERIC field; "Multiple values for the same field in one document is allowed" (Geo3DDocValuesField.java:34-35). Seven factories build a SortField (newDistanceSort, newPathSort, newOutsideDistanceSort, newOutsideBoxSort, newOutsidePolygonSort, newOutsideLargePolygonSort, newOutsidePathSort), each promising "If a document contains multiple values for the field, the closest distance ... is used", with that distance returned in FieldDoc (:167-174 and siblings). They use two comparators, Geo3DPointDistanceComparator and Geo3DPointOutsideDistanceComparator.
• The cursor: SortedNumericDocValues.nextValue() "Iterates to the next value in the current document. Do not call this more than docValueCount times for the document" (SortedNumericDocValues.java:28-32). In the default codec it is values.get(start++) with no bound (Lucene90DocValuesProducer.java:2069-2072). If every document of a segment has at most one value, the codec returns a singleton wrapper whose nextValue() returns the same value every time (:2002-2004).
• The repeated question: once the queue is full, TopFieldCollector calls compareBottom and then copy for the same competitive document (TopFieldCollector.java:92-93, 114-118, 203-213); with searchAfter it calls compareTop and then, for a hit after the cursor, copy, with compareBottom first once the queue is full, so an accepted hit gets three calls (:268-291). Only before the queue fills and without searchAfter is a hit just copied, with one call.

1. compareBottom advances to the document if the cursor is behind and reads its docValueCount() values (Geo3DPointDistanceComparator.java:97-134, read at :110).
2. copy calls computeMinimumDistance(doc) (:137-139). It advances only when doc > currentDocs.docID(), which is false for the same document, and reads docValueCount() more values (:163-182, read at :171). compareTop goes through the same method (:159-161).
3. Those extra reads return the next docValueCount() values in storage order, which belong to the following document or documents that have the field; past the last value they throw: on main and 10.5.2, RuntimeException(EOFException) on file directories and on the test framework's heap directory (MockDirectoryWrapper over ByteBuffersDirectory), IndexOutOfBoundsException on a plain ByteBuffersDirectory.
4. The slot stores the minimum of those values; value(slot) returns it as the hit's FieldDoc distance (:152-156), and the queue is ordered by it. Geo3DPointOutsideDistanceComparator has the same shape (:75-104, :107-109, :129-131, :133-152).

Latitudes {1}, {2}, {9, 0.5}, {5}, {6} degrees at longitude 0, one segment, top 2 by distance from (0, 0). Docs 0 and 1 fill the queue (one read each), so the bottom is doc 1 at 2°. Doc 2: compareBottom reads 9° and 0.5°, and 0.5° beats 2°. copy reads two more values, doc 3's 5° and doc 4's 6°, and stores 5°. Docs 3 and 4 do not beat 5°. Result: doc 0 at 1.00°, doc 2 at 5.00°, instead of doc 2 at 0.50°, doc 0 at 1.00°. The same on the unit sphere and on WGS84 (111,194 m and 555,976 m), with assertions on and off. Lucene's AssertingDirectoryReader fails inside copy with valueUpto=2 in.docValueCount()=2.

• Who is exposed. Users of the Geo3D doc-values sorts whose documents can hold several points, for example a store with several branches or a route with several stops, as the javadoc allows. A read-only code search found no use of this API in Elasticsearch, OpenSearch or Solr.
• What it takes. A segment with at least one document that has two or more points (then single-valued hits in that segment are re-read too), and either numHits below the number of hits or searchAfter. Those are necessary, not sufficient: a wrong read needs a hit that gets two or more value-reading calls, so a top-N search in which no hit after the first numHits is competitive stays correct (read from the code, not executed). Default codec, IndexWriterConfig, directory and searcher; deterministic; assertions on or off make no difference.
• What you see. While the extra reads stay inside the field's values: wrong order and a wrong FieldDoc distance, with no error. Only when they pass the end of the field's values does the request fail, on a healthy index: IndexOutOfBoundsException on 9.0.0; on 9.12.3 and later RuntimeException(EOFException: read past EOF ...) on file directories and IndexOutOfBoundsException on a plain heap directory (the versions in between were not run); 7.x and 8.x return a wrong value instead.
• Which value is wrong depends on storage order. The second read takes the next docValueCount() values in storage order, from the following document or documents, whether or not they match: in an executed probe a permitted hit reported the exact distance of a document hidden by a FILTER clause, by a live-docs reader wrapper or by a delete (0.2530 instead of 0.01745 on the unit sphere), and three searches with different sort origins recovered the hidden document's location to within 2.55e-05 degrees. This needs an application that returns FieldDoc sort values; only the distance and outside-distance sorts, filters, hard deletes and a live-docs wrapper were executed. The hit set itself is not widened.
• How likely. Plausible: in a segment with a multi-valued document, a top-N request re-reads whenever a hit after the first numHits is competitive, and a searchAfter page re-reads every hit it accepts, deterministically; but the API is little used and no public report was found.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<doc [2 at 55599.27446323976 m, doc 0 at 111193.82953747343] m> but was:<doc [0 at 111193.82953747343 m, doc 2 at 555975.6320987428] m>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.spatial3d.TestGeo3DDistanceSortMultiValued.testTopTwoIsHeadOfFullSort(TestGeo3DDistanceSortMultiValued.java:58)

• The javadoc promises it. "Multiple values for the same field in one document is allowed" (Geo3DDocValuesField.java:34-35); every factory says the closest value is used and that FieldDoc holds the distance (:167-174 and siblings). The returned distance belongs to another document. Neither the field nor the comparators are @lucene.experimental; PlanetModel, a parameter of every sort factory, is (PlanetModel.java:23-27), which bears on API stability and priority, not on whether a wrong documented result is intended.
• The read rule is explicit (SortedNumericDocValues.java:28-31), and the test framework enforces it: AssertingDirectoryReader fails with valueUpto=2 in.docValueCount()=2 at computeMinimumDistance:171 <- copy:138 <- TopFieldCollector...collectCompetitiveHit:116 (executed).
• The double question is Lucene's own and documented. LeafFieldComparator: compareBottom "Compare a new hit (docID) against the 'weakest' (bottom) entry", copy "Installs a new hit into the priority queue ... when a new hit is competitive" (LeafFieldComparator.java:32-37); TopFieldCollector does exactly that.
• The twins handle the same pattern. LatLonPointDistanceComparator (:68-69, :120-133, :192) and XYPointDistanceComparator (:63-64, :106-119, :176) read each document's values once into currentValues and reset per leaf.
• History shows an oversight. 6.x read with setDocument(doc) + valueAt(i), which can be repeated; 6.1.0-6.6.6 are correct (executed). LUCENE-7407 (f7aa200d406) replaced it with nextValue() in both Geo3D comparators and added the cache to the LatLon twin only; adding just that cache to 7.0.0 and 7.7.3 makes them correct (executed).
• Known pattern. When the iterator API was designed, LUCENE-7462 (#8514) made advanceExact accept the current document because a strict version "caused issues with comparators that may need to get the value multiple times".
• No statement of intent. No javadoc, CHANGES, MIGRATE, test or issue limits Geo3D sorts to single-valued fields; upstream has no Geo3D sort test at all.

The intended-behaviour checker built the strongest case: (a) FieldComparator and LeafFieldComparator are Expert and @lucene.experimental (so is PlanetModel, a parameter of every Geo3D sort factory), and nothing says in so many words that copy follows compareBottom for the same document, so a comparator could read once per call; (b) the feature looks unfinished: LUCENE-7212 (#8267) is still open with "tests" listed as missing, no upstream test sorts with Geo3D, the class javadoc still has a "TBD" list, and during design a committer asked how necessary multi-valued support really is; (c) nextValue() puts the burden on the caller, and the codec is deliberately unchecked, so a wrong value or an EOFException is the allowed outcome of breaking the rule.

Why it fails: the caller that breaks the rule is Lucene's own comparator, driven by Lucene's own collector in the order the LeafFieldComparator javadoc describes, and the two core twins cache values for exactly that order; missing tests do not make wrong results intended, multi-valued support is documented on a public field, and 6.x implemented it correctly; the codec skips the check because callers must respect docValueCount. Verdict: a defect (the adjudication: confirmed, high). Expect "low-traffic module" pushback on priority, not on validity.

One fix was built and validated: the per-document value cache of the 2D twins. Two alternatives were not built.

In Geo3DPointDistanceComparator and Geo3DPointOutsideDistanceComparator: add currentValues/valuesDocID and a setValues() that reads the current document's values once; call it in compareBottom and computeMinimumDistance and read from the array; reset valuesDocID = -1 in getLeafComparator. A copy of LatLonPointDistanceComparator.java:68-69, :120-133, :192, including its ordering assert (left out of the snippet). 2 files, +54/-4.

private long[] currentValues = new long[4];
private int valuesDocID = -1;

private void setValues() throws IOException {
  if (valuesDocID != currentDocs.docID()) {
    valuesDocID = currentDocs.docID();
    int count = currentDocs.docValueCount();
    if (count > currentValues.length) {
      currentValues = new long[ArrayUtil.oversize(count, Long.BYTES)];
    }
    for (int i = 0; i < count; i++) currentValues[i] = currentDocs.nextValue();
  }
}
// compareBottom, computeMinimumDistance: setValues(); read currentValues[i]; getLeafComparator: valuesDocID = -1;

Trade-offs

One long[] per comparator (4 longs at first; it grows to the largest value count seen and is kept for the comparator's life, across segments) and a doc-id check per call. Doc values are read once per document instead of two or three times, but compareBottom, compareTop and copy still each decode the points and compute distances. Same design as the core twins, so easy to review (in 2016 the LatLon comparator lived in lucene/sandbox). A helper shared by both comparators, or caching each document's minimum distance (keeping compareBottom's bounding-box shortcut), are alternatives. No benchmark was run.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: stock + the new test fails 6 of 7 with assertions on and off; + fix the new test passes 7 of 7 with assertions on and off.
• The whole lucene/spatial3d module: 20 suites, 188 tests, 3 skipped, 0 failures, with assertions on and off; :lucene:spatial3d:check -x test (ecjLint, forbiddenApis, checkGoogleJavaFormat) passes.
• Standalone runner with the fix: the independent reproducer's 10 tests (including the AssertingDirectoryReader probe), the original 5 tests and the hidden-document probe's 7 tests all pass.
• The fix's code comment was reworded after the fact checks (code unchanged); every run above was repeated with the reworded fix.
• Earlier: the same cache added to 7.0.0 and 7.7.3 fixed the history probe there.

In computeMinimumDistance, used by copy and compareTop, call currentDocs.advanceExact(doc) instead of advancing only when the cursor is behind; DocValuesIterator allows the current document as the target (DocValuesIterator.java:25-28).

Trade-offs

Smaller diff, but it relies on advanceExact of the same document resetting the value cursor: the Lucene90 iterators and AssertingLeafReader do, no javadoc says so. It decodes and computes distances two or three times per competitive hit.

Say in the javadoc that only one point per document is supported.

Trade-offs

Contradicts the field's own javadoc, 6.x behaviour and the 2D twins, and leaves the exception path in place.

Method: code reading, witnessed by a randomized metamorphic harness and a JUnit (the hunt's non-formal tier).

• The hunt unit checked metamorphic relations of SortField paging across modules: the top N must be the head of the full order, and searchAfter pages must concatenate to it.
• While writing the harness's reference for the Geo3D comparators, the finder saw that compareBottom and copy both loop over nextValue() for the same document.
• On the stock main jar the harness's Geo3D family gave 2,811 paging mismatches and 2,267 IndexOutOfBoundsExceptions in 3,000 random trials, counted together with a separate Geo3D searchAfter units bug (LFV-544); 0 with both fixes, while with only LFV-544's fix 1,070 mismatches and 853 exceptions remained.
• The finder reduced it to a 5-test public-API JUnit. A review by another model family (OpenAI GPT-5.6, read-only) widened the trigger to single-valued hits in a multi-valued segment, which a new executed test confirmed.