JoinUtil's numeric join on Float or Double throws 'values are out of order' when the from side has two or more distinct negative values

JoinUtil.createJoinQuery(fromField, multipleValuesPerDocument, toField, numericType, fromQuery, fromSearcher, scoreMode) is documented for int, long, float and double keys. For Float.class and Double.class it collects the from-side doc values, which are raw IEEE 754 bits when the from side uses FloatDocValuesField or DoubleDocValuesField (the bits JoinUtil decodes), sorts them as signed longs and streams them in that order into PointInSetQuery (ScoreMode.None) or PointInSetIncludingScoreQuery (scoring modes). Both constructors require the points in ascending order: PointInSetQuery drops equal neighbours, PointInSetIncludingScoreQuery rejects them. IEEE 754 is sign-magnitude, so for negative numbers the signed order of the bits is the reverse of the numeric order: -1 sorts before -2. As soon as the from query matches documents holding two different negative keys (or -0.0 and a negative key), createJoinQuery throws IllegalArgumentException: values are out of order instead of returning a query, in every ScoreMode, for single-valued from fields and for multi-valued ones that store raw bits (as upstream TestJoinUtil does; SortedNumericDocValuesField's javadoc recommends the sortable encoding instead), with assertions on or off, with default settings and at any index size. One negative key, positive keys and Integer/Long joins work, and with raw-bit from values no join returns wrong results. The bug arrived with the numeric point join in 6.0.0 (LUCENE-7095); releases before 10.4.0 show an UnsupportedOperationException raised while the same error message is built. Collecting the keys in their sortable form fixes it.

• Numeric query-time join: JoinUtil.createJoinQuery(fromField, multipleValuesPerDocument, toField, numericType, fromQuery, fromSearcher, scoreMode) runs fromQuery, collects the join keys of the matching documents from the doc values of fromField, and returns a query that matches the documents whose point field toField holds one of those keys. numericType is Integer, Long, Float or Double.
• Raw bits: FloatDocValuesField stores a float as Float.floatToRawIntBits(value) and DoubleDocValuesField a double as Double.doubleToRawLongBits(value): the IEEE 754 pattern, a sign bit followed by the magnitude.
• Point encoding: FloatPoint and DoublePoint encode a value as bytes that sort in numeric order (NumericUtils.floatToSortableInt, which flips the magnitude bits of negative numbers).

1. Collect. The collectors add each from-side doc value unchanged to a LongHashSet (JoinUtil.java:151, :239-240 multi-valued, :287-289 single-valued). For Float and Double that is the raw bit pattern.
2. Sort. After fromSearcher.search(fromQuery, collector) (:314) the values are sorted as signed longs: Arrays.sort(joinValuesList.buffer, 0, joinValuesList.size()) (:316-318).
3. Stream. The Float stream decodes each value with Float.intBitsToFloat and encodes it with FloatPoint.encodeDimension (:360-378); the Double stream does the same with Double.longBitsToDouble and DoublePoint.encodeDimension (:379-396).
4. Build. The stream goes into PointInSetIncludingScoreQuery for the scoring modes (:406) or PointInSetQuery for ScoreMode.None (:415). PointInSetQuery says "The packedPoints iterator must be in sorted order" (PointInSetQuery.java:74) and throws values are out of order otherwise (:116-122); PointInSetIncludingScoreQuery has the same check and also rejects duplicates (PointInSetIncludingScoreQuery.java:124-133). Both are plain if/throw, so they fire with assertions off too.

The signed sort gives -0.0, -1, -2, +0.0, 3; the point order is -2, -1, -0.0, +0.0, 3. Every non-NaN key with the sign bit set comes out reversed, so any two distinct ones throw. A single negative key is fine, because the negative block still comes before the non-negative block. Doubles behave the same with 64 bits. Integer and Long are two's complement, where signed order is numeric order. JoinUtil builds and sorts the stream itself; with the standard from-side encodings (raw bits, or the sortable encoding) the caller cannot make it ascend.

Edge cases found while validating the fix. A NaN whose sign bit is set (raw 0xFFC00000) sorts after every negative number, but FloatPoint encodes every NaN as the canonical NaN, the largest point; next to the key 1 it throws values are out of order: saw [ff c0 0 0] before [bf 80 0 0] (ScoreMode.None and Max). Separately, two different NaN bit patterns (0x7FC00000, 0x7FC00001) next to 1 make the scoring path throw unexpected duplicated value: [ff c0 0 0]. By reading only: a sign-bit NaN next to -1 or -0.0 does not throw (it is last in both orders), and two different sign-bit NaN patterns become one point, which ScoreMode.None drops and the scoring modes reject as a duplicate. All are rare inputs, and the recommended fix covers them.

• Who is hit. Applications that use the numeric JoinUtil.createJoinQuery overload (lucene-join module, @lucene.experimental) with Float.class or Double.class. Choosing that API is the only opt-in part: default codec and IndexWriterConfig, any index size (two from documents suffice), every ScoreMode, single-valued from fields and multi-valued ones that store raw bits.
• When. Whenever the from query matches documents holding at least two distinct non-NaN keys with the sign bit set. That depends on the data: a join that works today starts failing the day a second negative key is indexed. It is deterministic.
• What happens. createJoinQuery throws before any Query exists, so the request fails. With raw-bit from values: no wrong or partial results, no index damage. Releases 6.0.0 to 10.3.x throw an UnsupportedOperationException with no message, which is harder to diagnose.
• Downstream (code reading, not run): Elasticsearch and OpenSearch call only the global-ordinals JoinUtil variant. Solr's score join ({!join score=...}) calls the numeric variant with Float.class/Double.class since SOLR-18440 (on Solr main and branch_10x since September 2026, not in a Solr release yet), and Solr's single-valued float/double point doc values store the IEEE bit layout (Float.floatToIntBits / Double.doubleToLongBits, the same as raw bits for every non-NaN value), so such a Solr join whose from side matches two distinct negative values should hit this error once that code ships.
• How common. Float and double join keys are uncommon (in the original review they were described as allowed "strangely"), and no public report was found.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected a join on from-values {-1, -2}, but threw java.lang.IllegalArgumentException: values are out of order: saw [40 7f ff ff] before [3f ff ff ff]
       org.apache.lucene.search.join.TestNumericJoinNegativeFloats.testFloatJoinOnTwoNegativeValues(TestNumericJoinNegativeFloats.java:53)
     Caused by: java.lang.IllegalArgumentException: values are out of order: saw [40 7f ff ff] before [3f ff ff ff]
       org.apache.lucene.search.PointInSetQuery.<init>(PointInSetQuery.java:120)
       org.apache.lucene.search.join.JoinUtil$8.<init>(JoinUtil.java:415)
       org.apache.lucene.search.join.JoinUtil.createJoinQuery(JoinUtil.java:415)
       org.apache.lucene.search.join.TestNumericJoinNegativeFloats.testFloatJoinOnTwoNegativeValues(TestNumericJoinNegativeFloats.java:49)

• The contract. The method javadoc (JoinUtil.java:122-123): "Method for query time joining for numeric fields. It supports multi- and single- values longs, ints, floats and longs." toField "should be IntPoint, LongPoint, FloatPoint or DoublePoint" (:131-132); numericType is "either java.lang.Integer, java.lang.Long, java.lang.Float or java.lang.Double" (:133-134). Nothing restricts the sign.
• Deliberate support. In the LUCENE-7095 review (apache/lucene#8151, migrated from JIRA) @mikemccand wrote: "The javadocs state that only int and long are supported, yet float and double (strangely, but matching what the legacy numerics join supports?) are also allowed?" @martijnvg replied that they had updated the javadocs, which now list floats.
• The precondition binds JoinUtil. "The packedPoints iterator must be in sorted order" (PointInSetQuery.java:74) is a precondition on the code that builds the stream. The exception reports an internal invariant, not anything the user passed.
• Why tests never caught it. TestJoinUtil generates only non-negative link values: "positive nums makes it easier" (TestJoinUtil.java:1691), random.nextInt(0xFFFFFF) (:1696). The non-negative generator dates from LUCENE-5868 (bc51c86b4bb, 2015), an int/long-only join, before float keys existed; LUCENE-7571 (09b0eee5bbc, 2017) later capped it at 0xFFFFFF to avoid float precision loss and kept it non-negative.

The strongest case for "intended": the JoinUtil javadoc does not say how float keys must be stored on the from side, and SortedNumericDocValuesField's javadoc says "if you want to encode doubles or floats with proper sort order, you will need to encode them with NumericUtils" (SortedNumericDocValuesField.java:33-37, example NumericUtils.floatToSortableInt(-5.3f)); FloatField and DoubleField store that sortable form. So raw bits may be the wrong encoding, at least for multi-valued fields, and the sortable one may have been meant.

Why it fails: JoinUtil decodes with Float.intBitsToFloat (JoinUtil.java:369), which reads raw bits; FloatDocValuesField writes raw bits; the introducing commit's own tests write raw bits, single- and multi-valued (TestJoinUtil.java:1924, :1932). And the sortable encoding fails too: two negative sortable keys throw the same exception, and a single one silently matches nothing (count 0 instead of 1). Neither standard encoding joins two negative floats (by reading only, a contrived custom encoding with the sortable key in the high 32 bits and the raw float bits in the low 32 bits would, because the Float decode reads only the low 32 bits; no Lucene field writes that). The other arguments fail as well: @lucene.experimental (JoinUtil.java:54) allows the API to change, not documented inputs to fail, and the checked input is the stream JoinUtil built, so this is not input validation. Every reviewer, including a cross-model review and a devil's-advocate check with a works-as-intended lens, rated the intended case weak; the cross-model fact check of this doc agrees for the single-valued fields and rates it moderate for a multi-valued raw-bit field, because of the SortedNumericDocValuesField javadoc: there the API is inconsistent whichever encoding is preferred, so the fix review should settle the supported encoding. Verdict: a defect.

Three ways to order the keys correctly; the recommended one was built and validated through gradle, the hunt's comparator variant through the standalone runner. Whichever is chosen, add negative float and double keys, including -0.0, to the join tests.

In createJoinQuery, map each collected Float or Double doc value to the key the point encoding sorts by (NumericUtils.floatToSortableInt(Float.intBitsToFloat((int) v)), NumericUtils.doubleToSortableLong(Double.longBitsToDouble(v))) before it enters the key set and the score maps. The existing primitive Arrays.sort then gives point order, and the streams decode with NumericUtils.sortableIntToFloat / sortableLongToDouble. Integer and Long are unchanged. 1 file, +19/-4.

final LongUnaryOperator toJoinKey;
if (Float.class.equals(numericType)) {
  toJoinKey = v -> NumericUtils.floatToSortableInt(Float.intBitsToFloat((int) v));
} else if (Double.class.equals(numericType)) {
  toJoinKey = v -> NumericUtils.doubleToSortableLong(Double.longBitsToDouble(v));
} else {
  toJoinKey = v -> v;
}
// both collectors: long value = toJoinKey.applyAsLong(<doc value>);
// Float stream:  FloatPoint.encodeDimension(NumericUtils.sortableIntToFloat((int) value.value), ...)
// Double stream: DoublePoint.encodeDimension(NumericUtils.sortableLongToDouble(value.value), ...)

Trade-offs

A few bit operations (through a LongUnaryOperator) per collected value, no per-value allocation or boxing, and the sort stays primitive. Because the key is exactly what FloatPoint/DoublePoint sort by, NaN keys behave like the point fields: every NaN bit pattern becomes the canonical NaN, so the NaN edge cases stop throwing, and the scores of different NaN bit patterns are aggregated together. It does not change which from-side encodings work, and for a sortable-encoded from side it turns the stock exception for two negative keys into a silent zero match (see the separate observation below), so settle the encoding question in the same review. No benchmark or allocation profile was run.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: stock + the repro fails 4 of 4 with assertions on and off; + fix it passes with assertions on and off. The wider test: stock 6 of 10 fail, fix 10 of 10 pass. The NaN probe: stock 3 of 4 fail, fix 4 of 4 pass.
• With the fix: upstream TestJoinUtil 16 of 16; the whole lucene/join test task 19 classes, 122 tests, 3 skipped, 0 failures (the 16 existing classes: 104 tests); :lucene:join:check -x test (format, ecjLint, forbiddenApis) passes with the new tests in place.
• Standalone runner with the fix: the finder's repro (assertions on and off), the independent reproducer's test and the cross-model reviewers' tests pass; the only remaining failures are the sortable-encoding cases of the separate observation, as with the hunt's comparator sketch.
• fix.diff and fix-with-test.diff apply cleanly to a9779ef3c193. Not run: the full Lucene test suite, nightly mode and any benchmark.

Separate observation, not part of this bug. If the from side uses the sortable encoding (FloatField/DoubleField, or Solr's multi-valued float/double point fields), JoinUtil's raw-bit decode turns negative keys into different numbers and the join silently misses them: one negative FloatField key gives count 0 instead of 1 on stock main, and with this fix two negative FloatField keys give count 0 instead of 2 where stock throws. Which from-side encodings JoinUtil supports is undocumented, and Lucene points two ways: FloatDocValuesField/DoubleDocValuesField and TestJoinUtil use raw bits, while SortedNumericDocValuesField's javadoc (:33-37) tells callers to store floats and doubles in the sortable encoding. Ask the fix review to settle it (document raw bits as required, or support or reject the sortable encoding).

Before Arrays.sort, replace each value by NumericUtils.sortableFloatBits((int) v) / sortableDoubleBits(v) (each is its own inverse), sort, and map back.

Trade-offs

The smallest diff, at one site. But it orders by the raw bits, not by the canonical key the encoder uses, so a NaN with the sign bit set still sorts first, encodes last and throws (reasoning from the stock NaN probe's encoding, not run); mapping a float back is exact only for int bits widened to a long, which is what FloatDocValuesField writes.

After the primitive sort, box the values into a Long[], sort it with Comparator.comparingLong on floatToSortableInt(intBitsToFloat(...)) / doubleToSortableLong(longBitsToDouble(...)), and copy back.

Trade-offs

Correct order, and the collected values stay raw, but it boxes every distinct join value. By reasoning (not run), two different NaN bit patterns still give duplicate points, which the scoring path rejects.

What the checks validated

Through the standalone runner on main ea57360912ed: the finder's repro 4 of 4, the independent reproducer's test 10 of 10 and a cross-model reviewer's test 5 of 5 pass; upstream TestJoinUtil passes 16 of 16.

Method: code reading (the hunt's non-formal tier), by the hunt unit for the join and block-join module, on main ea57360912ed (2026-10-26).

• The unit traced the numeric JoinUtil pipeline: how the from-side values are collected, how they are sorted, and the order PointInSetQuery / PointInSetIncludingScoreQuery require. It noticed that Float/Double values are raw IEEE bits sorted as signed longs.
• It wrote a public-API JUnit (two negative keys, Float and Double, with and without scores), which failed 4 of 4 on main with assertions on and off and on 10.5.2 and 9.12.3, and a fix sketch that made it pass.
• The unit's formal and bounded-exhaustive tools (a block-join reference harness and SMT queries on ScoreMode semantics) do not cover this path, so they did not find it.