JoinUtil's numeric join reads a DoubleField or FloatField from field as raw IEEE bits, so a join on a negative value silently misses its matches

JoinUtil.createJoinQuery(fromField, multipleValuesPerDocument, toField, numericType, ...), the point-based numeric join, collects the from documents' doc values and looks them up as points on the to field. Its javadoc lists floats and doubles and allows SORTED_NUMERIC from fields, and names no encoding. Core's DoubleField and FloatField (since 9.5.0) write SORTED_NUMERIC doc values in the NumericUtils sortable layout, but the join decodes the collected longs as raw IEEE bits (Double.longBitsToDouble, Float.intBitsToFloat). The two layouts agree exactly when the sign bit is clear, so positive keys work and every key with the sign bit set is looked up as a different value: -5.0 is looked up as -0.8749999999999999, so the repro's self-join silently returns nothing; with two or more distinct negative normal keys (e.g. -5.0 and -2.0) it throws values are out of order (the related LFV-217 symptom). multipleValuesPerDocument=false throws for these fields, so no setting avoids it. Every release from 9.5.0 to 10.6.0 and main are affected. Decoding by the from field's doc-values type (SORTED_NUMERIC sortable, NUMERIC raw) fixes it, keeps raw-bit NUMERIC fields working and also fixes LFV-217 for NUMERIC fields and for DoubleField/FloatField.

1. What the field writes. DoubleField sets DocValuesType.SORTED_NUMERIC (document/DoubleField.java:52) and stores NumericUtils.doubleToSortableLong(value) (:73, and :105 in the setter); FloatField the same with floatToSortableInt (FloatField.java:52, :73, :105).
2. What the join collects. With multipleValuesPerDocument true (JoinUtil.java:228) the collector reads DocValues.getSortedNumeric (:250) and adds each nextValue() unchanged (:239-240). The single-valued branch reads DocValues.getNumeric (:297), which throws unexpected docvalues type SORTED_NUMERIC for these fields.
3. The sort. The collected longs are sorted as signed longs (:316-318), then streamed into a point-set query that needs ascending points.
4. The decode (fix site). Float stream: FloatPoint.encodeDimension(Float.intBitsToFloat((int) value.value), ...) (:368-369); Double stream: DoublePoint.encodeDimension(Double.longBitsToDouble(value.value), ...) (:387). That is the inverse of what DoubleDocValuesField / FloatDocValuesField write (raw bits, DoubleDocValuesField.java:39, FloatDocValuesField.java:40), not of what DoubleField / FloatField write.

-5.0 has raw bits 0xc014000000000000 and sortable bits 0xbfebffffffffffff; read as raw bits, 0xbfebffffffffffff is -0.8749999999999999, the point the join looks up. An independent test with -2.5 prints the query as join:{-1.7499999999999998}. A Z3 bit-vector model shows that the raw decode of the sortable layout gives back the original value exactly when the sign bit is clear, at 64 and 32 bits.

• One distinct negative from value: 0 hits, no error (DoubleField and FloatField).
• Two or more distinct negative normal from values (e.g. -5.0 and -2.0): IllegalArgumentException: values are out of order (the LFV-217 symptom, reached through this decode). -0.0 and the negative subnormal -4.9E-324 are looked up as NaN instead: with -5.0 they give 0 hits and no exception.
• Positive values, Integer and Long joins, and raw-bit fields (DoubleDocValuesField, a hand-written raw-bit SortedNumericDocValuesField) with one negative value: correct.

Wrong extra matches are possible but need a to document whose point equals the mis-decoded number: only this doc's fact check ran it (a from DoubleField -5.0 joins to a to DoublePoint -0.8749999999999999, count 1; 0 with the fix). Not part of this finding: the facet module's MultiDoubleValuesSource.fromDoubleField / fromFloatField and misc's DocValuesStats.SortedDoubleDocValuesStats, which read these fields the same way (other modules and fix sites).

• Who is hit. Callers of the numeric join with Double.class or Float.class whose from field is a DoubleField or FloatField, with negative join keys (signed scores, coordinates, temperatures, offsets).
• Default vs opt-in. Default codec, IndexWriterConfig and searcher; one document suffices. The API is opt-in: the lucene/join module, @lucene.experimental (JoinUtil.java:54), and Float/Double join keys are uncommon.
• No setting avoids it. For these fields multipleValuesPerDocument=true is the only value that does not throw.
• Silent, and hidden by positive data. One distinct negative key: 0 hits, no exception. Positive keys behave, so tests with positive data, like upstream TestJoinUtil, never see it. Two or more distinct negative normal keys (e.g. -5.0 and -2.0): an IllegalArgumentException instead.
• Downstream (read, not run). Solr 10.1.0 (released 2026-10-27; SOLR-18440) routes multi-valued float/double score joins to this method, and Solr writes those doc values in the sortable layout.
• Likelihood: plausible. Deterministic on the first negative key once the setup is used; the setup itself is rare.

The test is the hunt's minimal repro, unchanged; it passes :lucene:join:check -x test (google-java-format, ecj lint, forbidden APIs). The from and to sides are the same field of the same index, which keeps the test to one document; the hunt's and an independent verifier's two-index tests fail the same way and are evidence only.

Executed: gradle on main e1faeba08bde, assertions on (seed DEADBEEF) and off (seed 5EED1111): fails 1 of 1; with the fix it passes. The hunt's standalone runs of this test fail the same way on main with assertions on and off (seed DEADBEEF); the registry re-runs (seeds 5EED0E34, 5EED0D35) ran the finder's two-index test and fail too. newDirectory() and newIndexWriterConfig() randomize only the directory and writer settings; the mis-decode does not depend on them.

Use the same test as the regression test.

cp $REPRO/TestJoinNegativeDoubleField.java $LUCENE/lucene/join/src/test/org/apache/lucene/search/join/
cd $LUCENE && ./gradlew :lucene:join:test --tests TestJoinNegativeDoubleField -Ptests.seed=DEADBEEF -Ptests.asserts=true
# also with -Ptests.asserts=false; this test failed at both seeds run (DEADBEEF, 5EED1111)

TestJoinNegativeDoubleField > testJoinOnNegativeDoubleField FAILED
    java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:9B27BB8BCEE348AB]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:647)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.search.join.TestJoinNegativeDoubleField.testJoinOnNegativeDoubleField(TestJoinNegativeDoubleField.java:45)
1 test completed, 1 failed

• Contract. "It supports multi- and single- values longs, ints, floats and longs" and "when true fromField might be SORTED_NUMERIC" (JoinUtil.java:122-130); the to field is a FloatPoint/DoublePoint (:131-134). No encoding is named in JoinUtil, the join package docs, CHANGES or MIGRATE.
• Core's recipe. "Note that if you want to encode doubles or floats with proper sort order, you will need to encode them with NumericUtils" (SortedNumericDocValuesField.java:33-37).
• Other readers. SortedNumericSelector ("undo the numericutils sortability", :74-89, used by SortedNumericSortField) and the join module's own ToParentBlockJoinSortField (:319-320, :358-359) decode SORTED_NUMERIC floats/doubles as sortable. That holds for core and the join module; two readers in other modules (facet, misc) decode raw bits, see the devil's advocate.
• The join's own sort. JoinUtil.java:316-318 sorts the collected values as signed longs for a point-set query that needs ascending points: numeric order only for sortable bits.
• Tests. TestJoinUtil writes multi-valued FLOAT/DOUBLE link fields as raw bits (:1920-1928), but generates only non-negative values ("positive nums makes it easier", :1690-1696). It passes 16 of 16 with either decode. No join test uses DoubleField or FloatField.
• History. The raw decode is from LUCENE-7095 (2016, 6.0.0). DoubleField/FloatField came with #11997 (9.5.0), which did not touch JoinUtil; there jpountz wrote that multi-valued floats and doubles "should use NumericUtil#doubleToSortableLong", and the fields were made SORTED_NUMERIC-only so queries "do not have to specify whether floats/doubles are indexed via their raw bits or as sortable ints/longs".

The strongest "intended" case (built by the devil's-advocate verifier and the cross-model reviews): LUCENE-7095 paired the raw decode with raw-writing tests; DoubleDocValuesField/FloatDocValuesField write raw bits, and the multi-valued flag also reads NUMERIC fields ("single val can be handled by multiple-vals", TestJoinUtil.java:1535), so one raw convention for both paths; the sortable recipe is conditional ("if you want ... proper sort order") and SORTED_NUMERIC has no encoding metadata; the facet module's MultiDoubleValuesSource.fromDoubleField and misc's DocValuesStats make the same raw assumption; JoinUtil is @lucene.experimental, and no public report of it was found in the ~3.8 years since 9.5.0. A maintainer could answer "document that the from field must hold raw bits".

Why it fails: nothing in JoinUtil says raw bits or excludes DoubleField/FloatField; core's recipe and every SORTED_NUMERIC float/double reader in core and in the join module are sortable, and #11997 made the new fields sortable-only on purpose. A raw-bit convention covering negatives never worked: with raw bits two distinct negatives already throw (LFV-217), and its only test uses non-negative values. The only accepted flag for core's standard double field silently drops every negative key. And a compatible fix exists: the DocValuesType-keyed decode keeps raw-bit NUMERIC fields working. Verdict: not intended, a defect; the residual risk that maintainers prefer documenting a raw-bit precondition is why priority stays Low.

One fix was built and validated in a scratch worktree of e1faeba08bde; the alternatives are listed with their trade-offs. LFV-217 is a sibling: the recommended fix fixes this bug and LFV-217's recorded repro (a NUMERIC FloatDocValuesField), while LFV-217's own proposed fix, applied alone, would leave this bug in place. Neither fixes raw bits hand-written into SORTED_NUMERIC (see the trade-offs).

In JoinUtil.createJoinQuery's numeric path (+26/-2): per segment, the multi-valued collector checks the from field's DocValuesType; NUMERIC values (raw bits) are converted with NumericUtils.sortableFloatBits / sortableDoubleBits, SORTED_NUMERIC values are kept. The single-valued collector (always NUMERIC) converts Float/Double values the same way. The streams decode with sortableIntToFloat / sortableLongToDouble. Every collected value is then sortable, so the existing signed-long sort is numeric order (which also fixes LFV-217 for NUMERIC fields and for DoubleField/FloatField, not for raw bits hand-written into SORTED_NUMERIC). Integer and Long joins are untouched. The full patch is in the brief.

// multi-valued collector, per segment:
FieldInfo fi = context.reader().getFieldInfos().fieldInfo(fromField);
rawBits = (floatJoin || doubleJoin) && fi != null
    && fi.getDocValuesType() == DocValuesType.NUMERIC;
// per collected value (unconditionally for Float/Double in the single-valued collector):
if (rawBits) value = floatJoin ? NumericUtils.sortableFloatBits((int) value)
                               : NumericUtils.sortableDoubleBits(value);
// Float and Double streams:
FloatPoint.encodeDimension(NumericUtils.sortableIntToFloat((int) value.value), encoded.bytes, 0);
DoublePoint.encodeDimension(NumericUtils.sortableLongToDouble(value.value), encoded.bytes, 0);

Trade-offs

• Behaviour change. DocValuesType is not an encoding tag: the fix assumes NUMERIC holds raw bits and SORTED_NUMERIC sortable bits, the conventional mapping. An application that hand-wrote raw IEEE bits of negative values into a SortedNumericDocValuesField (against that class's sortable recipe) gets 0 hits instead of 1 for a single such key, and a silent 0 instead of today's values are out of order exception for two or more distinct such keys (both executed). Raw-bit NUMERIC fields keep working, single- and multi-valued. Needs a CHANGES/MIGRATE note, or an explicit encoding parameter if results must not change; backporting is a maintainer call.
• TestJoinUtil's multi-valued FLOAT/DOUBLE link fields still write raw bits; they pass because the values are non-negative. A maintainer may want them switched to the sortable layout.
• Cost: one FieldInfo lookup per segment and a few bit operations per collected Float/Double value. No benchmark was run.

What the checks validated

• Gradle, JDK 25, scratch worktree of e1faeba08bde (removed afterwards): the test fails on stock with assertions on (DEADBEEF) and off (5EED1111), and passes with the fix, on and off.
• With the fix: the test plus TestJoinUtil, 17 tests, 0 failures; the whole :lucene:join:test, 105 tests in 17 classes, 3 skipped, 0 failures; :lucene:join:check -x test passes.
• The hunt, with the same logic: the two-index repro, a DoubleDocValuesField probe with multipleValuesPerDocument=true and -5.0 (1 hit, as on stock), LFV-217's repro and upstream TestJoinUtil 16 of 16 all pass.
• Not shown: compatibility for arbitrary raw-bit SORTED_NUMERIC data (the existing tests use only non-negative raw bits); the behaviour change above is the known difference.

The hunt's first sketch: switch only the multi-valued Float/Double decode to sortableIntToFloat / sortableLongToDouble.

Trade-offs

Not recommended: the flag does not say which encoding is stored. A DoubleDocValuesField (NUMERIC, raw bits) joined with multipleValuesPerDocument=true and -5.0 goes from 1 hit to 0, and so does a raw-bit SORTED_NUMERIC -6.25 control (both executed).

State that the numeric join's from field must hold raw IEEE bits and that DoubleField/FloatField are not supported (possibly rejecting them).

Trade-offs

No behaviour change, but core's standard double and float fields could then not be joined through Double.class/Float.class at all, against core's own SORTED_NUMERIC recipe. Not validated.

A new overload that takes the from-side encoding, for a release line where results must not change (suggested by a cross-model reviewer and the finder).

Trade-offs

No change for existing callers; more API surface. Not built.

Method: non-formal (an executed differential probe; the Z3 bit-vector model only proved the trigger set afterwards).

• Round 22 of the hunt planned heterogeneous-schema probes with the property "one logical type stored with different doc-values encodings must decode to the same value on every consumer". Its own probe (2026-11-02, main e1faeba08bde, assertions on and off, also 10.6.0) joined from DoubleField, FloatField and a LongField control: negative keys gave 0, 0 and 1 hits, positive keys 1, 1 and 1.
• The round's skeptic re-ran it. The hunt unit that judged it reduced it to a public-API JUnit, modelled the trigger set in Z3 (the raw decode of the sortable layout is the identity exactly when the sign bit is clear) and checked its independence from LFV-217 both ways; a history step bisected it to #11997.
• An independent reproducer, cross-model reviews (OpenAI GPT-5.6), a devil's-advocate verifier, an adjudication and an audit then re-ran it, and a last check reduced the test to one document.
• An earlier sighting (2026-10-26, during LFV-217's verification: one negative FloatField key gave 0) was noted but not followed up then.