Doc-values exact and range queries miss or add hits when the index is sorted by the same raw-bit float field as FLOAT or DOUBLE

SortedNumericDocValuesRangeQuery, the query behind NumericDocValuesField.newSlowExactQuery and newSlowRangeQuery, has a shortcut for a single-valued field with a skip index that is also the index's primary sort field: it treats the matching documents as one run of consecutive doc IDs and finds the run's ends from the skip index. That is right only if the documents are in the order of their stored long values. The guard, canUsePrimarySortShortcut, checks the sort field's name, density and document blocks but never the sort type. A SortField.Type.FLOAT or DOUBLE index sort orders a field of raw IEEE bits (the FloatDocValuesField / DoubleDocValuesField encoding) by the decoded number, and for negative numbers that is the reverse of the order of the raw bits. With two documents, -1f and -2f, the exact query for -1f counts 0 instead of 1 and the one for -2f counts 2; a reverse sort over-matches and a MUST_NOT clause removes documents it should keep. No error, default codec, assertions on or off, every release from 10.0.0 to 10.6.0 and main. The sibling IndexSortSortedNumericDocValuesRangeQuery has allowed its index-sort optimisation only for INT and LONG since LUCENE-10466 fixed the same kind of type assumption there as a bug (2022; there it surfaced as a ClassCastException); the shortcut was modelled on that class but did not take over the restriction.

1. For a single-valued field with a skip index, scorerSupplier asks canUsePrimarySortShortcut whether it may use SortedSkipperScorerSupplier (SortedNumericDocValuesRangeQuery.java:141-147). The guard's javadoc: "Returns the primary sort field if it is safe to use the SortedSkipperScorerSupplier range shortcuts" (:229-231).
2. The guard returns null only if there is no primary sort field or it has another name, the field is not dense or the segment has document blocks (:237-253). It never reads SortField#getType.
3. SortedSkipperScorerSupplier then computes one doc-ID interval: it starts at the first document whose stored value is >= the lower bound and ends at the first one whose stored value is > the upper bound (mirrored for a reverse sort; SortedSkipperScorerSupplier.java:113-182). That is right only if the stored longs never decrease along the doc order.
4. IndexSorter's FLOAT sorter orders by Float.intBitsToFloat((int) value) and its DOUBLE sorter by Double.longBitsToDouble(value) (IndexSorter.java:382-385, :446-449). With two distinct negative values in a segment, the stored longs go down along the doc order.

After the FLOAT sort: doc 0 = -2f (raw -1,073,741,824), doc 1 = -1f (raw -1,082,130,432); the skip index reports min -1,082,130,432 and max -1,073,741,824. For the exact query [raw(-1f), raw(-1f)]:

• start: the segment minimum is not below the lower bound, so the run starts at doc 0 (SortedSkipperScorerSupplier.java:165-167);
• end: the segment maximum is above the upper bound, so it scans from doc 0 for the first value > raw(-1f), and doc 0's -1,073,741,824 already is (:173-180, :128);
• the run [0, 0) is empty (:83-89 in get(); count goes through bulkScorer(), :93-99, with the same check), so the count is 0. Doc 1, which holds raw(-1f), lies outside it. For raw(-2f) the same code gives [0, 2): both documents.

FloatField / DoubleField (sortable bits in SORTED_NUMERIC, sorted with SortedNumericSortField), LONG sorts, INT sorts over int-range values, fields without a skip index, sparse fields, segments with document blocks, and segments whose values are not NaN and include at most one distinct sign-bit-set value (one finite negative value, or -0.0 alone, still sorts in long order). NaN bit patterns are outside that rule: the FLOAT sort puts NaN after every other value, so a NaN with the sign bit set breaks it. With raw bits {0xFFC00000 (a NaN), raw(1f)}, the exact query for raw(1f) counts 2 and the one for 0xFFC00000 counts 0 instead of 1 each (executed on main; the repro uses finite values only). An INT sort over longs outside the int range reaches the same code, but storing such values under an INT sort is arguably out of contract. Since #16424 (10.6.0), Sort.getPrimarySortField also skips a leading sort field that has no values in the segment, or that is dense and whose skip index shows a single value, so FLOAT as the second sort field after such a field reaches the shortcut too; 10.5.2 is correct for that layout only.

• Who is hit. A single-valued NUMERIC doc-values field that holds raw IEEE bits (FloatDocValuesField / DoubleDocValuesField's encoding) with a skip index, built with NumericDocValuesField.indexedField(name, Float.floatToRawIntBits(v)) (FloatDocValuesField has no skip-index constructor of its own); an index sort on that field with SortField.Type.FLOAT or DOUBLE; every document of the segment with a value; and a doc-values exact or range query on the field (NumericDocValuesField.newSlowExactQuery / newSlowRangeQuery, or SortedNumericDocValuesField.newSlowRangeQuery). All public API, accepted by IndexWriter.
• The documented use. FieldType#setDocValuesSkipIndexType says a skip index "is typically useful on fields that are part of the index sort". The unusual part is the raw-bit float encoding together with that sort.
• What they see. Wrong count, wrong hits and wrong filters: matching documents missed, non-matching ones added (reverse sort, or the query for the more negative value), and documents removed by a MUST_NOT clause that should stay. No exception; the same with assertions off. Two documents with distinct negative values in one segment are enough.
• How likely. Rare (registry): few applications combine a hand-built raw-bit float field, a skip index, a FLOAT/DOUBLE index sort on it, negative values and a doc-values filter on the same field. For those that do, it is deterministic.

The test stores the floats with NumericDocValuesField.indexedField("f", Float.floatToRawIntBits(v)), which is FloatDocValuesField's encoding plus a skip index (FloatDocValuesField.indexedField is the same inherited method). It pins the default codec and has no explicit random input (newDirectory() is the test framework's randomized directory; the test failed under both seeds tried); one segment, single thread. Controls run by a verifier on the same two documents give 1 without the index sort, 1 without the skip index and 1 with a LONG sort.

TestDocValuesRangeQueryFloatIndexSort > testExactQueryOnFloatIndexSort FAILED
    java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:8D27622AAB57F4F9]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.document.TestDocValuesRangeQueryFloatIndexSort.testExactQueryOnFloatIndexSort(TestDocValuesRangeQueryFloatIndexSort.java:44)

# the same with -Ptests.asserts=false; with the recommended fix the test passes

• Contract. newSlowRangeQuery: "Create a range query that matches all documents whose value is between lowerValue and upperValue included" (NumericDocValuesField.java:91-92); newSlowExactQuery: "Create a query for matching an exact long value" (:132). Exact matching does not depend on any order.
• The guard promises safety. "Returns the primary sort field if it is safe to use the SortedSkipperScorerSupplier range shortcuts for this query" (SortedNumericDocValuesRangeQuery.java:229-231).
• The sibling learned it already. IndexSortSortedNumericDocValuesRangeQuery.java:208, :538: "The index sort optimization is only supported for Type.INT and Type.LONG". That came from LUCENE-10466 (#11502 / #749, Andriy Redko), a bug fix: "Ensure IndexSortSortedNumericDocValuesRangeQuery handles sort field types besides LONG" (lucene/CHANGES.txt:4286-4287). #13592's PR body (iverase): "Similar to what we do in IndexSortSortedNumericDocValuesRangeQuery for primary sort, we can do the same in ... SortedNumericDocValuesRangeQuery".
• Upstream tests. TestSortOptimization#testFloatSortOptimizationSkpperIndex indexes FloatDocValuesField.indexedField(field, Float.floatToRawIntBits(value)) and sorts searches by SortField.Type.FLOAT (TestSortOptimization.java:474-477, :492); TestIndexSorting#testBasicFloat and #testBasicDouble sort the index by FLOAT / DOUBLE over FloatDocValuesField / DoubleDocValuesField (TestIndexSorting.java:1465-1472, :1168-1175). These tests cover the two halves separately; none combines a skip index, a FLOAT/DOUBLE index sort and a doc-values range query, and every upstream test of the shortcut uses a LONG or STRING sort.
• Maintainers know the hazard. On #16573 HoustonPutman wrote that "NUMERIC expects floats and doubles to not be in sorted order"; on #16629 romseygeek wrote "I think we need to check that that [sic] we aren't making this assumption in other places as well", naming the sibling query. Nobody named this shortcut, and the discussions of #13592, #16085, #16424 and #16893 never mention the sort type.

The strongest case for "not supported": SortField.Type.FLOAT's javadoc says "Fields must either be not indexed, or indexed with FloatPoint" (SortField.java:82-84), and a skip index could be read as "indexed".

Why it fails: the only index structure the javadoc names for FLOAT is FloatPoint, and the class javadoc's general "Fields must be indexed in order to sort by them" (SortField.java:39-40) is followed by a paragraph about points ("Sorting on a numeric field that is indexed with both doc values and points may use an optimization", :42-44); the repro has no points; raw bits are exactly the encoding the FLOAT index sorter decodes (IndexSorter.java:382); upstream itself sorts a skip-indexed raw-bit field as FLOAT; and the exact-match contract does not depend on the sort at all. A second objection, that raw-bit range queries over negative floats are meaningless, does not touch the exact query. The independent checks found no surviving intended-behaviour argument; the cross-model reviews rated it "weak". The INT-over-out-of-range-longs variant is arguably out of contract ("Sort values are Integer"), so it is a note, not the claim.

Option A was applied in a scratch worktree of 645d4fe14a20 and run with gradle (the worktree was removed after the runs). Options B and C were not built.

+25 lines in SortedNumericDocValuesRangeQuery.java: after the existing checks, canUsePrimarySortShortcut returns null unless sortFollowsLongOrder(sf, skipper).

private static boolean sortFollowsLongOrder(SortField sf, DocValuesSkipper skipper) {
  boolean sortable = sf instanceof SortedNumericSortField;
  SortField.Type type = sortable ? ((SortedNumericSortField) sf).getNumericType() : sf.getType();
  boolean intRange =
      skipper.minValue() >= Integer.MIN_VALUE && skipper.maxValue() <= Integer.MAX_VALUE;
  return switch (type) {
    case LONG -> true;
    case INT -> intRange;
    // SortedNumericSortField decodes the sortable encodings of FloatField and DoubleField
    case FLOAT -> sortable && intRange;
    case DOUBLE -> sortable;
    default -> false;
  };
}

Trade-offs

• Keeps the shortcut for LONG, for INT on int-range segments and for sortable float/double index sorts; drops it for every layout where it can be wrong (all plain SortField FLOAT/DOUBLE sorts over raw bits, including segments where the shortcut happens to be right today, and INT segments outside the int range), where the query falls back to the skipper-driven iterator it already uses without an index sort (SortedNumericDocValuesRangeQuery.java:149-157).
• More logic than option B, and it must stay in step with how the sorters decode values. No benchmark was run; the format check (checkGoogleJavaFormat) passes on the diff.

What the checks validated

• Gradle, stock 645d4fe14a20: the repro fails with assertions on and off.
• Gradle with the fix: the repro plus all of org.apache.lucene.document.*, TestDocValuesQueries, TestGetPrimarySortField, TestIndexSortSortedNumericDocValuesRangeQuery, TestSortOptimization and TestIndexSorting: 671 tests, 0 failures (seed DEADBEEF, assertions on). With assertions off, the repro, TestDocValuesQueries and TestSortedSkipperScorerSupplier pass too (47 tests).
• The fact check, in its own scratch worktree: the repro plus six of these suites (178 tests) pass with the fix, the repro passes with assertions off, and the format check passes.
• Earlier hunt runs: the standalone runner passes the repro with the fix; 7 same-fix tests (DOUBLE, reverse, MUST_NOT, INT over long, constant first sort field) fail 6 of 7 stock and 0 of 7 with it; a bounded-exhaustive check over a three-value alphabet per configuration (every layout of up to 7 documents over those values), 40 sort/encoding/layout configurations, 1,023,048 checks, finds 0 mismatches with it.

Return null unless the (unwrapped) sort type is LONG or INT, as IndexSortSortedNumericDocValuesRangeQuery does.

Trade-offs

Simplest and consistent with the sibling class; loses the shortcut for sortable float/double index sorts, which are correct today. Add option A's int-range check to also cover INT over out-of-range longs. A LONG-only probe made an independent test pass.

Translate the long bounds into the decoded order.

Trade-offs

A raw-bit long range can cover a positive and a negative float range, so the matches are not one doc-ID run in general; it needs two intervals and more code for a rare layout.

Method: code reading during Z3 modelling, confirmed by a bounded-exhaustive check (the hunt's semi-formal tier). Found first in a hunt unit (doc-values range queries, 2026-10-25); the same root was found independently by three later units, all collapsed into this bug.

• While writing a Z3 model of SortedSkipperScorerSupplier, the AI tool noticed that the model's premise (the stored longs never decrease along the doc order) is not checked by the guard. Z3 then characterised the premise per sort type; among the documented sort/encoding pairings it holds for LONG, for INT over int-range values and for SortedNumericSortField FLOAT/DOUBLE over sortable bits, and fails for plain FLOAT/DOUBLE over raw bits. Its differential harness hit the mismatch at random (269 of 300 runs).
• The adjudication unit re-ran a wider Z3 order model and a bounded-exhaustive check on the stock jars: every index of up to 7 documents over a 3-value alphabet, 10 sort/encoding configurations, both directions and two sort layouts, 1,023,048 checks. All 40 configurations behave as the model predicts (16 unsafe ones fail, 24 safe ones are clean), and 0 mismatches remain with the fix.
• It then reduced the finding to the 2-document public-API JUnit. So this is a code-reading find that the formal model and the exhaustive check confirmed, not a blind discovery by a tool.