A FLOAT/DOUBLE sort on a skip-indexed NUMERIC field can drop the best hits when values are negative

Since 10.3.0 (#14672), a numeric sort on a field that has a doc-values skip index and no points prunes with the skip index: once more hits than the hits threshold (default 1,000) have been counted, NumericComparator turns the current bottom (and the searchAfter top) into a bound and skips each skip-index block (nominally 4,096 docs at level 0) whose min/max cannot beat it. For SortField.Type.FLOAT / DOUBLE, FloatComparator / DoubleComparator give that bound in sortable bits (NumericUtils.floatToSortableInt / doubleToSortableLong), but on a NUMERIC field the skip index records the min/max of the stored longs, which are raw bits (Float.floatToRawIntBits): the only NUMERIC encoding these comparators decode, the encoding FloatDocValuesField writes, and the layout of Lucene's own skip-index float sort test. The two encodings agree for non-negative numbers (+0.0 to +Infinity) and the canonical NaN, run in opposite directions for values whose sign bit is set, and also differ for NaNs with a non-canonical payload, because the bounds canonicalize NaN. So once a segment holds negative values (or -0.0 with a tie-break sort), blocks that hold the best hits can be skipped; whether a search goes wrong depends on the magnitudes of the block's values and of the bound. In a 10,000-doc index where doc i holds -i, the ascending top 3 is -8191, -8190, -8189 instead of -9999, -9998, -9997, a descending searchAfter(-9000) page is empty, and there is no error. Lucene's FloatField / DoubleField (SORTED_NUMERIC, sortable bits), fields with points and fields without a skip index are not affected. Every release from 10.3.0 to 10.6.0 (10.6.0 by reading) and main is affected; 10.0.0 to 10.2.2 are not. No upstream report was found.

• Raw bits: Float.floatToRawIntBits / Double.doubleToRawLongBits, what FloatDocValuesField / DoubleDocValuesField store in a NUMERIC doc-values field, and what FloatComparator / DoubleComparator decode (Float.intBitsToFloat).
• Sortable bits: NumericUtils.floatToSortableInt / doubleToSortableLong, the encoding of FloatPoint, FloatField and the comparators' pruning bounds. It first turns any NaN into the canonical NaN (Float.floatToIntBits), then keeps the bits when the sign bit is clear and flips all other bits when it is set, so signed-integer order equals numeric order.
• Skip index: per block of docs (nominally 4,096 at level 0, plus coarser levels), the min and max of the stored longs (NumericDocValuesField.indexedField, or FieldType.setDocValuesSkipIndexType(RANGE)).

1. NumericComparator.buildCompetitiveDISIBuilder uses points if the field has them, otherwise the doc-values skipper (NumericComparator.java:117-131), whatever the sort type or doc-values type.
2. Once more hits than the threshold (default 1,000) have been counted (until then updateCompetitiveIterator returns early, :242-244, also with a searchAfter top, which only adds a second bound), encodeBottom / encodeTop (:277-314) take the bounds from bottomAsComparableLong() / topAsComparableLong(), which for floats are NumericUtils.floatToSortableInt(bottom) (FloatComparator.java:110-117; doubles the same, DoubleComparator.java:111, :116).
3. new SkipBlockRangeIterator(skipper, minValueAsLong, maxValueAsLong) (NumericComparator.java:530) hands those sortable bounds to DocValuesSkipper.advance(min, max), which skips a block when minValue(0) > maxValue || maxValue(0) < minValue (DocValuesSkipper.java:125): raw block bounds against sortable bounds.
4. The same comparator reads each document's value as raw bits (FloatComparator.java:81). For segments without sign-bit values the encodings coincide, which is why Lucene's test with values 0..N-1 passes. With negative values, blocks holding the best hits can look non-competitive. A sign-bit value is needed but not enough: for floats, an ascending sort skips an all-negative block when the bit patterns of the block's smallest magnitude and of the bound's magnitude add up to more than 2^31 - 2 (roughly, both 2 or more); a descending sort skips one only when the magnitudes are small or the block holds only -0.0. On the repro's layout one -5 among positives, values in [-1, -0.0001] and the first descending page over -i were all correct; -i/1000 ascending was wrong (fact-check runs). -0.0 is the smallest raw int but sortable -1, so with a tie-break sort the -0.0 documents that should win the tie are skipped.

Repro, one segment of 10,000 docs, doc i = -i (doc 0 holds -0.0), TermQuery matching all, query cache off. Blocks: docs 0-4095 raw min -2,147,483,648 (-0.0), max -981,471,232 (-4095); docs 4096-8191 raw min -981,467,136 (-4096), max -973,080,576 (-8191); docs 8192-9999 raw min -973,078,528 (-8192), max -971,228,160 (-9999). The TermQuery runs on DenseConjunctionBulkScorer, which collects docs 0-8191 as one range and then asks the competitive iterator for the next doc. By then the bound is max = -1,174,345,730 (sortable bits of -8,163.0, the bottom at the last sampled refresh, minus 1); the last block's raw min -973,078,528 is greater, so advance(8192) returns NO_MORE_DOCS. The same index with doc i = +i also collects exactly 8,192 docs and is right, so the 8,192 is not part of the bug: it ends the term's first dense postings run (one level-1 postings block, 32 x 256 docs), which the bulk scorer collects as one range. On 10.3.x that block is 32 x 128 = 4,096 docs and the top 3 is -4095, -4094, -4093.

A NaN whose sign bit is set (Float.intBitsToFloat(0xFFC00000)) is also missed: Float.compare sorts it above every number, but its raw bits are negative. With docs 0-8191 holding 1000..9191, docs 8192-9998 holding 0..1806 and doc 9999 holding that NaN, the descending top 3 is docs 8191, 8190, 8189 instead of docs 9999, 8191, 8190. Two layouts whose skip index has no negative minimum fail too: every doc holding the positive-sign NaN 0x7f800001 (or the double 0x7ff0000000000001), descending with a reverse doc-id tie-break (the bound is the canonical NaN 0x7fc00000), and floats stored zero-extended (Integer.toUnsignedLong(Float.floatToRawIntBits(v))), ascending; the canonical NaN passes.

• Who is hit. A FLOAT/DOUBLE sort on a single-valued NUMERIC doc-values field that holds raw bits, with a skip index (NumericDocValuesField.indexedField(name, Float.floatToRawIntBits(v)), the double equivalent, or FieldType.setDocValuesSkipIndexType(RANGE)) and no points. It is opt-in, but it is the layout of Lucene's own TestSortOptimization#testFloatSortOptimizationSkpperIndex, and FieldType#setDocValuesSkipIndexType says a skip index is "typically useful on fields that are part of the index sort".
• Trigger. A value whose raw and sortable bits differ in the segment (a negative value; -0.0 together with a tie-break sort; a NaN with the sign bit set; rarer, a NaN with another payload and a tie-break sort), and pruning that engages: more matches than the hits threshold (default 1,000; a searchAfter alone does not start it), with a query whose scorer consults the competitive iterator. Even then not every search is wrong: it depends on the magnitudes of the values in each block and of the bound (see What goes wrong). Reproduced with a TermQuery (also with the default query cache), MUST + FILTER conjunctions, 3 and 5 segments, an index sorted on the field, and random unclustered values (ascending top 5 wrong in 20 of 20 mixed-sign random indices). A dense single-segment MatchAllDocsQuery and a pure SHOULD disjunction did not prune in the probes and stayed correct. Descending without searchAfter was correct on random data.
• What they see. Deterministic, silent wrong or missing top-N hits: ascending pages miss the smallest values, descending searchAfter pages can be empty, -0.0 ties come out in the wrong order. Every returned hit still matches the query and carries its own value. Default codec, IndexWriterConfig, searcher and hits threshold; -ea and -da the same.
• Not hit. FloatField / DoubleField and other SORTED_NUMERIC sortable fields with SortedNumericSortField (0 wrong results in 119,460 randomized checks), fields with points, fields without a skip index. Elasticsearch, OpenSearch and Solr (default branches read on 2026-10-26, not run) do not build the affected layout; custom applications are unknown.
• How likely. Registry likelihood "plausible": the layout is hand-built, but on it ordinary negative data and an ordinary sorted query over more than 1,000 hits fail deterministically at about 10,000 docs (the repro, the independent reproduction's random data, the intended-behaviour check's natural data); some value ranges stay correct by luck. Priority medium.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[-9999.0, -9998.0, -9997.0]> but was:<[-8191.0, -8190.0, -8189.0]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.TestFloatSortSkipIndexNegative.testAscendingFloatSortWithNegativeValues(TestFloatSortSkipIndexNegative.java:57)

• Contract. SortField.Type.FLOAT: "Sort using term values as encoded Floats. Sort values are Float and lower values are at the front. Fields must either be not indexed, or indexed with FloatPoint." (SortField.java:82-86; DOUBLE :94-98). The field has no points. FloatComparator: "Comparator based on Float#compare ... This comparator provides a skipping functionality – an iterator that can skip over non-competitive documents." (FloatComparator.java:26-29).
• Upstream test. #14672 added testFloatSortOptimizationSkpperIndex with FloatDocValuesField.indexedField(field, Float.floatToRawIntBits(value)) (TestSortOptimization.java:474-478); its helper checks that a searcher that "hides index structures to force a linear scan" returns "the same hits" (:1499-1520). It indexes only 1f * i, i >= 0 (:484-487), where raw and sortable bits are equal. There is no double variant.
• Self-contradiction. FloatComparator decodes the NUMERIC values with Float.intBitsToFloat (:81) but compares the skip index built from those values with sortable bounds (:110-117, NumericComparator.java:530).
• CHANGES 10.3.0: "Numeric sort queries can take advantage of DocValuesSkipper to dynamic prune." (CHANGES.txt:1582), with no type or sign restriction; 10.2.2 returns the right hits.
• Precedent. In #16573 / #16629 (fixed in 10.5.2) maintainers treated the same raw-vs-sortable mismatch in range-query code as a bug: on #16573 HoustonPutman wrote that "NUMERIC expects floats and doubles to not be in sorted order", and on #16629 romseygeek, about points min/max not matching the actual double values, wrote "I think we need to check that that [sic] we aren't making this assumption in other places as well", naming IndexSortSortedNumericDocValuesRangeQuery. Earlier competitive-iterator false negatives were fixed as bugs (LUCENE-10106, LUCENE-10126).

The intended-behaviour checker built the strongest case: the FLOAT javadoc says "not indexed, or indexed with FloatPoint", and a skip index is "indexed"; Lucene has no first-class float field with a NUMERIC skip index, so the user owns the encoding; TestSegmentReordering stores sortable bits in a NUMERIC skip-indexed field and sorts it with Type.FLOAT, which could be read as the convention; and SortField.setOptimizeSortWithIndexedData(false) turns pruning off.

Why it fails: the same PR's own test uses raw bits plus a skip index and asserts pruned hits equal a full scan, and, although "indexed" read literally is ambiguous (a maintainer may raise it), the class javadoc's "indexed with both doc values and points" (SortField.java:42-44) shows the FloatPoint sentence is the points precondition. Raw bits are the only NUMERIC encoding FloatComparator decodes correctly: sortable bits in a NUMERIC field give wrong sort values for negatives even without a skip index (executed control), so TestSegmentReordering only works because its values are non-negative. The off switch is deprecated, "should only be used for compatibility with 8.x indices that got created with inconsistent data across fields" (SortField.java:608-629). Nothing in javadoc, CHANGES, MIGRATE, tests or the PR reviews limits skipper pruning to non-negative values. Verdict: not intended, a defect. The cross-model reviewer with the same lens rated the intended argument weak.

Options A and B were implemented and validated with gradle in a scratch worktree at ea57360912ed (removed after the runs), together with a randomized differential test written for the check (random FLOAT and DOUBLE indices of 2,000-20,000 docs over 1-4 segments, with negative, -0.0, infinite and NaN values, random direction, tie-break, page size and searchAfter; stock main fails it 3 of 3 on 5 seeds; its DOUBLE values come from floats, and it has no positive-sign non-canonical NaN and no zero-extended values). The fact checks then found that both key on a negative skipper minimum, which misses two rarer encodings; an edge-encoding test (stock fails 3 of 4) and option A+, which closes that gap, were added. Option C was not written. Until a fix lands, users can turn pruning off with the deprecated SortField.setOptimizeSortWithIndexedData(false) or an exact total-hit count, or change the field: FloatField / DoubleField (points plus sortable SORTED_NUMERIC doc values, no skip index) sorted with SortedNumericSortField prune through points and are correct; to keep a doc-values skip index without points, store sortable bits with SortedNumericDocValuesField.indexedField(name, NumericUtils.floatToSortableInt(v)) and sort with SortedNumericSortField (executed: correct and pruned). Open PR #16163 touches the same builder but keeps the comparison (read, not run); today pruning waits for the hits threshold even with searchAfter, and #16163 would change that.

Raw and sortable bits are equal exactly for the raw range from +0.0 to +Infinity (plus the canonical NaN). Option A+ keeps option A and also gives up when the segment's raw maximum is above +Infinity's bits, which catches every NaN payload and zero-extended float bits. The change is +19 lines in buildCompetitiveDISIBuilder (option A's lines plus the ones below).

// raw bits from +0.0 to +Infinity equal their sortable bits; also give up when the segment
// holds a NaN (its payload may differ from the canonical NaN of the bounds) or float bits
// stored zero-extended
long positiveInfinityBits =
    NumericComparator.this instanceof FloatComparator
        ? Float.floatToRawIntBits(Float.POSITIVE_INFINITY)
        : Double.doubleToRawLongBits(Double.POSITIVE_INFINITY);
if (rawIeeeBits && (skipper.minValue() < 0 || skipper.maxValue() > positiveInfinityBits)) {
  return null;
}

Trade-offs

• Everything option A gives up, and also skip-index pruning on segments that hold any NaN, including the canonical NaN, where stock pruning was right.
• The same style points as option A. No benchmark was run.

What the checks validated

• The repro runner: the edge-encoding test 4 of 4 (-ea, -da; stock, A and B fail 3 of 4), the finder's repro 6 of 6 (-ea, -da), the differential 3 of 3 on 5 seeds, the independent reproduction's test (seed 1A2B3C4D) 6 of 6, the intended-behaviour check's (seed 7E57AB1E) 5 of 5.
• Gradle in a fresh extraction of ea57360912ed: the 163-test set above plus the edge test, 167 tests, 0 failures (seed DEADBEEF, assertions on; seed CAFEBABE, assertions off); tidy only joined the if onto one line. These runs used an earlier wording of the code comment; the final file was re-run with the repro runner.
• Not run: the whole search package, other modules.

The hunt's sketch. For the sign-extended raw bits FloatDocValuesField / DoubleDocValuesField write, a segment whose skipper minimum (DocValuesSkipper.minValue(), segment-wide) is non-negative has no value with the sign bit set, and there raw and sortable bits are identical, except NaNs with a non-canonical payload. It keys on the doc-values type, because SORTED_NUMERIC float/double values are sortable-encoded and already correct; a negative-sign NaN also makes the minimum negative. Gap (executed): a positive-sign non-canonical NaN with a tie-break sort, and floats stored zero-extended, keep a non-negative minimum, so option A still fails 3 of the 4 edge-encoding tests; option A+ closes it. The code below is the diff's text.

DocValuesSkipper skipper = reader.getDocValuesSkipper(field);
if (skipper != null) {
  // FLOAT/DOUBLE NUMERIC doc values hold raw IEEE bits (FloatDocValuesField /
  // DoubleDocValuesField), whose signed-long order differs from the sortable encoding used
  // for the bottom/top bounds as soon as a value has its sign bit set.
  FieldInfo fi = reader.getFieldInfos().fieldInfo(field);
  boolean rawIeeeBits =
      (NumericComparator.this instanceof FloatComparator
              || NumericComparator.this instanceof DoubleComparator)
          && fi != null
          && fi.getDocValuesType() == org.apache.lucene.index.DocValuesType.NUMERIC;
  if (rawIeeeBits && skipper.minValue() < 0) {
    return null;
  }
  return new DVSkipperCompetitiveDISIBuilder(skipper, this);
}

Trade-offs

• Gives up skip-index pruning on every segment that holds a negative value, -0.0 or a negative-sign NaN, the same as 10.2.x, which never pruned with the skipper; non-negative segments keep it. On a mixed-sign segment (-i for the first 8,192 docs, +i after) it visits all 10,000 docs where option B visits 8,192, both correct.
• The instanceof test and the fully qualified DocValuesType are style points a maintainer may want changed (a hook on the comparator, an import). No benchmark was run.

What the checks validated

• Gradle, stock: the repro fails 6 of 6 with assertions on and off.
• Gradle with the fix: the repro, the differential test and 12 upstream classes (TestSortOptimization, TestSort, TestSearchAfter, TestTopFieldCollector, TestTopFieldCollectorEarlyTermination, TestDenseConjunctionBulkScorer, TestBooleanQuery, TestSortedNumericSortField, TestSortRandom, TestSkipBlockRangeIterator, TestElevationComparator, comparators.TestNumericComparator): 163 tests, 0 failures (seed DEADBEEF, assertions on; seed CAFEBABE, assertions off).
• Gradle with the fix: all of org.apache.lucene.search.* in :lucene:core with the repro, 2,336 tests, 0 failures, 49 skipped; tidy leaves the fix unchanged.
• The repro runner: the finder's, the independent reproduction's, the cross-model review's and the reviewer's tests pass; the bounded-exhaustive harness and a randomized differential (2,113,290 checks) find 0 mismatches; the differential test passes on 5 seeds; the security probe's hit lists no longer depend on hidden documents.
• Not run: the rest of :lucene:core, other modules.

+63/-4 in NumericComparator.java. buildCompetitiveDISIBuilder tells the skipper builder whether the field holds raw float (32) or double (64) bits (a FLOAT/DOUBLE sort on a NUMERIC field). When it does and skipper.minValue() < 0, doUpdateCompetitiveIterator replaces the sortable range [lo, hi] by the smallest raw-bit range that holds the raw bits of every value in it, plus every NaN bit pattern when the range reaches above +Infinity.

if (lo < 0) { // sign bit set: [lo, min(hi, -1)] in reverse order
  min = Math.min(hi, -1) ^ highest; // highest = Integer.MAX_VALUE or Long.MAX_VALUE
  max = lo ^ highest;
}
if (hi >= 0) { // sign bit clear: the same bits
  min = Math.min(min, Math.max(lo, 0));
  max = Math.max(max, hi);
}
if (hi > positiveInfinity) { // NaNs may compete: every NaN bit pattern, either sign
  min = Math.min(min, (positiveInfinity + 1) | lowest);
  max = highest;
}

Trade-offs

• Can keep pruning for ascending sorts on segments with negative values: the mixed-sign probe visits 8,192 docs instead of 10,000 (the pruned block is the positive one); the all-negative probe visits all 10,000, because the answers are in the last block; pruning of negative blocks follows from the mapping but was not measured. Non-negative segments are unchanged.
• Still gives up pruning when the competitive range covers both signs, for example a descending sort whose bottom is negative, because one raw range then spans nearly everything; an exact version needs a two-range block test.
• More code than option A, and it must stay in step with NumericUtils. A first variant that mapped on every raw-bit segment failed upstream TestNumericComparator.testEmptyCompetitiveIteratorOptimization (the NaN clause stopped a descending sort on non-negative values from pruning); limiting it to skipper.minValue() < 0 fixed that.
• The same gap as option A: it activates only when skipper.minValue() < 0, so it still fails 3 of the 4 edge-encoding tests; closing it needs option A+'s maximum check (not written for B).

What the checks validated

• Gradle with the fix: the same 163 tests, 0 failures (seed DEADBEEF, assertions on; seed CAFEBABE, assertions off); all of org.apache.lucene.search.* with the repro and the differential, 2,339 tests, 0 failures, 49 skipped; tidy clean.
• The repro runner: the differential passes on 5 seeds and with --da; the finder's repro (-ea, -da), the independent reproduction's test (with its pruning-engaged control) and the intended-behaviour check's test pass; the security probe shows 0 wrong searches and hit lists that do not depend on hidden documents.
• Not run: the rest of :lucene:core, other modules, a benchmark.

Teach SkipBlockRangeIterator (or a sibling) to accept the union of the two raw-bit intervals a sortable range maps to, so no pruning is lost.

Trade-offs

Most code; touches a public class; untested.

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier), a hunt unit (sorting / TopFieldCollector), 2026-10-25, main ea57360912ed.

• The finder wrote a harness that runs the real TopFieldCollector on the stock jar against an independent brute-force sort for every small index layout (tiny codec with a skip interval of 2, at most 5 docs, several field encodings and sort types).
• While building it, the finder noticed by code reading that the skipper bounds are sortable-encoded while the skipper's block min/max are raw bits. The harness then witnessed it: for NUMERIC + skip index + FLOAT with the values (-3, -2), 77,738 of 2,597,184 checks were wrong; for DOUBLE with (-0.0, 0.0), 30,090 of 4,328,640. With the fix sketch both gave 0.
• A default-codec JUnit with 10,000 docs on the stock main jar confirmed it (6 of 6 failing). A randomized differential at scale (up to 30,000 docs, 1-4 segments, deletions, index sorts) found 268 wrong results in 11,489 checks on the affected family and 0 in 119,460 on the sortable SORTED_NUMERIC family.
• So it is a code-reading find confirmed by the exhaustive check, not a blind discovery by the tool.