Sort.getPrimarySortField skips an index-sort field that still orders the documents, so range queries on the next sort field return wrong hits

Sort.getPrimarySortField(LeafReader), added in 10.6 by #16424, tells three doc-values range queries which field the segment is effectively sorted by: it skips leading index-sort fields that "would have no effect" in the segment, so a query on the second sort field can binary-search instead of testing every document. It decides "no effect" from the reader it is given (the field is missing from getFieldInfos(), or its skipper holds one value), but the documents' order belongs to the segment. When the first sort field is a BinarySortField whose key comes from another doc-values field, or a FilterLeafReader hides it while keeping the index sort, the method skips a field that orders every document, and the range queries on the next sort field cut a doc-ID range out of documents that are not in that order. In a 5-document index f:[1 TO 3] returns 0 hits instead of 2 and LongField.newRangeQuery throws IllegalArgumentException; over 55 ranges of an 8-document index 43 are wrong (46 for LongField). It affects main and 10.6.0, with assertions on and off; 10.5.2 and older are correct. Reading the evidence from the unwrapped segment and skipping only the built-in sort-field classes fixes every repro, and the neighbouring core suites pass.

• Index sort: IndexWriterConfig.setIndexSort(new Sort(c, f)) stores each segment's documents ordered by c, then by f among equal c. The reader reports it in getMetaData().sort(), "the order in which documents from this index are sorted" (LeafMetaData.java:32).
• Index-sort range shortcut: NumericDocValuesField, SortedNumericDocValuesField and SortedDocValuesField.newSlowRangeQuery on a skip-indexed field, and LongField / IntField.newRangeQuery, check whether the segment's primary sort is the queried field. If it is, and the query's other guards hold (for the doc-values queries a single-valued field whose skip index covers every document; for LongField / IntField an INT or LONG sort and a one-dimensional points index, among others), the matches form one doc-ID range and the query finds its two ends instead of testing every document.
• Sort.getPrimarySortField(reader) (new in 10.6, #16424): the primary sort those queries ask for. It skips leading sort fields that "would have no effect" in the segment (Sort.java:141-143), so a query on the second sort field can use the shortcut when the first one is empty or constant there.

1. It decides "no effect" from the reader: the field is missing from reader.getFieldInfos() ("Field has no values in this segment", Sort.java:156-159), or its skipper is dense with minValue() == maxValue() (:164-169).
2. The order is a property of the segment, and a reader can omit the first sort field while the segment is still ordered by it:
   • a BinarySortField subclass derives its key from another doc-values field, as its javadoc allows (BinarySortField.java:41-44); no field of its own name exists;
   • a FilterLeafReader hides the field and, by default, still delegates getMetaData() (FilterLeafReader.java:477-481).
3. getPrimarySortField returns f, and the range query on f (SortedNumericDocValuesRangeQuery.java:237-247, SortedSetDocValuesRangeQuery.java:288-298, IndexSortSortedNumericDocValuesRangeQuery.java:198, 425, 523) cuts a doc-ID range out of a segment that is not ordered by f.

Index sorted by (c, f), five documents, after forceMerge(1):

f:[1 TO 3] matches docs 2 and 4. Assuming f ascends, the shortcut starts the range at doc 0 (no f is below 1) and ends it before the first document from there with f > 3, which is doc 0 itself (f = 5): the empty range [0, 0), 0 hits (SortedSkipperScorerSupplier.java:113-182). LongField.newRangeQuery finds the two ends through the points index instead, gets start 2 and end 0, and DocIdSetIterator.range(2, 0) throws IllegalArgumentException: minDoc must be < maxDoc.

A bounded-exhaustive check of the stock jar (every layout of up to 5 documents over (c, f) in {0,1} x {1,2,3}, every range on f, a LONG NumericDocValuesField with a skip index, c hidden by the test framework's FieldFilterLeafReader) finds 0 of 55,980 checks wrong on the plain reader and 22,120 wrong, in 5,964 of 9,330 layouts, through a reader that hides c; 10.5.2 has 0.

• Who. An index sort of at least two fields (opt-in), doc-values range or exact queries on the second sort field (newSlowRangeQuery of NumericDocValuesField, SortedNumericDocValuesField and single-valued SortedDocValuesField / SortedSetDocValuesField with a skip index, or LongField / IntField.newRangeQuery without one), and one of:
  • a BinarySortField subclass whose key is derived from other doc-values fields (@lucene.experimental, added in 10.5.0 by #16202, whose javadoc invites such keys);
  • a reader wrapper that hides the first sort field and keeps the index sort: FilterLeafReader's default, the test framework's FieldFilterLeafReader ("exposes only a subset of fields"), and field-level-security views built like it. One widely used downstream field-level-security reader says it is based on FieldFilterLeafReader, filters FieldInfos and keeps getMetaData(); whether its queries reach these shortcuts was not checked;
  • weakest: a BinarySortField subclass on a skip-indexed NUMERIC field that is constant in the segment, keyed on (that field, another field).
• Effect. Wrong hits and counts, with no error, for many ranges on the second sort field (43 of 55 in an 8-document test; 22,120 of 55,980 checks, about 40%, in the bounded check): missing matches and sometimes documents outside the range (in a 66-range probe, 16 ranges on the skip-index path and 20 on the LongField path returned extra hits; f:[2 TO 9] also returned the document with f = 1). LongField.newRangeQuery (and, by reading, IntField's) can throw IllegalArgumentException instead. Five documents are enough.
• What still holds. In the security probe's fixture no deleted document was returned and no other FILTER clause of the same query was violated; nothing is written, and a fixed jar gives the right answers on the same index.
• Released in 10.6.0. Main and 10.6.0 (released 2026-10-29); 10.5.2 and the older releases tested are correct, and no release before 10.6 has the method.
• May widen. Open PR #16857 reuses the same skip rules for TopFieldCollector early termination, and open PR #16699 extends the shortcut to FLOAT and DOUBLE sorts.
• How likely. Rare: the opt-in multi-field index sort plus an uncommon reader or sort-field setup. Once that setup exists, many range queries on the second sort field are wrong.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.search.TestDerivedSortKeyRangeQuery.testRangeOnSecondSortField(TestDerivedSortKeyRangeQuery.java:80)

• The method's own contract. "If sorting against a field would have no effect then that SortField is skipped over ... This can happen for fields with no values in the segment, or fields that have only a single distinct value" (Sort.java:141-143); the code comment says "Field has no values in this segment" (:157). The segment has values of c and is ordered by them; the BinarySortField key orders every document.
• The shortcut's contract. IndexSortSortedNumericDocValuesRangeQuery uses it only if "its primary sort is on the same field as the query", and "This fallback must be an equivalent range query -- it should produce the same documents" (IndexSortSortedNumericDocValuesRangeQuery.java:46, :55). NumericDocValuesField.newSlowRangeQuery "matches all documents whose value is between lowerValue and upperValue included" (NumericDocValuesField.java:91-92).
• The trigger is documented. BinarySortField: "A subclass may override getSortKeyDocValues(LeafReader) to derive an order-preserving key from the stored bytes and/or other doc-values fields of the segment" (BinarySortField.java:41-44, again at :150-152), and IndexingChain accepts a sorter that reads other fields. FilterLeafReader passes every call through, "possibly transforming the data along the way" (FilterLeafReader.java:30-31), delegates getMetaData() (:477-481), and its notes cover only live docs, numDocs and cache helpers (:37-42).
• History. #16424 (merged 2026-07-30) describes using "index-level stats from FieldInfo or associated DocValuesSkipper entries"; its one review thread is about density, and nothing in it discusses wrappers or derived keys. CHANGES: "Use secondary or lower sorts to accelerate XXDocValuesRangeQueries if the primary sort field is empty or only contains a single value" (lucene/CHANGES.txt:645-646). Before it, the shortcuts used only the first sort field, and the releases tested (10.5.0-10.5.2, 10.4.0, 10.0.0, 9.12.3, 9.10.0) give the right hits through the same wrapper and sort; by reading, so does every release before 10.6.
• Tests. TestGetPrimarySortField covers only plain segment leaves (testPrimaryWithNoValuesInSegmentIsSkipped, :137, uses a field that is truly absent); TestBinarySortField's two subclasses read only their own field. No upstream test runs these queries through a field-hiding reader or with a derived key.

The strongest works-as-designed case (the devil's advocate, the cross-model intended lens, the adjudication): a reader that hides an index-sort field but keeps getMetaData().sort() contradicts itself, so the wrapper should drop or rewrite the sort. There is real support for this: on main, a 10.6 snapshot and 10.5.2, a search sorted by (c, f) through such a reader returns view-inconsistent top hits (f = [1, 5] instead of [1, 3]), because TopFieldCollector early termination trusts the metadata sort. And BinarySortField is experimental; its in-tree subclasses read only their own field.

Why it fails: no javadoc states that obligation, FilterLeafReader delegates the sort by default and Lucene's own field-subset reader keeps it; the query here never references the hidden field and was right through the same wrapper on 10.5.2, while the older inconsistency affects only sorts on the hidden field; a reader that hides c completely (FieldInfos, terms, points, every doc-values type, the skipper and norms) fails the same way; and the BinarySortField form needs no wrapper at all.

Two fixes were built and run in a scratch worktree at a9779ef3c193. A also covers custom sort fields other than BinarySortField and is the suggested one, provisionally: it was validated on main only and assumes the wrapper only hides data. B is the finder's sketch. Until a fix ships, a hiding wrapper that also returns getMetaData() without the sort gets the right hits (executed on main, a 10.6 snapshot and 10.5.2).

Read FieldInfos and the skipper from FilterLeafReader.unwrap(reader), the segment that holds the order, and return any sort field that is not exactly SortField, SortedNumericSortField, SortedSetSortField or BinarySortField as the primary sort without trying to skip it. +19/-5 in Sort.java.

LeafReader segment = FilterLeafReader.unwrap(reader);
for (SortField sf : sort.fields) {
  String field = sf.getField();
  if (field == null || isOrderedByOwnValues(sf) == false) {
    return sf; // custom, or a key that may come from other fields
  }
  if (segment.getFieldInfos().fieldInfo(field) == null) continue;
  DocValuesSkipper skipper = segment.getDocValuesSkipper(field);
  ... // dense and minValue() == maxValue(): continue, as today
}
// isOrderedByOwnValues: sf.getClass() is SortField, SortedNumericSortField,
// SortedSetSortField or BinarySortField

Trade-offs

• The built-in classes keep #16424's optimisation: an absent or constant field really is a no-op in a segment. Every subclass, BinarySortField subclasses included, is returned as the primary sort, so it never feeds a shortcut on another field; that only costs the optimisation, never correctness.
• The range queries still read the queried field's values and skipper through the wrapper, as today.
• Gap: a wrapper that is not a FilterLeafReader (a direct LeafReader subclass that delegates getMetaData()) is not unwrapped. Not benchmarked; the change adds one unwrap loop per call.
• Assumes the wrapper only hides data, so that the segment's FieldInfos and skippers describe the sort it reports; FilterLeafReader also allows transforming wrappers (rewritten metadata, synthesized fields), and no test covers one. The exact-class allowlist changes what a public static method returns for harmless subclasses. Maintainers may prefer a more conservative rule; documenting the wrapper obligation is legitimate for the wrapper form.
• Main only: a branch_10_6 backport needs the 10.x method's try/catch for IOException (and its annotation). No fix A run used the RC1 artifacts or the 10.6 snapshot.

What the checks validated

Gradle in a scratch worktree at a9779ef3c193, seed DEADBEEF: the 5 repro classes (13 tests) and 68 neighbouring core suites (TestGetPrimarySortField, TestDocValuesQueries, TestIndexSortSortedNumericDocValuesRangeQuery, TestBinarySortField, the sort tests, TestIndexSorting and every org.apache.lucene.document test): 747 tests, 0 failures, 26 skipped, with assertions on and off; on stock the same run fails exactly the 13 repro tests. :lucene:core:check -x test with the fix and the new test passes (google-java-format, ecj lint, forbidden APIs).

The finder's sketch: the same unwrap, and field == null || sf instanceof BinarySortField returns the sort field as primary. +9/-5 in Sort.java.

Trade-offs

Fixes the same cases, but other custom SortField subclasses with their own index sorter, whose key need not be their named field's values, are still skipped.

What the checks validated

Gradle, assertions on: the same 747 tests, 0 failures, 26 skipped. The finder's earlier runner check: its 5 repro tests and 72 upstream tests pass.

Say in FilterLeafReader (and getPrimarySortField) that a reader hiding an index-sort field must drop or rewrite getMetaData().sort().

Trade-offs

A legitimate maintainer choice for the wrapper form, and the workaround users can apply today; it leaves the BinarySortField form, which needs no wrapper. The javadoc change itself is text; the wrapper side, a reader that drops the sort, is the executed workaround.

Remove the "not in FieldInfos" rule and keep the dense single-value skipper rule.

Trade-offs

Still misfires through the skipper rule: the independent reproducer's diagnostic shadow fixed 4 of 5 tests, and the skipper-rule test still failed. It also changes what TestGetPrimarySortField.testPrimaryWithNoValuesInSegmentIsSkipped asserts (by reading).

Make getPrimarySortField return the first sort field, the rule before #16424.

Trade-offs

The repros pass (the follow-up's revert check on main and the 10.6 snapshot), but #16424's optimisation and its TestGetPrimarySortField assertions go.

Method: code inventory, then metamorphic testing and bounded-exhaustive translation validation (the hunt's "semi-formal" tier).

• The hunt unit covered the index-sort shortcuts of the doc-values range queries. Listing every consumer of Sort.getPrimarySortField over the non-sandbox sources showed the rule that a sort field missing from the reader's FieldInfos is skipped.
• A metamorphic probe hid a field with FieldFilterLeafReader and compared the hits of range queries on the other field: 595 of 600 wrong on main and a 10.6 snapshot built from the RC1 commit, 0 of 600 on 10.5.2. A BinarySortField derived-key probe: 99 of 100 wrong on main and the 10.6 snapshot, 0 of 100 on 10.5.2.
• Bounded-exhaustive translation validation of the stock jar against brute-force range semantics: every layout of up to 5 documents over (c, f) in {0,1} x {1,2,3}, index sort (c, f) LONG, NumericDocValuesField with a skip index, every range on f, c hidden by FieldFilterLeafReader (its FieldInfo only); the BinarySortField and LongField forms are not part of it. Plain reader 0 of 55,980 checks wrong; reader hiding c 22,120 wrong in 5,964 of 9,330 layouts; 10.5.2 and the fix sketch 0.
• The finder then wrote the public-API JUnit tests, and independent checkers reproduced them.