Doc-values range queries throw NullPointerException on a leaf that reports Sort.INDEXORDER as its index sort

Since 10.0.0, doc-values range queries on a skip-indexed, single-valued field have a shortcut for segments sorted by the queried field. To decide, they ask Sort.getPrimarySortField(reader) for the segment's primary index sort field and compare its name with the query's field, sf.getField().equals(field). getPrimarySortField returns a sort field without a name on purpose ("Custom field that we don't know anything about, so return it as primary"), and SortField.FIELD_DOC and FIELD_SCORE have no name. So on a leaf whose reader reports Sort.INDEXORDER (a true statement about any segment) the comparison throws NullPointerException instead of skipping the shortcut, and IndexSearcher.search and count fail. The same unguarded comparison is in SortedNumericDocValuesRangeQuery and in three places of IndexSortSortedNumericDocValuesRangeQuery. IndexWriter rejects DOC and SCORE index sorts and Lucene's readers report one only through a merge-only expert wrapper, so in practice only applications with their own LeafReader are hit; the failure is loud and no wrong hits are returned.

1. The shortcut is tried for a field with a doc-values skip index and single-valued sorted values (document/SortedSetDocValuesRangeQuery.java:121-131, condition singleton != null && skipper != null at :124). SortedDocValuesField.newSlowRangeQuery and newSlowExactQuery build this query (document/SortedDocValuesField.java:96-116).
2. It asks for the primary sort field: canUsePrimarySortShortcut calls Sort.getPrimarySortField(reader) (:288-290).
3. The helper returns a nameless field on purpose. getPrimarySortField (search/Sort.java:145-148) uses primarySortFieldIndex (:154-183), which skips sort fields that are no-ops in the segment but returns a field with no name at once: if (field == null) { // Custom field that we don't know anything about, so return it as primary (:162-164). Sort.INDEXORDER = new Sort(SortField.FIELD_DOC) (:44), FIELD_DOC = new SortField(null, Type.DOC) (search/SortField.java:124), and getField() "Could return null if the sort is by SCORE or DOC" (:380-386).
4. The caller dereferences it (the fix site). if (sf == null || sf.getField().equals(field) == false) (:291) throws, although the method's javadoc says it returns "null if the shortcut cannot be applied" (:280-287).
5. Twin sites, same expression. document/SortedNumericDocValuesRangeQuery.java:240 (reached from :141-147, for example by NumericDocValuesField.newSlowRangeQuery on an indexedField), and search/IndexSortSortedNumericDocValuesRangeQuery.java (reached by LongField.newRangeQuery and IntField.newRangeQuery, document/LongField.java:138-146, with no skip index needed) at :433 (in getDocIdSetIteratorOrNullFromBkd, reached from count and from the scorer path; the executed throw site), :206 (in count, when the field's points do not cover every document) and :536 (in getDocIdSetIteratorOrNull, after :433 has already been reached, so it never throws first; fixed for consistency).
6. The other operations tested on that leaf work. TermQuery, FieldExistsQuery, sorted searches (by a string field, a long field and Sort.INDEXORDER, including their early-termination check) and IndexWriter.addIndexes all accept a leaf that reports Sort.INDEXORDER; of the operations tested, only the range-query shortcuts throw.

• The repro: 3 documents (a, b, c), range b..c. The unwrapped segment counts 2 (the diagram probe, same documents); the wrapped one throws at SortedSetDocValuesRangeQuery.java:291.
• A fix-check probe (10 documents, values 0..9, range 2..5) on the same kind of leaf: all three query classes throw, for search and for count (:291, SortedNumericDocValuesRangeQuery:240, IndexSortSortedNumericDocValuesRangeQuery:433); the unwrapped segment gives 4 hits each.
• A leaf that reports new Sort(SortField.FIELD_SCORE) throws the same way, but a score is not a document order, so it is not the headline case.
• By reading only (not executed): since 10.6.0 a leading sort field that is a no-op in the segment and is followed by FIELD_DOC also reaches the dereference; 10.0.0 to 10.5.2 compared only getSort()[0].

• Who. Applications with their own LeafReader (a FilterLeafReader, a FilterCodecReader or a custom reader) whose getMetaData() reports an index sort whose first effective field is FIELD_DOC (such as Sort.INDEXORDER) or FIELD_SCORE, built with the public LeafMetaData constructor ("Expert: Sole constructor. Public for use by custom LeafReader impls."; the record is @lucene.experimental).
• Queries hit. Executed: SortedDocValuesField.newSlowRangeQuery and newSlowExactQuery and NumericDocValuesField.newSlowRangeQuery on skip-indexed single-valued fields, and LongField.newRangeQuery on a LongField that is single-valued in the segment (for a multi-valued one only count can reach the call, when the points cover every document; by reading). By reading: SortedSetDocValuesField (single-valued in the segment) and IntField.
• Default vs opt-in. Not reachable in normal use of IndexWriter or Lucene's readers: IndexWriterConfig.setIndexSort(Sort.INDEXORDER) throws "Cannot sort index with sort field <doc>", and Sorter rejects such fields too. Two contrived in-tree routes exist, and the claim rests on neither. The expert SortingCodecReader.wrap(reader, null, sort) reports the caller's sort without checking it and gives the same exception (executed), but its javadoc reserves it for merging. And, by reading only (not executed), setIndexSort checks nothing but the sort field's IndexSorter, so an application-defined SortField subclass with no field name, its own IndexSorter and its own registered sort-field provider would pass that check and be stored in the segment; Lucene's own sort-field providers all read a field name, so this too needs application code.
• Effect. An unchecked NullPointerException from IndexSearcher.search or count for that query; no wrong or missing hits returned silently, no index change.
• Scale. Any size; 3 documents suffice.
• Likelihood: contrived. An application has to advertise a DOC or SCORE sort as its leaves' index sort (rather than null or the wrapped reader's metadata) and then run one of these range queries; no public report of it was found since 9.5.0.

TestDocValuesRangeQueryIndexOrderSort (one method, extends LuceneTestCase, package org.apache.lucene.document of the lucene/core tests). The writer uses the default codec explicitly (TestUtil.getDefaultCodec()), so the doc-values skip index exists whatever codec the seed draws; it failed on every recorded seed. The wrapper returns the segment's own metadata with only the sort replaced, and disables caching (null cache helpers), as a FilterLeafReader that changes metadata must.

As a regression test the same method fits in TestDocValuesQueries.

cp $REPRO/TestDocValuesRangeQueryIndexOrderSort.java $LUCENE/lucene/core/src/test/org/apache/lucene/document/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.document.TestDocValuesRangeQueryIndexOrderSort -Ptests.seed=1205DEAD
# and with -Ptests.asserts=false: the same failure

TestDocValuesRangeQueryIndexOrderSort > testRangeQueryOnIndexOrderSortedLeaf FAILED
    java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because the return value of "org.apache.lucene.search.SortField.getField()" is null
        at __randomizedtesting.SeedInfo.seed([1205DEAD:FA19303DE5B1D211]:0)
        at org.apache.lucene.document.SortedSetDocValuesRangeQuery.canUsePrimarySortShortcut(SortedSetDocValuesRangeQuery.java:291)
        at org.apache.lucene.document.SortedSetDocValuesRangeQuery$1.scorerSupplier(SortedSetDocValuesRangeQuery.java:126)
        at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:830)
        at org.apache.lucene.search.IndexSearcher.searchLeaf(IndexSearcher.java:842)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:814)
        ...
        at org.apache.lucene.search.IndexSearcher.count(IndexSearcher.java:528)
        at org.apache.lucene.document.TestDocValuesRangeQueryIndexOrderSort.testRangeQueryOnIndexOrderSortedLeaf(TestDocValuesRangeQueryIndexOrderSort.java:70)
1 test completed, 1 failed

• The callers' own contracts. canUsePrimarySortShortcut returns "the primary sort field if it is safe to use the ... range shortcuts for this query, or null if the shortcut cannot be applied" (SortedSetDocValuesRangeQuery.java:280-287; the same words at SortedNumericDocValuesRangeQuery.java:229-236). IndexSortSortedNumericDocValuesRangeQuery's class javadoc: when "its primary sort is on the same field as the query" does not hold, "the search is delegated to fallbackQuery" (:43-53).
• The callee returns such a field by design, with a comment saying so (Sort.java:162-164), and SortField.getField() documents null for SCORE and DOC (SortField.java:380-381).
• The metadata is truthful and accepted. LeafMetaData.sort is "the order in which documents from this index are sorted, or null if documents are in no particular order" (LeafMetaData.java:32-33); index order is true for any segment, and the constructor checks only the version fields, not the sort (:49-61). No documentation was found that calls reporting this redundant DOC order either supported or unsupported; that is the maintainers' call.
• Sibling code is null-safe. SortingCodecReader compares field.equals(sf.getField()) (SortingCodecReader.java:901); FieldValueHitQueue.canEarlyTerminate compares whole sort fields (FieldValueHitQueue.java:202-214).
• No test, CHANGES entry or discussion says DOC or SCORE leaf sorts are unsupported: TestGetPrimarySortField and TestDocValuesQueries have no nameless-field case, and the #13592 and #16424 reviews do not mention DOC or SCORE sorts.

The strongest case for "intended": advertising Sort.INDEXORDER as an index sort is misuse. IndexWriter rejects DOC and SCORE index sorts, LeafMetaData is expert and experimental, and null is the canonical "no particular order".

Why it fails: none of that is a documented precondition. The writer rejects DOC and SCORE because it has no IndexSorter to reorder documents with, which is about applying a sort, not about describing an order that needs no reordering. The helper the queries call returns such fields on purpose, and the queries' own javadocs promise null or a fallback. The cross-model lenses rated this argument moderate and rejected it; the devil's-advocate verifier found no documented precondition. A maintainer may still call it a "don't do that" corner and give it low priority, which is a fair triage outcome for a Low bug; but a NullPointerException is not the documented answer.

Both options were built and checked through gradle on 5adff78bb168, one at a time, and with a fix-check probe over all three query classes.

field.equals(sf.getField()) instead of sf.getField().equals(field) at SortedSetDocValuesRangeQuery.java:291, SortedNumericDocValuesRangeQuery.java:240 and IndexSortSortedNumericDocValuesRangeQuery.java:206, :433, :536 (5 lines in 3 files). field is never null: the constructors call Objects.requireNonNull.

    SortField sf = Sort.getPrimarySortField(reader);
    if (sf == null || field.equals(sf.getField()) == false) {
      return null;
    }

Trade-offs

Smallest and local; covers DOC, SCORE and any other nameless primary; matches SortingCodecReader.java:901; one string comparison per leaf, as before. It leaves getPrimarySortField free to return nameless fields, so a future caller has to remember the same guard.

What the checks validated

• Gradle in a scratch worktree at 5adff78bb168, seed 1205DEAD: the repro fails without the fix (assertions on and off) and passes with it (assertions off).
• With the fix and assertions on: the repro (1), TestDocValuesQueries (41), TestIndexSortSortedNumericDocValuesRangeQuery (29), TestGetPrimarySortField (9) and TestSortedSkipperScorerSupplier (5): 85 tests, 0 failures. checkGoogleJavaFormat passes with the fix and the repro in place.
• The fix-check probe gives 4 hits for search and count in all three classes on the index-order leaf. The full core suite was not run.

In Sort.getPrimarySortField (Sort.java:145-148), return null when the effective primary sort field has no name. primarySortFieldIndex is unchanged, so early termination in FieldValueHitQueue is not affected.

    SortField sf = reader.getMetaData().sort().fields[idx];
    return sf.getField() == null ? null : sf;

Trade-offs

One place, and it protects future callers. But it changes what a public static method returns, for external callers too (only the in-tree callers were tested), so its javadoc ("or null if none is configured") needs a sentence, and so does primarySortFieldIndex's ("the field that getPrimarySortField would return").

What the checks validated

The repro passes (assertions on), the probe gives 4 hits everywhere, and the repro plus the same 4 suites pass through gradle with B instead of A: 85 tests, 0 failures.

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

• The finder checked canUsePrimarySortShortcut against its javadoc ("or null if the shortcut cannot be applied") for every value Sort.getPrimarySortField can return, and found that the nameless primary it returns on purpose is dereferenced.
• The finder wrote a public-API JUnit test and the one-token fix that makes it pass. A first draft used the merge-only SortingCodecReader.wrap(reader, null, sort); after the refute-lens review pointed that out, the test was rebuilt on a FilterLeafReader and every run was redone.
• The twin site in SortedNumericDocValuesRangeQuery has the same root cause and is part of this finding. An independent reproducer, two executed cross-model reviews, a devil's-advocate verifier and an adjudication reproduced it with their own tests ("Prior reports, history and checks").