GroupingSearch never rewrites its group sort or within-group sort against the searcher, unlike IndexSearcher#search

GroupingSearch takes a group sort and a within-group sort and runs its collectors on the searcher passed to search(searcher, query, ...). Since 7.2.0, IndexSearcher#search(query, n, sort) binds a Sort to the searcher with sort.rewrite(this) before sorting; GroupingSearch never does, although it rewrites its own groupEndDocs query against that searcher. A sort that works in IndexSearcher#search therefore throws in GroupingSearch ("SortField needs to be rewritten ..." for a ValueSource#getSortField sort, "This DoubleValuesSource must be rewritten" for a DoubleValuesSource.fromQuery sort), and a sort on a legacy QueryValueSource through asDoubleValuesSource() silently scores with the default Similarity: on a ClassicSimilarity searcher a two-document group comes back as doc0=0.147, doc1=0.125 (default-BM25 scores) instead of doc1=1.0, doc0=0.894. Group order and group heads go wrong the same way; the set of matching documents never changes. Every release from 7.2.0 to 10.6.0 and main is affected. Rewriting both Sorts at the start of the two grouping methods (4 added lines) fixes every case; all 56 existing grouping tests stay green.

1. IndexSearcher binds the Sort. searchAfter, which search(Query, int, Sort) uses, does final Sort rewrittenSort = sort.rewrite(this); and builds its TopFieldCollectorManager from it (IndexSearcher.java:730-733). Sort.rewrite rewrites each field and returns this when nothing changes (Sort.java:88-99). It is the only Sort.rewrite call in the non-test sources under lucene/.
2. GroupingSearch does not. groupByFieldOrFunction passes the field groupSort to FirstPassGroupingCollectorManager (GroupingSearch.java:148), sortWithinGroup to AllGroupHeadsCollectorManager (:160) and both to TopGroupsCollectorManager (:201-202). groupByDocBlock rewrites its groupEndDocs query with searcher.rewrite(this.groupEndDocs) (:223) and then passes both Sorts as given to BlockGroupingCollectorManager (:228, :233). The collector managers take only a Sort and never see the searcher.
3. Loud path. The collectors build comparators with SortField.getComparator (FirstPassGroupingCollector.java:116, AllGroupHeadsCollector.java:241, BlockGroupingCollector.java:237). A Type.REWRITEABLE field, which ValueSource#getSortField returns (ValueSource.java:369-384), throws "SortField needs to be rewritten through Sort.rewrite(..) and SortField.rewrite(..)" (SortField.java:555-557); a DoubleValuesSource.fromQuery source throws "This DoubleValuesSource must be rewritten" until it is rewritten (DoubleValuesSource.java:768).
4. Silent path. ValueSource#asDoubleValuesSource() returns a wrapper with no searcher (ValueSource.java:184-185); its values get a function context whose searcher is null (:198-200), and only rewrite(searcher) returns a wrapper that carries the searcher (:244-245). QueryValueSource then finds no searcher and scores its query with new IndexSearcher(ReaderUtil.getTopLevelContext(readerContext)) (QueryValueSource.java:116-123): the same reader, the default Similarity.
5. What does not change. The collectors only see documents the caller's query matched on the caller's searcher (GroupingSearch.java:173, :175, :211, :237): the matching documents and groups are the same; only order, sort values and group heads differ, or the request throws.

On ClassicSimilarity, body:a scores sqrt(tf)/sqrt(length) with idf 1 (both documents match): doc 1 gets sqrt(1)/sqrt(1) = 1.0 and doc 0 gets sqrt(8)/sqrt(10) = 0.894. A plain IndexSearcher over the same reader (default BM25) scores them 0.14679137 and 0.12456752, and those are exactly the sort values GroupingSearch returns.

• Who is hit. Applications that group with GroupingSearch (field, function or doc-block grouping) and sort groups or the documents inside a group by a queries-module function sort (ValueSource#getSortField), a query-score sort (DoubleValuesSource.fromQuery) or an IndexReaderFunctions sort that needs the searcher, such as docFreq or sumTotalTermFreq (all loud), or by a legacy QueryValueSource through asDoubleValuesSource() (silent when the Similarity is not the default).
• Default vs opt-in. Sort.RELEVANCE (the default) and field sorts are unaffected. The silent form also needs a searcher Similarity other than the default BM25: ClassicSimilarity, BooleanSimilarity and BM25 with other parameters were all executed. No codec, analyzer or index setting is involved; assertions on or off.
• Scale. Any index size (two documents are enough); deterministic: every grouped search with such a sort.
• Silent vs loud. The loud forms fail on the first grouped search with a message that names the rewrite. The silent form returns plausible but wrong orderings, sort values and group heads with no signal.
• How likely. Rare: upstream grouping tests and docs use only field sorts and relevance, and nobody has reported it since 7.2.0 (2017).
• Workaround. setGroupSort(sort.rewrite(searcher)) and setSortWithinGroup(sort.rewrite(searcher)), once per searcher: on the example it gives doc1=1.0, doc0=0.894.

The test sits in org.apache.lucene.search.grouping only for locality; it uses only public API. ByteBuffersDirectory, core StandardAnalyzer, the default IndexWriterConfig, two documents (one segment), a plain IndexSearcher with ClassicSimilarity, nothing random: deterministic, single thread. It runs GroupingSearch before IndexSearcher#search, so no version can hand the searcher from one to the other through the shared Sort object. The same test is the suggested regression test.

On 9.1 to 9.12, GroupDocs has public fields: write grouped.groups[0].scoreDocs instead of scoreDocs() (one line).

cp $REPRO/TestGroupingSearchSortRewrite.java $LUCENE/lucene/grouping/src/test/org/apache/lucene/search/grouping/
cd $LUCENE
./gradlew :lucene:grouping:test --tests org.apache.lucene.search.grouping.TestGroupingSearchSortRewrite -Ptests.seed=DEADBEEF
# production setting (assertions off): add -Ptests.asserts=false

# gradle, main 5adff78bb168, seed DEADBEEF, assertions on (the same with -Ptests.asserts=false)
TestGroupingSearchSortRewrite > testSortWithinGroupMatchesIndexSearcher FAILED
    org.junit.ComparisonFailure: expected:<doc[1=1.0 doc0=0.8944271802902222]> but was:<doc[0=0.14679136872291565 doc1=0.12456752359867096]>
        at org.apache.lucene.search.grouping.TestGroupingSearchSortRewrite.testSortWithinGroupMatchesIndexSearcher(TestGroupingSearchSortRewrite.java:72)
1 test completed, 1 failed
BUILD FAILED

# with the recommended fix, assertions on and off
:lucene:grouping:test (SUCCESS): 1 test
BUILD SUCCESSFUL

# JUnit runner on the released jars (seed DEADBEEF, -ea): the same message on 10.6.0, 10.5.2, 10.0.0, 9.12.3 and 9.1.0

# exploration probe on the same two documents (main 5adff78bb168)
PROBE classic scores of body:a: doc1=1.0 doc0=0.8944272
PROBE default(BM25) scores of body:a: doc0=0.14679137 doc1=0.12456752
PROBE GroupingSearch within-group, raw sort: doc0=0.14679136872291565 doc1=0.12456752359867096
PROBE GroupingSearch within-group, sort.rewrite(searcher): doc1=1.0 doc0=0.8944271802902222
PROBE IndexSearcher#search(q, 10, sort): doc1=1.0 doc0=0.8944271802902222
PROBE IndexSearcher, ValueSource#getSortField: doc1=1.0 doc0=0.8944271802902222
PROBE GroupingSearch, ValueSource#getSortField: java.lang.IllegalStateException: SortField needs to be rewritten through Sort.rewrite(..) and SortField.rewrite(..)
PROBE IndexSearcher, DoubleValuesSource.fromQuery: doc1=1.0 doc0=0.8944271802902222
PROBE GroupingSearch, DoubleValuesSource.fromQuery: java.lang.UnsupportedOperationException: This DoubleValuesSource must be rewritten
PROBE control, FIELD_DOC within group: doc0=0 doc1=1
PROBE control, default-Similarity searcher, raw sort: doc0=0.14679136872291565 doc1=0.12456752359867096 vs IndexSearcher doc0=0.14679136872291565 doc1=0.12456752359867096

• The class that holds the searcher rewrites. LUCENE-7736 (7.2.0) moved the rewrite into IndexSearcher (IndexSearcher.java:730); its author wrote that Sort.rewrite() "wasn't being called by IndexSearcher or SortRescorer" (#8787, 2017-04-10). A consumer left behind then was fixed as a bug: "LUCENE-7736: CoveringQuery needs to rewrite its ValuesSource." (6b621ea659b on branch_7x and 3f0b240853d on master, 2017-11-22). Earlier, Solr fixed another consumer that holds the searcher in the same way: SOLR-6118, "expand.sort bug for function queries; needed to sort.rewrite(searcher)" (6ffdda4545e, 2014).
• GroupingSearch holds the searcher. search(searcher, query, ...) is given the IndexSearcher, and groupByDocBlock rewrites its groupEndDocs query against it (GroupingSearch.java:223). Only the Sorts are left out.
• No precondition is documented. The class javadoc (:41-45), search (:115-129), setGroupSort ("Specifies how groups are sorted", :285-294) and setSortWithinGroup (:296-305) say nothing about pre-rewriting; neither do the package docs, MIGRATE.md or CHANGES.txt.
• An upstream test expects agreement. BaseGroupSelectorTestCase#testSortGroups sets the same Sort on GroupingSearch and on searcher.search(topLevel, 1, sort) and asserts (assertArrayEquals, :118) what its code comment states: "The top group should have sort values equal to the sort values of the top doc of a top-level search sorted by the same Sort" (BaseGroupSelectorTestCase.java:83-121, comment :112-114); it uses field sorts only.
• No consistent semantics explains the current answers. One Sort object gives one order through IndexSearcher#search and another through GroupingSearch; a sort IndexSearcher accepts throws in GroupingSearch; and in the silent case the grouped order depends on a Similarity the application never chose.

The strongest case for "works as designed": callers rewrite Sorts. That was the rule when GroupingSearch was written (LUCENE-3778, 2012, when IndexSearcher did not rewrite either); the REWRITEABLE javadoc and the error message tell you to call Sort.rewrite; upstream TestFunctionQuerySort still calls sf.rewrite(searcher) itself (TestFunctionQuerySort.java:65); Solr's grouping weights its sorts before building the collectors; the low-level collector managers have no searcher; #5619 describes caller-side rewriting as the status quo; and GroupingSearch is @lucene.experimental.

Why it fails: unlike the collector managers, GroupingSearch#search is handed the searcher and already rewrites its own query with it; since 7.2.0 the Lucene class that holds the searcher rewrites, and a missed consumer was fixed as a bug; no documentation states the precondition; and a silent fallback to a different Similarity is not a plausible design. The argument does fairly make the loud forms an API-consistency defect rather than corruption. Two devil's-advocate reviews (one by an independent reviewer from another model family, OpenAI GPT-5.6) rated it moderate and not persuasive; the adjudication agreed.

Four options. A was built and validated on main 5adff78bb168; B to D were not built. Workaround until a fix ships: setGroupSort(sort.rewrite(searcher)) and setSortWithinGroup(sort.rewrite(searcher)) once per searcher.

In groupByFieldOrFunction, before the collector managers are built, and in groupByDocBlock, after the groupEndDocs rewrite, add two local variables that shadow the fields. The rest of both methods is unchanged. One file, +4 lines.

// groupByFieldOrFunction, before the first collector manager; groupByDocBlock, after endDocsQuery
final Sort groupSort = this.groupSort.rewrite(searcher);
final Sort sortWithinGroup = this.sortWithinGroup.rewrite(searcher);

Trade-offs

• Cost: Sort.rewrite returns this when no field changes, so field sorts and relevance get no new objects; a function sort is rewritten once per search, as in IndexSearcher#search. Already-rewritten Sorts are unaffected (a second rewrite is harmless).
• The returned TopGroups#groupSort / withinGroupSort then hold the rewritten SortFields, as TopFieldDocs#fields does after IndexSearcher#search.
• GroupingSearch then shares the path IndexSearcher uses for these sorts, with its two defects listed on this site: LFV-770 (a rewritten toSortableLongDoubleValuesSource() sorts on (long) casts) and LFV-567 (a ValueSource sort context shared across the slices of an executor search). Fix LFV-770 with or before this.
• Subclasses that override the protected grouping methods do not get the rewrite.

What the checks validated

• Gradle in a scratch worktree of 5adff78bb168, OpenJDK 25.0.3, seed DEADBEEF: stock, the repro fails with assertions on and off; with the fix it passes, on and off.
• Every test of the grouping module passes with the fix and the repro in place: 15 classes, 57 tests (56 existing + the repro), 0 failures.
• checkGoogleJavaFormat, ecjLint, forbiddenApisMain and forbiddenApisTest of lucene/grouping pass.
• With the patched class compiled over the stock jars, the probe's ValueSource#getSortField and fromQuery sorts no longer throw, and the default-Similarity control is unchanged. A second probe on the same two documents, run separately with the fix applied, shows both sorts then return doc1=1.0, doc0=0.8944271802902222 in GroupingSearch, the same as IndexSearcher#search; on the stock jars both throw.
• On a9779ef3c193 and ea57360912ed the same 4 lines passed the finding's 3 tests, the independent reproducer's 6 tests and the cross-model reviewers' own tests.

Javadoc on setGroupSort and setSortWithinGroup: "rewrite the Sort against the searcher first".

Trade-offs

No code change, but it keeps the silent trap, ties a configured GroupingSearch to one searcher, and leaves it inconsistent with IndexSearcher#search.

Throw when the function context has no searcher, instead of creating a default-Similarity IndexSearcher (QueryValueSource.java:116-123).

Trade-offs

Turns the silent form into a loud one everywhere, not only in GroupingSearch, but changes behaviour for callers that rely on the fallback and does nothing for the loud forms. A separate question for maintainers.

The generic direction of #5619: every consumer rewrites.

Trade-offs

The managers take only a Sort and never see a searcher, so this needs an API change in every grouping manager; much broader than A.

Method: semi-formal (bounded-exhaustive translation validation) of the APIs that consume a Sort or a values source.

• A ClassFile scan of the stock jars listed the grouping collectors' getComparator sites (FirstPassGroupingCollector:116, the TopGroups/SearchGroup comparators, AllGroupHeadsCollector:241, BlockGroupingCollector:237) with no Sort rewrite on the path from GroupingSearch#search.
• A test program on the stock main jars (assertions off) then enumerated every index of 1-3 documents over a 12-value per-document alphabet, in all 1-2 segment splits and deletion masks (42,648 index configurations), x 3 searcher Similarities (BM25, Classic, Boolean) x the 282 applicable pairs of 18 values sources and 23 consumer APIs (some consumers take only double or only long sources): 36,080,208 runs of a consumer fed the raw Sort or source, compared with the same consumer fed it pre-rewritten against the searcher.
• The GroupingSearch group-sort and within-group consumers were silent (different results, no error) under Classic and Boolean similarity for every legacy query() function source (group heads: under Boolean), and loud (an exception) for fromQuery sources and for the IndexReaderFunctions sources that need the searcher (docFreq, sumTotalTermFreq; termFreq gave the same result either way). The IndexSearcher#search control was never silent.
• The case was reduced to a 3-test public-API JUnit test (2026-10-27), which failed with assertions on and off.