GroupingSearch ignores setIncludeMaxScore when grouping by doc block: with field sorts every maxScore is NaN (since 8.0.0)

GroupingSearch.setIncludeMaxScore ("Whether to include the score of the most relevant document per group", on by default) is ignored when grouping by doc block (new GroupingSearch(Query groupEndDocs)). groupByDocBlock passes needsScores = groupSort.needsScores() || sortWithinGroup.needsScores() to BlockGroupingCollectorManager and never reads includeMaxScore, so with a field group sort and a field within-group sort the collector runs with ScoreMode.COMPLETE_NO_SCORES and TopGroups.maxScore and every GroupDocs.maxScore are NaN, the value TopGroups documents as "scores were not computed". Two-pass grouping in the same class passes the flag on and reports the real values (0.22094907 overall in the example; groups 0.19659248 and 0.22094907). The hits, their order and the groups are unaffected, nothing throws, and a relevance sort on either side hides it. It dates from LUCENE-8412 (8.0.0), which removed the separate setIncludeScores option that had kept doc-block scoring on by default; the doc-block path never read includeMaxScore in any version. The fix is one line: add includeMaxScore to that expression, the rule the two-pass path already follows.

• Doc-block grouping: each group is indexed with one IndexWriter.addDocuments call and a query (groupEndDocs) matches the last document of each block. new GroupingSearch(Query groupEndDocs) (GroupingSearch.java:106) dispatches to groupByDocBlock (:132-133).
• Two-pass grouping: new GroupingSearch(String groupField) (or a function or group selector); its second pass is TopGroupsCollectorManager.
• setIncludeMaxScore: "Whether to include the score of the most relevant document per group" (GroupingSearch.java:329-338), default true (:56). TopGroups.maxScore: "Highest score across all hits, or Float.NaN if scores were not computed" (TopGroups.java:59).

1. groupByDocBlock (GroupingSearch.java:221-237) builds new BlockGroupingCollectorManager<>(groupSort, groupOffset, groupLimit, groupSort.needsScores() || sortWithinGroup.needsScores(), groupEndDocs, ...); the fourth argument (:231) is needsScores. The field includeMaxScore is read only at :205 (two-pass); its setter (:335-336) writes it.
2. With two field sorts needsScores is false. BlockGroupingCollector.scoreMode() then returns COMPLETE_NO_SCORES (BlockGroupingCollector.java:506), so no scores are computed.
3. In getTopGroups, groupMaxScore = needsScores ? Float.NEGATIVE_INFINITY : Float.NaN (:302) and the maximum is taken only if (needsScores) (:307-311): every group keeps NaN, and TopGroups.maxScore, a Math.max over them (:339-341), is NaN too.
4. BlockGroupingCollectorManager.reduce (:148-159) passes the collectors' results to TopGroups.mergeBlockGroups, which keeps their GroupDocs (TopGroups.java:430-432) and merges the overall maximum from NaN with nonNANmax, which stays NaN when every input is NaN (:106-110, :404, :413).
5. The two-pass path passes includeMaxScore to TopGroupsCollectorManager (GroupingSearch.java:197-205), and TopGroupsCollector scores when getMaxScores || withinGroupSort.needsScores() (TopGroupsCollector.java:128).

In these runs the hits, their order and the groups are the same on both paths. The minimal repro is one block of one document under MatchAllDocsQuery (score 1.0): NaN instead of 1.0.

• Who is hit. Code that groups block-indexed documents through GroupingSearch(Query groupEndDocs) with a field group sort and a field within-group sort (for example products grouped by parent and sorted by price) and reads TopGroups.maxScore or GroupDocs.maxScore, to show a relevance figure or normalise scores per group.
• Default or opt-in. Doc-block grouping and field sorts are opt-in (both sorts default to relevance, GroupingSearch.java:51-52); the ignored option is at its default, true (set in code at :56; the setter's javadoc does not state the default). GroupingSearch is @lucene.experimental (:44).
• Effect. A requested statistic is missing (NaN), visibly, not a plausible wrong number. Nothing throws. Code that compares, thresholds or divides by it gets NaN arithmetic.
• Scale. None needed: one document is enough. It needs at least one returned group; with no matching group there is no score to report, so NaN is right there anyway.
• Workaround. Build BlockGroupingCollectorManager directly with needsScores=true (executed: real values). It is not a drop-in replacement: the caller builds the end-of-block Weight itself (BlockGroupingCollectorManager.java:51-53, :91) and, with intra-segment concurrency, must keep each document block within one search slice (:36-43).
• How likely: rare. The hunt's downstream checks found no use of BlockGroupingCollector in Solr, Elasticsearch or OpenSearch, and their issue and PR trackers have nothing on it; private applications are not visible to such searches.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<1.0> but was:<NaN>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.junit.Assert.assertEquals(Assert.java:701)
       org.apache.lucene.search.grouping.TestBlockGroupingIncludeMaxScore.testMaxScoreWithFieldSorts(TestBlockGroupingIncludeMaxScore.java:52)

• Contract. GroupingSearch.java:329-338: "Whether to include the score of the most relevant document per group", no restriction. setAllGroups: "When grouping by doc block ... this option doesn't have any influence" (:344-345); setAllGroupHeads: "This feature isn't enabled when grouping by doc block" (:371).
• Package docs. package-info.java:143-151: "Or alternatively use the GroupingSearch convenience utility" with new GroupingSearch(groupEndDocs) and groupingSearch.setIncludeMaxScore(needsScores). (As written it sets no within-group sort, so its default relevance sort scores anyway: it shows the mismatch, it is not itself broken.)
• The collector's own javadoc. BlockGroupingCollector.java:205-207: needsScores is true "because relevance is included in the withinGroupSort or because you plan to pass true for either getSscores or getMaxScores" (2011 text; those parameters no longer exist).
• Lucene 8 intent. LUCENE-8405 (#9452, 5 days before the cause) removed TopDocs.maxScore but added an explicit group maximum to BlockGroupingCollector and a MaxScoreCollector to the two-pass collector; its MIGRATE text limits the removal to "IndexSearcher and TopFieldCollector" (grouping max scores survived the change; no design commitment is stated). LUCENE-8412 (#9459), the cause, is about TopFieldCollector's trackDocScores; its CHANGES entry (lucene/CHANGES.txt:6676) and MIGRATE entry mention neither GroupingSearch nor the removal of setIncludeScores. Its discussion touches grouping only once: Steven Rowe reported a Solr TestDistributedGrouping failure (.grouped[a_i1].doclist.maxScore:1.0!=null) bisected to 55bfadb, and the follow-up d010cdd87ee changed only Solr's TopGroupsFieldCommand. Nothing there mentions GroupingSearch, doc-block grouping or includeMaxScore.
• Sibling path. Two-pass grouping passes the flag (GroupingSearch.java:205, TopGroupsCollector.java:128).
• Why tests missed it. No test calls setIncludeMaxScore; TestGrouping.java:1632-1633 has the per-group maxScore check as a commented-out TODO since 2011; TestBlockGrouping and TestGroupingSearch use a relevance sort on at least one side.

The strongest case for "intended" (the refute reviewer from another model family, OpenAI GPT-5.6, the intended-behaviour devil's advocate and the review; all rate it moderate): Lucene 8 stopped computing scores for field-sorted searches, and MIGRATE suggests a separate query for the top score; LUCENE-8412 switched TestGrouping's own block collector to the same sort-only expression, so the change looks deliberate; includeMaxScore never controlled the doc-block path (on 7.7.3, setIncludeScores(false) with setIncludeMaxScore(true) also gives NaN, executed); NaN is a documented value; the 2026 package-info line came from a mechanical rename in #16508; and the BlockGroupingCollectorManager javadoc defines needsScores by the sorts only.

Why it fails: LUCENE-8405 kept explicit grouping max scores when scores were requested, and the two-pass path still honours the flag; the TestGrouping edit changed no assertion, and the records inspected (the commits, CHANGES, MIGRATE, code comments and the #9459 discussion) document no decision to drop doc-block max scores; an option of the same class, documented and on by default, is ignored on one path while the class documents such exceptions for its other options; and NaN means "not computed" when the caller asked for them. The TestGrouping edit is some evidence of intent, which is why the case is moderate. The 7.7.3 point weakens the word "regression" (do not call it a regression of setIncludeMaxScore itself), not the bug. Verdict: a defect; upstream may still prefer a documentation fix (see Suggested fix).

Option A was built and validated; B and C were not built. The adjudication's confidence is medium only because upstream may prefer B.

Add includeMaxScore to the needsScores argument of BlockGroupingCollectorManager in groupByDocBlock (GroupingSearch.java:231), the rule the two-pass path follows (TopGroupsCollector.java:128). Both paths then behave the same, and setIncludeMaxScore(false) starts to mean something on the doc-block path too.

--- a/lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupingSearch.java
+++ b/lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupingSearch.java
@@ -228,7 +228,7 @@
             groupSort,
             groupOffset,
             groupLimit,
-            groupSort.needsScores() || sortWithinGroup.needsScores(),
+            groupSort.needsScores() || sortWithinGroup.needsScores() || includeMaxScore,
             groupEndDocs,
             sortWithinGroup,
             groupDocsOffset,

Trade-offs

• Cost. A doc-block search with field-only sorts that leaves the flag at its default now runs with ScoreMode.COMPLETE instead of COMPLETE_NO_SCORES and computes scores; two-pass grouping already pays this by default. Not measured. Callers who do not need max scores get the old behaviour with setIncludeMaxScore(false); worth a line in CHANGES.
• Compatible with the separate sort-rewrite change to the same method (LFV-816).

What the checks validated

• In a scratch worktree of 645d4fe14a20 with gradle and JDK 25.0.3: stock + the repro test fails (1 of 1) with assertions on and off; with the fix the repro passes, the whole :lucene:grouping:test passes (15 suites, 57 tests, 0 failed, with -Ptests.asserts=true and false), and :lucene:grouping:check -x test (google-java-format, ecjLint, forbiddenApis) is clean.
• With the fixed class shadowed in the standalone runner: the finder's 2 tests (incl. the two-pass control) and the independent reproduction's 5 tests (incl. its 3 controls) pass; earlier, on a9779ef3c193, 51 upstream grouping tests in 11 classes passed.

Say in the setIncludeMaxScore javadoc that it has no influence when grouping by doc block (as setAllGroups and setAllGroupHeads do for theirs), drop or correct setIncludeMaxScore(needsScores) in the doc-block example (package-info.java:149), point to BlockGroupingCollectorManager with needsScores=true, and fix the stale BlockGroupingCollector constructor javadoc (:205-207).

Trade-offs

No runtime cost and no behaviour change, but GroupingSearch then still offers no way to get doc-block max scores with field-only sorts, and its default-on flag keeps meaning different things on its two paths.

Score on the doc-block path only when setIncludeMaxScore(true) was called explicitly (a tri-state field).

Trade-offs

Avoids A's cost change for callers who never touched the flag, but makes the default of one option mean different things on the two paths, the inconsistency the issue is about.

Method: semi-formal (the finder's tier; the decisive step was code reading of two rows of a mechanical inventory).

• Inventory. With the JDK 25 ClassFile API the finder scanned 20 non-test jars for score-related call sites: 561 rows of six kinds, 72 of them sites that pass ScoreMode.COMPLETE_NO_SCORES or TOP_DOCS to a consumer (the others are score calls, weight creations, null scorers, scorer-derived modes and conditional modes).
• Triage. Two of those rows, GroupingSearch.groupByDocBlock:224 (the COMPLETE_NO_SCORES weight for the end-of-block query) and BlockGroupingCollector.scoreMode:506 (COMPLETE_NO_SCORES when needsScores is false), led to reading groupByDocBlock, whose needsScores argument (:231) ignores includeMaxScore, unlike the two-pass path. The unit's dynamic sweep could not see it, because no score is ever requested there.
• Repro. A public-API JUnit with two blocks, field sorts, a plain-search oracle and a two-pass control (found 2026-10-28); this doc's one-document test is a further reduction.