Grouping by score ranges reads a placeholder score when a pass requests no scores

DoubleRangeGroupSelector and LongRangeGroupSelector group hits by ranges of a values source, and they hand that source the collector's scorer, so a source that reads the score (DoubleValuesSource.SCORES) can group hits into score bands. But a GroupSelector cannot say that it needs scores, and the four grouping collectors that use one decide whether the query computes scores from their sorts or reducer alone. In a pass where none of them needs scores, the query runs in COMPLETE_NO_SCORES, the selector reads a placeholder (0 for the term and boolean queries of every repro) for every hit, and the groups come out silently wrong: one empty group with a field group sort, the right groups without their documents with includeMaxScore=false, and 1 group or head instead of several from AllGroupsCollectorManager or AllGroupHeadsCollectorManager run alone. It has been there since the selectors were added in 8.6.0; the GroupingSearch defaults (relevance group sort, includeMaxScore=true) are not affected.

• Group selector (GroupSelector): decides the group of each hit. DoubleRangeGroupSelector and LongRangeGroupSelector read a value per hit from a values source and put the hit into a fixed-width range (DoubleRangeFactory(0, 1, 10): ranges of width 1 from 0 to 10).
• Score-dependent source: a DoubleValuesSource whose needsScores() is true, such as DoubleValuesSource.SCORES (the hit's score) or its toLongValuesSource(). The range selectors feed it the collector's scorer through DoubleValuesSource.fromScorer (DoubleRangeGroupSelector.java:60-63, LongRangeGroupSelector.java:61-64).
• Score mode: each collector tells the query whether it needs scores (Collector.scoreMode()). In COMPLETE_NO_SCORES, "scores won't be available" (ScoreMode.java:25-26); a non-scoring term weight, for example, uses a dummy scorer that returns 0 (TermQuery.java:77-91).
• Two-pass grouping: GroupingSearch runs a first pass (FirstPassGroupingCollector) to find the top groups, then a second pass (TopGroupsCollector, a SecondPassGroupingCollector) to collect each group's documents; AllGroupsCollector and AllGroupHeadsCollector can run alongside or on their own.

1. GroupSelector has setScorer but no needsScores() (GroupSelector.java:34-72), so a selector cannot ask for scores.
2. The four collectors that use a group selector take their score mode from their sorts or reducer only, each pass on its own: the first pass from the group sort (FirstPassGroupingCollector.java:105, 124-127), the second pass from the reducer, which needs scores only for the max score or a within-group sort that needs them (SecondPassGroupingCollector.java:76-79, TopGroupsCollector.java:127-128), AllGroupHeadsCollector from its sort (:119-122), and AllGroupsCollector never (:91-94, "the result is unaffected by relevancy").
3. So with a field group sort the first pass runs without scores. The selector reads 0 for all five hits and keeps one group, [0, 1).
4. The second pass scores, because includeMaxScore defaults to true, and the selector now reads the real scores 1, 2, 3, 1, 3. None falls into [0, 1), the only group, so advanceTo returns SKIP for every hit (DoubleRangeGroupSelector.java:75, SecondPassGroupingCollector.java:90): one empty group.
5. With the relevance group sort but includeMaxScore=false and a field within-group sort, the first pass is right and the second pass reads 0: the three right ranges come back empty. AllGroupsCollector and AllGroupHeadsCollector alone read 0 for every hit: 1 range, 1 head.

DoubleValuesSource.SCORES.needsScores() is true (DoubleValuesSource.java:349-368); the information exists, but no grouping collector asks the selector. When a sort happens to need scores, the selector is right by accident.

Not every query is hit. Without scores the value a scorer returns is unspecified (ScoreMode only says scores "won't be available"). Term and boolean queries and ConstantScoreQuery clauses return 0 (TermQuery.java:77-91, ConstantScoreQuery.java:141-146, 202-203), so they are grouped wrongly; queries whose score is a constant boost in every mode, such as MatchAllDocsQuery or a point range query (ConstantScoreWeight), still return their real score and are grouped right (executed).

Five documents with tags {a}, {a,b}, {a,b,c}, {a}, {a,b,c}; a OR b OR c of ConstantScoreQuery(TermQuery) clauses scores them 1, 2, 3, 1, 3 (a plain search, asserted by the control test). Expected for every configuration (with setGroupDocsLimit(10)): {[1,2)=[0, 3], [2,3)=[1], [3,4)=[2, 4]}. Group sort by id: {DoubleRange(0.0, 1.0)=[]} (totalHitCount 5, totalGroupedHitCount 0; with setAllGroups(true) also totalGroupCount 1). LongRangeGroupSelector over SCORES.toLongValuesSource(): {LongRange(0, 1)=[]}. includeMaxScore=false with a within-group sort by id: {[1,2)=[], [2,3)=[], [3,4)=[]}. AllGroupsCollectorManager alone: [DoubleRange(0.0, 1.0)]. AllGroupHeadsCollectorManager with sort id alone: heads [0] instead of [0, 1, 2]. An independent reproducer with 12 documents whose scores are 10.8 to 16.8 recorded what the selector reads under an id sort: 0.0 for all 12.

• Who. Grouping with DoubleRangeGroupSelector or LongRangeGroupSelector over a values source that needs scores: DoubleValuesSource.SCORES, its toLongValuesSource(), or (by reading, not run) an expression that binds _score to it, the pattern the expressions module documents (SimpleBindings.java:35), plus any one of: (a) a group sort that does not need scores; (b) includeMaxScore=false and a within-group sort that does not need scores, even with the default relevance group sort; (c) AllGroupsCollectorManager, or AllGroupHeadsCollectorManager with a sort that does not need scores, run on its own.
• Where (c) shows. On 10.6.0 (the 10.6 branch) and main. On 8.6.0-10.5.2 those stand-alone collectors did not forward the scorer, and the same search throws a NullPointerException instead (a related bug in the same selectors); #16247 (released in 10.6.0) turned that exception into these wrong groups.
• Default vs opt-in. Grouping by score ranges is opt-in; the GroupingSearch defaults (relevance group sort, includeMaxScore=true) request scores in both passes and are right. Once configured with a trigger, a search is mis-grouped whenever the placeholder falls into a different range from the real scores, as for the repro's boolean query; constant-score queries such as MatchAllDocsQuery are grouped right.
• Scale. None: five documents in one segment show it, deterministically, with assertions on and off.
• Effect. Silent wrong groups: one empty group, the right groups without documents, documents missing from their groups, or 1 group or head instead of several. Nothing fails on main with a plain IndexSearcher; totalHitCount stays right while totalGroupedHitCount drops to 0.
• How likely. Rare: grouping by the score is unusual, the defaults hide it, no use of the range selectors was found in Elasticsearch, OpenSearch or Solr, and no public report was found in the searched sources.
• Workaround. Add a score tie-break to the group sort, new Sort(new SortField("id", SortField.Type.INT), SortField.FIELD_SCORE) (executed), and keep includeMaxScore=true or a within-group sort that needs scores. For the stand-alone collectors, run them in a MultiCollectorManager next to a collector that needs scores (executed by the fact check: 3 right ranges, heads [0, 1, 2]).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[DoubleRange(1.0, 2.0), DoubleRange(2.0, 3.0)]> but was:<[DoubleRange(0.0, 1.0)]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.grouping.TestScoreRangeGroupingFieldGroupSort.testGroupByScoreRangeSortedByField(TestScoreRangeGroupingFieldGroupSort.java:75)

• Selector contract. DoubleRangeGroupSelector takes "a DoubleValuesSource to retrieve double values per document" (DoubleRangeGroupSelector.java:46; LongRangeGroupSelector.java:47 the same for longs), with no restriction; the package javadoc lists it as grouping "based on the value of a DoubleValuesSource" (package-info.java:67-70).
• The bridge serves this case. setScorer binds source.getValues(context, DoubleValuesSource.fromScorer(scorer)). DoubleValuesSource.getValues says callers "may safely pass null" for the scores when needsScores() is false (DoubleValuesSource.java:60-66), so the scorer, which the selectors pass to every source, is needed only by score-dependent sources; the class javadoc documents scores as a value source through fromScorer (:43-46).
• Score-mode contract. COMPLETE_NO_SCORES: "Produced scorers will allow visiting all matches but scores won't be available" (ScoreMode.java:25-26); Collector.scoreMode(): "Indicates what features are required from the scorer" (Collector.java:55-56).
• Unrelated settings change membership. setIncludeMaxScore is documented as "Whether to include the score of the most relevant document per group" (GroupingSearch.java:329-338); turning it off empties every group. A group sort by a field merges every hit into [0, 1).
• Convention and precedent. DoubleValuesSortField.needsScores() returns producer.needsScores() (DoubleValuesSource.java:637-640), which is why a sort by a score-dependent source works in grouping today. #14654 (CHANGES 10.3.0, Bug Fixes) fixed a missing pass-through of the same kind: "would throw errors when used if the DoubleValuesSource needed scores".
• Tests. No upstream grouping test uses a score-dependent source: TestDoubleRangeGroupSelector and TestLongRangeGroupSelector use field sources, and nothing in lucene/grouping/src/test uses SCORES or fromScorer. The case is untested, not decided.
• History. LUCENE-6912 (2015) wrote the sort-based score modes and the "unaffected by relevancy" comment, when no selector could read a scorer. LUCENE-7889 (2020) added setScorer and the range selectors with the score bridge and changed no score mode; its PR has no review comments and its discussion never mentions scores.

The strongest case for "intended" (built by the devil's-advocate checker and the other-family reviews; the reviews rate it moderate, the devil's advocate gives no rating): AllGroupsCollector states "the result is unaffected by relevancy"; GroupSelector never had a way to ask for scores, its javadoc says nothing about score availability (GroupSelector.java:25-33), and ValueSourceGroupSelector.setScorer is empty, so the module may never have meant group keys to depend on scores; the defaults work; a score tie-break is a workaround; the tests use field sources only; the collectors and GroupingSearch are marked @lucene.experimental, which leaves maintainers room to narrow the contract; and LUCENE-7889's commit message gives numeric-valued fields as the motivation.

Why it fails: the comment predates score-reading selectors by 4.5 years and was not revisited when they were added; the fromScorer bridge is needed only by score-dependent sources, so wiring it and never requesting scores looks like an oversight; no javadoc restricts the source, SCORES is public and documented, and the numeric-field motivation says why the feature was added, not what it excludes; and the groups change with unrelated settings, which no contract can intend. Verdict: most likely not intended, a defect born with the feature; a javadoc restriction remains a legitimate maintainer alternative. The fact checks read it the same way. The expected behaviour is inferred from the API's shape and the needsScores convention, not quoted from a grouping javadoc, so a maintainer may prefer to document a restriction. Not security: wrong groups through correct use of public APIs, chosen by the application.

One code fix was built and run in a scratch worktree at a9779ef3c193; the javadoc alternative was not run. Both are legitimate.

The finder's fix, formatted with gradlew tidy: a non-abstract GroupSelector#needsScores() returning false; DoubleRangeGroupSelector and LongRangeGroupSelector override it with source.needsScores(); FirstPassGroupingCollector, SecondPassGroupingCollector (so also TopGroupsCollector and DistinctValuesCollector), AllGroupsCollector and AllGroupHeadsCollector OR it into their score modes. +26/-4 in 7 files.

// GroupSelector
public boolean needsScores() {
  return false;
}
// DoubleRangeGroupSelector, LongRangeGroupSelector
@Override
public boolean needsScores() {
  return source.needsScores();
}
// FirstPassGroupingCollector
this.needsScores = groupSort.needsScores() || groupSelector.needsScores();
// SecondPassGroupingCollector (AllGroupHeadsCollector: sort.needsScores(); AllGroupsCollector: the selector alone)
return groupReducer.needsScores() || groupSelector.needsScores()
    ? ScoreMode.COMPLETE
    : ScoreMode.COMPLETE_NO_SCORES;

Trade-offs

• A new public, non-abstract method on a public abstract class: additive and source-compatible. A custom GroupSelector that reads scores must override it; say so in the javadoc and in CHANGES.
• A score-dependent selector now makes the query compute scores: a cost only for those users, and what they need for a correct result. Score-independent selectors are unchanged.
• It follows the existing convention (DoubleValuesSortField.needsScores() returns producer.needsScores()).
• It drops the "the result is unaffected by relevancy" comment, which could stay as "unless the selector reads scores".
• Out of scope (a related bug in the same selectors): the score-less cache replay (setCaching(n, false)) still cannot serve a score-dependent selector, and DistinctValuesCollector's value selector never gets setScorer.

What the checks validated

Gradle in a scratch worktree at a9779ef3c193, seed DEADBEEF: the whole :lucene:grouping:test with the 5 repro classes added, 19 classes, 82 tests, 0 failures, with assertions on and off; all 19 repro tests that fail on stock pass. :lucene:grouping:check -x test with the fix and the primary test (google-java-format, ecj lint, forbidden APIs): BUILD SUCCESSFUL. The repro runner's re-runs pass with the fix shadowed too (the finder's test 4 of 4, the reproducer's 13 of 13, the probe on main); those used the finder's untidied copy of the same change, which differs only in line wrapping. An independent fact check repeated the primary test with the fix and the static checks in a fresh scratch worktree: 7 of 7 pass, BUILD SUCCESSFUL. The brief's own scratch worktree was deleted and its HEAD was not logged. Not run: the full Lucene test suite, benchmarks.

Say in the javadoc of both range selectors that the source must not need scores unless a sort that needs scores is used, and correct the AllGroupsCollector comment. Not run.

Trade-offs

The smallest change and a legitimate maintainer choice, but a documented public input keeps giving a silent wrong answer, and the selectors' fromScorer bridge would then serve only passes whose sort or reducer already requests scores.

Throw IllegalArgumentException when source.needsScores() is true. Not run.

Trade-offs

Breaks configurations that work today (relevance sorts); not recommended.

Method: static bytecode inventory plus code reading, then a public-API test (the hunt's "semi-formal" tier; the lead came from reading rows of a mechanical inventory, which a strict reading could put in the non-formal tier).

• The hunt unit looked for weights or collectors that run without scores while something reads scores. It scanned the class files of 20 non-test jars with the JDK 25 class-file API: 561 rows, among them every site where COMPLETE_NO_SCORES or a needsScores()-conditional mode reaches a consumer and every DoubleValuesSource.fromScorer bridge. Each non-test row was triaged by reading.
• The bridge rows DoubleRangeGroupSelector.setScorer and LongRangeGroupSelector.setScorer sat next to grouping collectors whose scoreMode() is COMPLETE_NO_SCORES unless a sort or reducer needs scores. The finder wrote the public-API JUnit test, which failed 3 of 4 with assertions on and off.
• A secondary dynamic sweep ran 5,696 module compositions under the test framework's asserting searcher; its only new scorer-protocol sites were this bug's first-pass and second-pass collectors, including the second-pass-only trigger. A planted non-scoring collector that reads scores fired as a positive control.
• Independent checks then reproduced it (see the checks below).