Grouping by a function such as sum(a, b) splits documents without a value into several null groups

The finding in 30 seconds

A group key is a MutableValueFloat: a value plus an exists flag. Its contract says a missing value must be 0, and its equality compares the value. Lucene's generic value filler sets the two separately, so sum(a, b) with b absent yields keys {missing, 5.0} and {missing, 7.0}: two different null groups.

Background: grouping by a function

GroupingSearch(ValueSource, Map) groups the hits of a query by the value of a function, for example sum(a, b) over two doc-values fields.
For each hit, a value filler writes the function's value and an exists flag into a reusable MutableValueFloat; that object is the group key.
Hits for which the value does not exist belong in one null group, as when you group by a plain field they lack.

Diagram of the bug: two documents, a = 5 and a = 7, neither has b. sum(a, b) reports exists = false but floatVal 5.0 and 7.0; the value filler copies both into the group key, giving {exists=false, value=5.0} and {exists=false, value=7.0}. MutableValueFloat's equality compares the value, so the group map holds two keys and the search returns two null groups with 1 hit each, where one null group with 2 hits is expected. — The repro's two documents go through the value filler: each becomes a null group key that still carries its partial sum, 5.0 or 7.0, so grouping returns two null groups instead of one; with the fix both keys are {missing, 0} and form one group. Select the still to open it at full size. Generated with AI, checked against the code.

The finding, step by step

Index two documents with a float doc-values field a = 5 and a = 7 and no field b. Group all documents with GroupingSearch by the function sum(a, b).
You expect one null group with 2 hits: sum(a, b) does not exist for either document. Grouping the same documents by the missing field b alone gives exactly that.
You get 2 groups, both null, 1 hit each, and no error. Behind each null is the partial sum: sum says it does not exist but still returns 5 + 0 = 5 (and 7), and the value filler copies both the flag and that value into the group key.
The key class, MutableValueFloat, requires the value to be 0 when it does not exist, and its equality compares the value, so 5.0 and 7.0 make two keys. With assertions on, the search throws AssertionError; since 10.4.0, asking for 1 group throws NullPointerException.

Who is affected

Applications that group by a function (GroupingSearch(ValueSource, Map) or ValueSourceGroupSelector) over fields some documents lack, such as sum(a, b). Silent: the group count is wrong and the null groups cannot be told apart; since 10.4.0 a small group limit can fail the search. Default settings, two documents, but an opt-in, rarely used API: priority Low.

Mechanism, terms and numbers508 words

GroupingSearch(ValueSource, Map) groups hits by the value of a function. A document for which the function does not exist should land in the one null group, as it does when you group by a plain field it lacks. With sum(a, b), a document that has a but no b instead keeps its partial sum: sum's exists() is false while floatVal() returns a + 0, and the generic value filler (FloatDocValues.java:95-98) copies both into the group key. MutableValueFloat documents that a missing value must be 0 "for proper operation" and includes the value in equals/hashCode, so two documents with a = 5 and a = 7 give two null groups. Silent with assertions off; AssertionError with them on; since 10.4.0, a group limit of 1 throws NullPointerException in the second pass. Every release from 5.0.0 to 10.6.0 and main.

Terms

FunctionValues: the per-segment values of a function (ValueSource). floatVal(doc) gives the value, exists(doc) whether the document has one.
Value filler: FunctionValues.getValueFiller() returns an object that copies a document's value into a reusable MutableValue. ValueSourceGroupSelector uses it to build the group keys.
MutableValueFloat: a float value plus an exists flag. Its class javadoc: "the caller is responsible for ensuring that any instance where exists is set to false must also value set to 0.0F for proper operation" (MutableValueFloat.java:19-23), asserted in toObject, equalsSameType, compareSameType and hashCode (:29, :50, :57, :67).

The bug

sum does not exist, but has a value. MultiFloatFunction.exists is MultiFunction.allExists, false when any input is missing (MultiFloatFunction.java:52-53, :87-90). Its floatVal is func(doc, valsArr) (:83-85), for sum the sum of the inputs' floatVal (SumFloatFunction.java:35-41), and a missing FloatFieldSource contributes 0f (FloatFieldSource.java:59-64). For a = 5 and no b: exists = false, floatVal = 5.0.
The filler copies both. FloatDocValues.getValueFiller().fillValue runs mval.value = floatVal(doc); mval.exists = exists(doc); (FloatDocValues.java:95-98), so the key is {exists=false, value=5.0}; for a = 7, {exists=false, value=7.0}.
Two keys, two null groups. equalsSameType compares value (MutableValueFloat.java:49-53) and hashCode hashes it (:66-69), so the first pass's group map (FirstPassGroupingCollector.java:228, :256) holds two keys. Both groups have toObject() == null and print (null), so the caller cannot tell them apart. With assertions on, the hashCode assert fails first.
The NullPointerException (since 10.4.0). #15167 made ValueSourceGroupSelector.setGroups fold every missing top group into one includeEmpty flag (ValueSourceGroupSelector.java:87-97), and advanceTo accepts every missing value when it is set (:63-68). With a group limit of 1, the top group is {false, 5.0}; the document with {false, 7.0} is accepted too, and GroupReducer.collect finds no group for it (GroupReducer.java:71-74). Before 10.4.0 the second pass skipped it and totalGroupedHitCount came out 1 instead of 2.

The Double, Int, Long and Bool fillers have the same two lines (DoubleDocValues.java:160-163, IntDocValues.java:130-133, LongDocValues.java:139-142, BoolDocValues.java:95-98); read, not executed.

Exposure and likelihood245 words

Who. Applications that group with GroupingSearch(ValueSource, Map) or ValueSourceGroupSelector by a function that can be missing while its value is not zero and differs between documents. Executed: sum over optional float fields and over optional int doc-values fields, both through FloatDocValues' filler (sum is a float function). Not affected (read): plain numeric field sources (missing value 0), string sources (they clear the value to length 0, as MutableValueStr asks) and product with a missing input (0). A one-input function such as linear(a, m, c) gives the same c for every missing document: one null group, so it trips only the asserts.
Effect. With assertions off, no error: one null group per distinct partial value, each holding a slice of the documents that belong together, and inflated group counts (adding two documents with both fields gives 4 groups instead of 3). Since 10.4.0, a group limit below the number of distinct partial values makes the second pass throw NullPointerException; from 5.0.0 to 10.3.2 the same search silently under-counts totalGroupedHitCount.
Default vs opt-in. Default index settings, two documents, no scale. The API is opt-in: grouping by a function (GroupingSearch is @lucene.experimental), and open #10409 proposes deprecating grouping by ValueSource.
Likelihood: rare. It needs a sum-style function over fields that some matching documents lack. The code has behaved this way since LUCENE-5961 (2014; first released in 5.0.0, February 2015), and the documented searches found no public report of it. Priority Low.

Minimal repro

One JUnit test, one method, public API: two documents, GroupingSearch by sum(a, b), assert one group. Single-threaded; it failed with every seed run, and fails with assertions off (2 groups) and on (AssertionError).

TestGroupByFunctionMissingValues.java (lucene/grouping, package org.apache.lucene.search.grouping, 1 test)56 lines

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.search.grouping;

import java.util.HashMap;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.FloatDocValuesField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.queries.function.ValueSource;
import org.apache.lucene.queries.function.valuesource.FloatFieldSource;
import org.apache.lucene.queries.function.valuesource.SumFloatFunction;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.MatchAllDocsQuery;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.util.mutable.MutableValue;

public class TestGroupByFunctionMissingValues extends LuceneTestCase {

  public void testMissingFunctionValuesFormOneGroup() throws Exception {
    try (Directory dir = newDirectory();
        IndexWriter w = new IndexWriter(dir, new IndexWriterConfig())) {
      for (float a : new float[] {5, 7}) {
        Document doc = new Document();
        doc.add(new FloatDocValuesField("a", a));
        w.addDocument(doc);
      }
      try (DirectoryReader reader = DirectoryReader.open(w)) {
        ValueSource sum =
            new SumFloatFunction(
                new ValueSource[] {new FloatFieldSource("a"), new FloatFieldSource("b")});
        TopGroups<MutableValue> groups =
            new GroupingSearch(sum, new HashMap<>())
                .search(new IndexSearcher(reader), MatchAllDocsQuery.INSTANCE, 0, 10);
        // expected one null group (b is missing); actual: two groups (-ea: AssertionError)
        assertEquals(1, groups.groups.length);
      }
    }
  }
}

Command

cp $REPRO/TestGroupByFunctionMissingValues.java $LUCENE/lucene/grouping/src/test/org/apache/lucene/search/grouping/
cd $LUCENE && ./gradlew :lucene:grouping:test --tests TestGroupByFunctionMissingValues -Ptests.seed=DEADBEEF -Ptests.asserts=false
# with -Ptests.asserts=true: AssertionError at MutableValueFloat.hashCode instead (unset, the seed picks: ~75% on)

Expected: One group, the null group, with 2 hits.
Observed: java.lang.AssertionError: expected:<1> but was:<2> with assertions off; AssertionError at MutableValueFloat.hashCode(MutableValueFloat.java:67) with them on.

All files of this repro: repro/LFV-1126/

More files, environment, full output180 words

TestGroupByFunctionMissingValues (one method, extends LuceneTestCase, package org.apache.lucene.search.grouping of the lucene/grouping tests). It indexes two documents with FloatDocValuesField("a", 5) and ("a", 7) and no b with a default IndexWriterConfig, groups MatchAllDocsQuery.INSTANCE by SumFloatFunction(FloatFieldSource("a"), FloatFieldSource("b")) with GroupingSearch(ValueSource, Map), and asserts one group. It passes :lucene:grouping:checkGoogleJavaFormat as is.

It uses MatchAllDocsQuery.INSTANCE, so it compiles from 10.4.0 on; for older branches use new MatchAllDocsQuery(). As a regression test, the same test fits next to TestGroupingSearch.

Environment

lucene	`main 5adff78bb168 (12.0.0-SNAPSHOT); the 10.4.0 and 10.6.0 release jars fail the same way`
jdk	`OpenJDK 25.0.3 (build 25.0.3+9-LTS)`
os	`Linux aarch64 (nothing platform-specific)`

Command (environment)

cp $REPRO/TestGroupByFunctionMissingValues.java $LUCENE/lucene/grouping/src/test/org/apache/lucene/search/grouping/
cd $LUCENE && ./gradlew :lucene:grouping:test --tests TestGroupByFunctionMissingValues -Ptests.seed=DEADBEEF -Ptests.asserts=false
# with -Ptests.asserts=true: AssertionError at MutableValueFloat.hashCode instead (unset, the seed picks: ~75% on)

Observed output

# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF -Ptests.asserts=false
TestGroupByFunctionMissingValues > testMissingFunctionValuesFormOneGroup FAILED
    java.lang.AssertionError: expected:<1> but was:<2>

# the same with -Ptests.asserts=true (stack trace abridged: ... marks left-out frames)
TestGroupByFunctionMissingValues > testMissingFunctionValuesFormOneGroup FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:128BC027DAD23259]:0)
        at org.apache.lucene.core@12.0.0-SNAPSHOT/org.apache.lucene.util.mutable.MutableValueFloat.hashCode(MutableValueFloat.java:67)
        at java.base/java.util.HashMap.hash(HashMap.java:338)
        at java.base/java.util.HashMap.put(HashMap.java:619)
        at org.apache.lucene.search.grouping.FirstPassGroupingCollector.collectNewGroup(FirstPassGroupingCollector.java:256)
        at org.apache.lucene.search.grouping.FirstPassGroupingCollector.collect(FirstPassGroupingCollector.java:231)
        ...
        at org.apache.lucene.search.grouping.GroupingSearch.search(GroupingSearch.java:131)

Why it looks unintended

MutableValueFloat's javadoc: a value whose exists is false must be 0 "for proper operation", checked by four asserts.
The issue that added the rule (LUCENE-5790) put that duty on the value setters; here the setter is Lucene's own filler.
Grouping's model is one null group: grouping by the missing field b alone gives exactly one.

The full argument190 words

Contract. "any instance where exists is set to false must also value set to 0.0F for proper operation" (MutableValueFloat.java:19-23), asserted at :29, :50, :57 and :67. It was added in LUCENE-5790 (#6852, 4.10.0), where Yonik Seeley wrote "Value setters currently ensure that the default value (0 for numerics) is set if exists == false" and Chris Hostetter documented that duty and added the asserts. The setter here is Lucene's filler, not user code.
Grouping convention. FirstPassGroupingCollector's javadoc speaks of "putting them in a null group" (:87-88); GroupDocs.groupValue "may be null if hits did not have the groupField" (GroupDocs.java:25-26); #15167 keeps one includeEmpty flag for all missing values; grouping the repro's documents by FloatFieldSource("b") gives one null group with 2 hits.
History. The filler lines date from 2011 (Solr code that LUCENE-3249 moved into the queries module). LUCENE-5961 (#7023, 5.0.0) later made sum's exists() false when an input is missing, while floatVal() kept the partial sum. Its discussion covers exists() only; fillers and grouping are not mentioned. The available LUCENE-5961 record does not show this interaction being considered or decided.

The devil's-advocate check122 words

The strongest case for "intended": LUCENE-5961 deliberately changed exists() to false while leaving the nonzero floatVal() computation unchanged; MutableValue is @lucene.internal, GroupingSearch and the value filler are @lucene.experimental; open #10409 wants to deprecate grouping by ValueSource.

Why it fails: LUCENE-5961 changed exists() only, and the contract it collided with puts the zeroing on the setter, which is Lucene's code. Experimental tags allow API changes, not wrong results. #10409 is an open proposal: on main and 10.6.0 nothing here is deprecated, and the grouping package docs list ValueSourceGroupSelector as a selector. This argues for a low priority, not for "works as designed"; the intended-behaviour check rebutted it point by point, the cross-model reviewer rated it weak, and the adjudication agreed.

Suggested fix

Recommended Zero the value in the generic fillers when the function does not exist (built; validated)

In Lucene's generic value fillers, set the value to zero when the function does not exist: three lines each in the Float, Double, Int, Long and Bool fillers. Tested with gradle: the repro passes with assertions on and off, the grouping module and the function tests pass, and the 1-group search no longer throws.

// FloatDocValues.getValueFiller(): fillValue (the same in Double/Int/Long/BoolDocValues)
mval.value = floatVal(doc);
mval.exists = exists(doc);
if (mval.exists == false) {
  mval.value = 0.0f; // a missing value must be canonical (see MutableValueFloat)
}

All options and trade-offs3 options · 504 words

Three options; the first was built and tested, the other two were not. The choice is the maintainers'.

Zero the value in the generic fillers when the function does not exist (built; validated) (recommended)

After the two existing assignments in fillValue, reset the value to the type's zero when exists is false, in FloatDocValues, DoubleDocValues, IntDocValues, LongDocValues and BoolDocValues (+15 lines, nothing else). The call order is kept, so sources that advance an iterator in exists() (such as FloatFieldSource) see the same calls as today; the adjudication's first sketch set exists first instead.

// FloatDocValues.getValueFiller(): fillValue (the same in Double/Int/Long/BoolDocValues)
mval.value = floatVal(doc);
mval.exists = exists(doc);
if (mval.exists == false) {
  mval.value = 0.0f; // a missing value must be canonical (see MutableValueFloat)
}

Trade-offs

The payload of a missing value changes to 0, which the contract already requires. So everything that looks at it changes with it: equals, hashCode and compareSameType, hence maps, sets and any other consumer of these fillers. Anything that orders these values then puts a missing value just before a real 0 (compareSameType breaks the tie on exists), not at its old partial sum.
Only the FloatDocValues change is exercised by a test (the independent int-field test also goes through it, since sum is a float function); the Double, Int, Long and Bool edits are the same three lines, checked by reading.
Covers these five fillers only. QueryValueSource has its own filler, which writes its defVal for a missing value: one constant value, so it trips only the asserts and forms one null group; that case is not part of this finding. The string fillers (StrDocValues, DocTermsIndexDocValues) and the base FunctionValues filler, which never sets exists and which if and def use, are separate and unchanged.

What the checks validated

Gradle in a scratch worktree at 5adff78bb168, JDK 25, seed DEADBEEF: the repro fails without the fix and passes with it, assertions on and off.
With the fix: the whole lucene/grouping test module (57 tests) and org.apache.lucene.queries.function.* (111 tests, 1 skipped) pass under a random root seed; :lucene:queries:check and :lucene:grouping:check without tests (format, ecj lint, forbidden APIs) pass.
An independent group-limit-1 test (asserts 1 group and totalGroupedHitCount 2) passes with the fix and throws the NullPointerException without it: once all missing values are equal, the single includeEmpty flag is right.

Canonicalise in ValueSourceGroupSelector.advanceTo (not built)

Reset a missing key before grouping uses it.

Trade-offs

Fixes grouping only. advanceTo sees a generic MutableValue, so it needs a type-specific reset, and other users of the fillers stay exposed.

Make MutableValue* equality ignore the value when exists is false (not built)

Change equalsSameType, hashCode and compareSameType of the five primitive-value classes in core (MutableValueFloat, -Double, -Int, -Long, -Bool), and drop the asserts.

Trade-offs

Also covers any other non-canonical setter, but changes core classes and reverses the LUCENE-5790 decision: Chris Hostetter offered this alternative there and kept the zero-when-missing rule instead.

Downstream note

A service that lets its users pick a group function over sparse fields may show them split null groups, or fail their request with a NullPointerException.

How it was found

A proof-obligation audit of MutableValueFloat's contract (a missing value must be 0) traced the code that sets such values and found the generic value fillers copying a nonzero partial value. A one-test JUnit on the public grouping API witnessed it.

The method in full93 words

Method: semi-formal (a contract audit with an executable witness; no model checker or proof).

Audit. The obligation "exists == false implies value == 0" from MutableValueFloat's javadoc and asserts was traced to its setters. The generic fillers set value and exists independently, so the obligation holds only if every function's floatVal is 0 whenever its exists is false. sum (and other functions since LUCENE-5961) breaks that.
Witness. The one-test JUnit above, on the stock main jars, with assertions on and off; then independent reproductions, release runs and the fix check.

Prior reports, history and checks

Prior reports

No prior public report found by the documented searches (2026-11-02, re-checked): GitHub issues and PRs with the JIRA archive, commits, mailing lists, downstream repositories and the web. Nearest items are background only: LUCENE-5790 (the rule), LUCENE-5961 (the trigger), #15167 (the throwing path) and #10409 (deprecation proposal).

Searches5 searches

Background, not reports: #6852 (LUCENE-5790, set the zero-when-missing rule; fixed a different compareTo defect), #7023 (LUCENE-5961, created the trigger; discusses exists() only), #15167 (added the path that throws), open #10409 (LUCENE-9369, proposes deprecating grouping by ValueSource) and open PR #16548 (guards a null leafCollector in GroupReducer.collect; the groups.get(value) NullPointerException remains).

What was searched, when, and what turned up
Where	When	How	Found
GitHub issues and PRs (apache/lucene, including the migrated JIRA archive); apache/lucene-solr, apache/solr	2026-11-02	16 issue and PR queries through the scripted prior-report search plus manual searches: getValueFiller, MutableValueFloat, ValueSourceGroupSelector, grouping null group function, exists() of function values	no report; background items only
Commits and open PRs	2026-11-02	3 commit queries; the changed files of all 408 open PRs and of the 32 PRs updated since 2026-10-31	no PR touches the fillers, MultiFloatFunction, util.mutable or ValueSourceGroupSelector; #16548 touches GroupReducer.collect but leaves this NullPointerException
JIRA archive and mailing lists	2026-11-02	4 JIRA-archive queries, JIRA full-text over SOLR and LUCENE; 20 queries over dev@, java-user@, issues@, commits@, plus solr-user@ and users@solr	no report
Downstream and the web	2026-11-02	Elasticsearch, OpenSearch, Solr; 7 web queries (one more throttled and not retried)	nothing on point
Re-check	2026-11-02	the same class names on apache/lucene; items updated since 2026-11-01 mentioning grouping or ValueSource	nothing new

History

Introduced in 5.0.0 by LUCENE-5961 (595587963a7, 2014), which made sum report exists() == false while keeping its partial value; the older filler then started writing nonzero missing values. The NullPointerException since 10.4.0 (#15167). Fails on every release run from 5.0.0 to 10.6.0 and on main; 4.10.4 passes.

Versions tested6 rows

Introduced by LUCENE-5961 (#7023, Chris Hostetter, 2014-10-16): 595587963a7 on branch_5x, first released in 5.0.0, and d8146460f0e on trunk; SVN commits, no GitHub PR. It made sum/product (MultiFloatFunction) and pow/div (DualFloatFunction) report exists() == false when an input is missing. Two older pieces met there: the filler lines (0742b30b724, 2011-04-15, Solr; moved into the queries module by LUCENE-3249, a993ae90323) and the zero-when-missing rule with its asserts (LUCENE-5790: 474d846bb5b on trunk, d7bc5944845 on branch_4x, first released in 4.10.0).

The NullPointerException comes from #15167 (a4ebe984087 on main, 898d38a4fd3 on branch_10x, gaobinlong, 2025-10-06, first released in 10.4.0), which made the second pass accept every missing value. Before that, the second pass skipped the extra missing values and under-counted.

Not fixed upstream: the newest upstream commits on the five files involved predate 5adff78bb168 (read 2026-11-02). A tag sweep finds the same code in all 99 release tags from 5.0.0 to 10.6.0.

Versions and branches executed, with results
Version or commit	Verdict	Result
main `5adff78bb168` (12.0.0-SNAPSHOT)	affected	fails: 2 groups instead of 1 (assertions off), `AssertionError` (on); group limit 1: `NullPointerException`
10.6.0, 10.5.2, 10.4.0	affected	two null groups; group limit 1: `NullPointerException` (run on 10.4.0 and 10.6.0)
10.3.2, 9.12.3, 8.11.4, 7.7.3, 6.6.6, 5.5.5	affected	two null groups; group limit 1: `totalGroupedHitCount` 1 instead of 2, no error
5.0.0	first affected	two null groups `{false, 5.0}` and `{false, 7.0}`; `AssertionError` with assertions on
5.0.0 with the filler fix	fix check	`main()` probe passes: one null group, `totalGroupedHitCount` 2
4.10.4	last release before	not affected (`main()` probe; the JUnit test does not compile there): `sum(a, b)` still reported `exists() == true`, so the documents form ordinary non-null groups 5.0 and 7.0

Independent checks

An independent reproducer, a reviewer from another model family (OpenAI GPT-5.6), an intended-behaviour devil's advocate, an adjudication, a credibility audit and a final repro check all confirmed it; this doc adds gradle runs of the repro without and with the fix, the grouping and function test suites, and the group-limit-1 check.

Checks and verdicts285 words

This doc adds gradle runs in a scratch worktree at 5adff78bb168: the one-test repro without and with the recommended fix (assertions on and off), the whole grouping test module and the queries function tests with the fix, the queries and grouping static checks, and an independent group-limit-1 test without and with the fix.

Each independent check and its verdict
Check	Verdict	What it found
finder	reproduced	The one-test JUnit fails on main (assertions on and off) and on the 10.6.0 jars; the finder's cross-model check (OpenAI GPT-5.6, executing) rated it real.
Independent reproducer	real	Own test (`IntFieldSource`, randomized `IndexWriterConfig`) fails on main, 10.6.0, 10.5.2 and 9.12.3; group-limit-1 probe: `NullPointerException` on 10.4.0 and later, under-count on 10.3.2.
cross-model refute review	real, high	OpenAI GPT-5.6, executing: own test gives 3 groups instead of 2; its control (both inputs missing, partial sum 0) passes; the `FloatDocValues` fix passes it in both modes.
intended-behaviour devil's advocate	real	Grouping by the missing field `b` alone gives one null group; LUCENE-5961's design, the internal and experimental tags and #10409 examined and rejected.
adjudication	confirmed, high, Low	Re-ran the test on clean main (fail, assertions on and off) and the independent reproduction's test (fail); the fix shadow passes. LFV-427 is related (NaN keys), with another fix site.
separate re-run, credibility audit and final repro check	confirmed	Fails on main with assertions on and off; one test, two documents; passes with the fix.
this doc's fix validation	fix validated	Gradle: repro fails on stock and passes with fix A (on and off); grouping 57 tests and function 111 tests pass; static checks pass; group-limit-1 test passes with the fix, throws without it.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

PNG 545 KB

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,592 words

Finding record

LFV-1126, snapshot of 2026-11-06

Registry title: Grouping by a function such as sum(a,b) puts documents without a value into several null groups, one per partial value, because the function value fillers keep the partial value when exists is false (AssertionError with -ea; second-pass NullPointerException since 10.4.0)
Area: grouping / function grouping (queries function value fillers: FloatDocValues and siblings; contract in core util.mutable MutableValueFloat)
Priority: Low, rank 328: Grouping by a ValueSource function such as sum(a,b) silently splits documents without a value into several null groups in every release since 5.0.0, and since 10.4.0 a small group limit can make the second pass throw NullPointerException. It needs the rarely used, experimental function-grouping path over fields that some documents lack, so it is rare.
Impact class: silent wrong results
Impact note: Only applications that group with GroupingSearch(ValueSource, Map) or ValueSourceGroupSelector by a function whose exists() can be false while its value is not zero (executed for SumFloatFunction over optional float doc-values fields) are hit: documents for which the function does not exist land in one '(null)' group per distinct partial value instead of one null group, silently with -da and with default index settings and no scale [exec]; since 10.4.0 a group limit below the number of distinct missing payloads makes the second pass throw NullPointerException at GroupReducer.collect, and with -ea MutableValueFloat.hashCode throws AssertionError [exec]; the Double/Int/Long/Bool fillers follow the same pattern [read, not executed].
Summary: You index two documents with a float doc-values field a = 5 and a = 7 and no field b, and group all documents with GroupingSearch by the function sum(a, b). You expect one null group holding both documents, because sum(a, b) does not exist for either of them and the grouping javadocs put documents without a group value 'in a null group'. You get two groups that both read as null, one per partial sum, because FloatDocValues' value filler stores the partial sum as the value of a MutableValueFloat whose exists flag is false, against MutableValueFloat's documented rule that a missing value must be 0; with -ea the search throws AssertionError in MutableValueFloat.hashCode, and since 10.4.0 a group limit of 1 makes the second pass throw NullPointerException.
Introduced: 595587963a77b4899b716b664c3cac8a5a13d9de (branch_5x merge, svn r1632432; first released in 5.0.0) and the trunk commit d8146460f0eca4641f0e48793b6053a88be5c7d1 (svn r1632414; on that line from 6.0.0), both for LUCENE-5961 'FunctionValues.exist(int) isn't returning false in cases where it should for many math based value sources' (Chris M. Hostetter, hossman; migrated as apache/lucene#7023; SVN commits, no GitHub PR). It made sum/product (MultiFloatFunction) and pow/div (DualFloatFunction) report exists() false when an input is missing, while floatVal() keeps returning the partial result; this made the bug reachable. The filler lines that copy value and exists independently are older (0742b30b724, 2011-04-15, Solr; moved into the Lucene queries module by LUCENE-3240/LUCENE-3249 in 2011), and the zero-when-missing rule and its asserts come from LUCENE-5790 (474d846bb5b, 4.10.0) [read: git history and blame; exec: the history probe gives two ordinary groups on 4.10.4 and two null groups on 5.0.0]. The second-pass NullPointerException is a later consequence of the same root cause, via #15167 (a4ebe984087 on main, 898d38a4fd3 on branch_10x, gaobinlong, author date 2025-10-06, first released in 10.4.0) [exec: 10.3.2 under-counts, 10.4.0 throws]. (2014-10-16 (author dates of d8146460f0e, 19:05Z, and 595587963a7, 20:25Z); the NullPointerException path 2025-10-06 (#15167))
First affected version: 5.0.0 [exec: the history probe shows two null groups {exists=false, 5.0} and {exists=false, 7.0} on 5.0.0, and two ordinary non-null groups 5.0 and 7.0 on 4.10.4, where sum(a,b) still reported exists=true]; the second-pass NullPointerException from 10.4.0 [exec]
Affected versions: every release from 5.0.0 through 10.6.0 (released 2026-10-29, 4ae946e5a9b) and main 5adff78bb168 (12.0.0-SNAPSHOT) [read: tag sweep, all 99 release tags 5.0.0-10.6.0 in the history clone contain MultiFunction.allExists in MultiFloatFunction.exists() and the independent FloatDocValues filler]. Executed with -da (documents without a value split into one null group per partial sum): 5.0.0, 5.5.5, 6.6.6, 7.7.3, 8.11.4, 9.12.3, 10.3.2, 10.4.0, 10.5.2, 10.6.0 and main; with -ea (AssertionError at MutableValueFloat.hashCode): 5.0.0, 9.12.3, 10.5.2, 10.6.0 and main. With a group limit of 1 the second pass throws NullPointerException at GroupReducer.collect on 10.4.0, 10.6.0 and main; from 5.0.0 to 10.3.2 the same search instead silently reports totalGroupedHitCount 1 instead of 2 [exec]. Not affected: 4.10.4 and earlier 4.x, where sum(a,b) reported exists=true (4.10.4 executed with -ea and -da). Not new in 10.6 (10.5.2 fails). The one-test JUnit compiles from 10.4.0 on (it uses MatchAllDocsQuery.INSTANCE); older versions ran a main() probe of the same search or the independent reproduction's test. Not fixed upstream: the newest upstream commits on FloatDocValues.java (8ef6a0da568, 2020-12-28), MultiFloatFunction.java (54283150a81, 2026-06-14), MutableValueFloat.java (2d6ad2fee6d, 2020-12-23), ValueSourceGroupSelector.java (a4ebe984087, 2025-10-06) and GroupReducer.java (2cbf261032d, 2021-01-05) all predate 5adff78bb168 (author dates, read through the GitHub API at 2026-11-02; upstream main was 1c2072f0284). Executed on Linux aarch64 with JDK 25.0.3 only.
Status: documented (brief, checks and a per-finding doc exist)
Verification: Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by: semi-formal: proof-obligation audit
Upstream and prior reports: none found (2026-11-02, on top of the finder's, the independent reproduction's, the intended-behaviour check's and the adjudication's GitHub searches; rechecked the same day): a scripted prior-report search over 16 GitHub issue and PR queries on apache/lucene, 3 commit queries, 2 downstream-repository queries (Elasticsearch, OpenSearch, Solr), 20 mailing-list queries (dev@, java-user@, issues@, commits@) and 4 JIRA-archive queries (997 distinct hits, none with high and 4 with medium overlap, all read), plus manual GitHub searches on apache/lucene, apache/lucene-solr and apache/solr, full reads of #6852, #7023, #9951, #9151, #15167, #10409, #11570, #7778, #7763, #5865, #10060, #14855 and PRs #840 and #847, JIRA full-text queries over SOLR and LUCENE, list searches of solr-user@, users@solr and java-user@, the file lists of all 408 open apache/lucene PRs and of the 32 PRs updated since 2026-10-31 (none touches the fillers, MultiFloatFunction, QueryValueSource, util.mutable or ValueSourceGroupSelector; open PR #16548 touches GroupReducer.collect but only guards a null leafCollector, so the groups.get(value) NullPointerException remains) and 7 web queries: no report of the fillers writing a nonzero value into a missing MutableValue, of grouping by a function splitting the null group, or of the second-pass NullPointerException. Same-day recheck: GitHub search on apache/lucene for 'getValueFiller' (#847, #840), 'MutableValueFloat' (#4305, #9151, #3957), 'ValueSourceGroupSelector' (#9162, #10409, #14855, #10368, #15167, #9152, #8752, #10388), 'grouping null group function' (11 hits, none on point), and items updated since 2026-11-01 mentioning grouping (only #16847, unrelated) or ValueSource (0): nothing new. Related, not reports: #6852 / LUCENE-5790 (added the zero-when-missing rule and asserts; fixed a different compareTo defect), #7023 / LUCENE-5961 (the exists() change that created the trigger; never mentions fillers or grouping), #15167 (added includeEmpty, the path that throws), #10409 / LUCENE-9369 (open proposal to deprecate grouping by ValueSource) and #16548 (open, see above).
Lucene commit tested: main 5adff78bb168 (12.0.0-SNAPSHOT): the one-test repro with gradle in a scratch worktree, without and with the recommended fix (2026-11-03 UTC), and with the standalone runner (2026-11-02); the release jars 5.0.0, 5.5.5, 6.6.6, 7.7.3, 8.11.4, 9.12.3, 10.3.2, 10.4.0, 10.5.2 and 10.6.0 (fail) and 4.10.4 (not affected: the main() probe shows ordinary non-null groups), through the repro, an equivalent main() probe or an independent test.

Lucene and upstream sources

Finding record: lucene/queries/src/java/org/apache/lucene/queries/function/docvalues/FloatDocValues.java:84-100 (at 5adff78bb168) (getValueFiller; :95-98 fillValue: mval.value = floatVal(doc); mval.exists = exists(doc), set independently; fix site)
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/MultiFloatFunction.java:52-53 (at 5adff78bb168) (exists = MultiFunction.allExists), :80-90 (FunctionValues: floatVal returns func(doc), the partial result; exists delegates to allExists)
lucene/core/src/java/org/apache/lucene/util/mutable/MutableValueFloat.java:19-23 (at 5adff78bb168) (class javadoc: an instance with exists false must have value 0.0F 'for proper operation'), :29, :50, :57, :67 (asserts), :50-53 equalsSameType and :66-68 hashCode include value
lucene/grouping/src/java/org/apache/lucene/search/grouping/ValueSourceGroupSelector.java:63-75 (at 5adff78bb168) (advanceTo ACCEPTs every missing payload when includeEmpty), :87-97 (setGroups folds every missing top group into one includeEmpty flag, #15167)
lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupReducer.java:71-74 (at 5adff78bb168) (collect: groups.get(value).leafCollector, NullPointerException for a missing payload that is not a top group)
lucene/grouping/src/java/org/apache/lucene/search/grouping/FirstPassGroupingCollector.java:87-88 (at 5adff78bb168) (javadoc: documents without the group field go 'in a null group'); lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupingSearch.java:44 (@lucene.experimental)
https://github.com/apache/lucene/issues/6852 (LUCENE-5790), https://github.com/apache/lucene/issues/7023 (LUCENE-5961), https://github.com/apache/lucene/pull/15167, https://github.com/apache/lucene/issues/10409 (LUCENE-9369), https://github.com/apache/lucene/pull/16548, read 2026-11-02
The finding in 30 seconds: lucene/queries/src/java/org/apache/lucene/queries/function/docvalues/FloatDocValues.java:84-100 (at 5adff78bb168) (getValueFiller; :95-98 fillValue sets value and exists independently; fix site)
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/MultiFloatFunction.java:52-53 (at 5adff78bb168) (exists = allExists), :80-90 (floatVal = func, exists delegates); valuesource/SumFloatFunction.java:35-41; valuesource/FloatFieldSource.java:59-64 (0f when missing)
lucene/core/src/java/org/apache/lucene/util/mutable/MutableValueFloat.java:19-23 (at 5adff78bb168) (class javadoc), :29, :50, :57, :67 (asserts), :49-53 (equalsSameType), :66-69 (hashCode)
lucene/grouping/src/java/org/apache/lucene/search/grouping/FirstPassGroupingCollector.java:213-256 (at 5adff78bb168) (collect, collectNewGroup, groupMap); ValueSourceGroupSelector.java:63-75, :87-97; GroupReducer.java:71-74
lucene/queries/src/java/org/apache/lucene/queries/function/docvalues/DoubleDocValues.java:160-163, (at 5adff78bb168) IntDocValues.java:130-133, LongDocValues.java:139-142, BoolDocValues.java:95-98 (same pattern, read)
Who is affected: lucene/grouping/src/java/org/apache/lucene/search/grouping/GroupingSearch.java:44 (at 5adff78bb168) (@lucene.experimental), :96-98 (the ValueSource constructor)
https://github.com/apache/lucene/issues/10409 (LUCENE-9369, open: deprecate grouping by ValueSource)
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/BytesRefFieldSource.java:110-114 (at 5adff78bb168) (string filler clears the value); ProductFloatFunction.java:35-38; LinearFloatFunction.java:54-60; lucene/core/src/java/org/apache/lucene/util/mutable/MutableValueStr.java:21-24
Why it looks unintended: lucene/core/src/java/org/apache/lucene/util/mutable/MutableValueFloat.java:19-23, (at 5adff78bb168) :29, :50, :57, :67
lucene/grouping/src/java/org/apache/lucene/search/grouping/FirstPassGroupingCollector.java:87-88; (at 5adff78bb168) GroupDocs.java:25-26; ValueSourceGroupSelector.java:87-97; package-info.java:65-66
lucene/core/src/java/org/apache/lucene/util/mutable/MutableValue.java:22 (at 5adff78bb168) (@lucene.internal); lucene/queries/src/java/org/apache/lucene/queries/function/FunctionValues.java:142, :153 (@lucene.experimental)
https://github.com/apache/lucene/issues/6852 (LUCENE-5790: Yonik Seeley's and Chris Hostetter's comments); https://github.com/apache/lucene/issues/7023 (LUCENE-5961); https://github.com/apache/lucene/pull/15167; https://github.com/apache/lucene/issues/10409, read 2026-11-02
Suggested fix: lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/QueryValueSource.java:198-207 (at 5adff78bb168) (its own filler writes defVal for a missing value)
https://github.com/apache/lucene/issues/6852 (LUCENE-5790: the equals/hashCode alternative offered and not chosen)
lucene/core/src/java/org/apache/lucene/util/mutable/MutableValueFloat.java:56-63 (at 5adff78bb168) (compareSameType: ties on value broken by exists); lucene/queries/src/java/org/apache/lucene/queries/function/FunctionValues.java:155-169 (base filler never sets exists); valuesource/IfFunction.java:108-111, MultiFunction.java:141-144 (if and def use it); valuesource/FloatFieldSource.java:68-79 (exists advances the iterator)
Prior reports: https://github.com/apache/lucene/issues/6852; https://github.com/apache/lucene/issues/7023; https://github.com/apache/lucene/pull/15167; https://github.com/apache/lucene/issues/10409; https://github.com/apache/lucene/pull/16548

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

TestGroupByFunctionMissingValues.java (lucene/grouping, package org.apache.lucene.search.grouping, 1 test): repro/LFV-1126/TestGroupByFunctionMissingValues.java (56 lines)

About the diagrams

The diagram shows the repro's scenario at Lucene main 5adff78bb168.

assets/LFV-1126/bug.png (545 KB)

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.