JoinUtil's single-valued term join forgets a from-document that scored exactly 0, so ScoreMode.Min and ScoreMode.Avg give too-high, order-dependent scores

JoinUtil.createJoinQuery(fromField, multipleValuesPerDocument=false, toField, fromQuery, fromSearcher, scoreMode) aggregates the scores of all from-documents that share a join value into one score for the matching to-documents. The single-valued collectors (TermsWithScoreCollector.SV and SV.Avg) keep one running score per join value and test Double.compare(existing, 0.0d) == 0 to decide "this is the first from-document for this join value". But 0 is a legal score: a document that matches only a FILTER clause next to SHOULD clauses scores 0, and so does a FunctionScoreQuery over a value that is 0 or missing. After such a document, the next from-document with the same join value is treated as the first: ScoreMode.Min overwrites the stored 0, and ScoreMode.Avg, while every earlier score of that join value was 0 (a leading run of zeros), restarts the sum and the count. With from scores 0 and 2 both modes return 2, where the ScoreMode javadoc and the multi-valued join on the same documents give 0 and 1. Which documents match does not change; only scores, and so ranking, are wrong, with legal non-negative scores always too high and never above the join value's Max score, and they depend on collection order. Max, Total and None are correct (apart from the sign of a zero), and the multi-valued, numeric and global-ordinals joins are not affected. Avg has been wrong since the scoring join was added in 4.0.0-ALPHA (2012), Min since it was added in 5.2.0 (2015), with assertions on or off. Using BytesRefHash.add's "new value" result as the first-occurrence signal fixes it (+4/-3).

• Query-time join: JoinUtil.createJoinQuery(fromField, multipleValuesPerDocument, toField, fromQuery, fromSearcher, scoreMode) runs fromQuery, collects the join values of the matching from-documents from the doc values of fromField, and returns a query that matches the documents whose toField holds one of those values. With a ScoreMode other than None, each join value also gets one score aggregated from its from-documents, and a to-document gets the score of its join value.
• Single-valued path: multipleValuesPerDocument=false reads SortedDocValues and uses TermsWithScoreCollector.SV (Min, Max, Total) or SV.Avg; true reads SortedSetDocValues and uses MV / MV.Avg (GenericTermsCollector.java:38-52, :116-130). The package example uses false for a single-valued from field (package-info.java:94).

1. One slot per join value. SV.collect adds the from-document's value to a BytesRefHash (TermsWithScoreCollector.java:120); add returns an id >= 0 exactly when the value is new (BytesRefHash.java:314-316). New Min slots are filled with +Infinity, Max slots with -Infinity (TermsWithScoreCollector.java:127-131); Total and Avg slots start at 0 (:126, :183-184).
2. The first-document test uses the score. SV.collect reads existing = scoreSums[ord] and, if Double.compare(existing, 0.0d) == 0 (:136-137), stores the current score as if it were the first (:138); otherwise it applies Total, Min (if (current < existing), :144-148) or Max. SV.Avg.collect does the same at :189-192 and resets the count to 1; the score is sum / count (:203).
3. Min: the slot starts at +Infinity, so the test does not fire for the first document. It fires once the stored minimum is exactly 0, and the next score replaces it.
4. Avg: with non-negative scores a sum of exactly 0 means every earlier score was 0, so a leading run of zeros is forgotten; a zero after a positive score is counted.
5. Max, Total: 0 is their neutral value, so overwriting it changes nothing, apart from the sign of a zero: a +0.0 then a -0.0 (legal: AssertingScorable asserts only score >= 0) leaves -0.0, where Math.max, addition and the multi-valued join give +0.0 (executed); only Float.compare, which HitQueue uses to order hits (HitQueue.java:76), tells them apart. The test is a plain if, so -da changes nothing either.
6. The multi-valued collectors in the same file have no such test: they take min/max directly against the ±Infinity fill or add (:236-250) and count every score (:277-278).

Max and Total are right in every row, and the multi-valued join gives the expected column everywhere.

• Who is hit. Callers of JoinUtil.createJoinQuery(fromField, false, toField, fromQuery, fromSearcher, ScoreMode.Min | ScoreMode.Avg) on a SORTED doc-values from field, which is what the package example uses for a single-valued field (package-info.java:94). Default codec, IndexWriterConfig and IndexSearcher; any index size; assertions on or off.
• When. The from query scores a from-document exactly 0, and that document is collected before another from-document with the same join value (for Avg, before any positive score of that join value). Lucene produces 0 on purpose: FILTER clauses "do not participate in scoring" (BooleanClause.java:35); FunctionScoreQuery returns 0 for a "missing value, negative value or NaN" (FunctionScoreQuery.java:243-251); BoostQuery accepts a boost of 0 (BoostQuery.java:40-42). For example "rank products by their worst review" with a rating that is 0 or missing. Typical text-relevance from queries, for example a TermQuery under the default BM25 similarity, do not score exactly 0 and are unaffected.
• What happens. Silent wrong scores, with legal non-negative scores always too high and never above the join value's Max score, so to-documents rank too high and a score threshold on the join lets the wrong ones through. Which documents match, deletions and FILTER / MUST_NOT restrictions are unchanged (the follow-up's probe: the match set always equalled a brute-force set). A concrete flip from the probe: product P has reviews [0, 0], Q has [3, 3, 3], and P ranks below Q. After one more review of P with rating 5, P's correct Min and Avg are 0 and 1.67 (the multi-valued join's values), but the single-valued join returns 5 for both, so P ranks above Q (3).
• Order-dependent. Collection order is segment order, then document id, so the same from-documents indexed in another order give different scores.
• Downstream (code reading, not run): Solr's score join passes multipleValuesPerDocument=true for term joins; Elasticsearch and OpenSearch has_child/has_parent use the global-ordinals variant; Yelp's nrtsearch CrossIndexQuery can reach the affected collector when a client picks a min or avg score mode on a single-valued field.
• How common. A rarely used combination; the documented searches found no report since the code was added in 2012. For its users it is deterministic.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<0.0> but was:<2.0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.junit.Assert.assertEquals(Assert.java:701)
       org.apache.lucene.search.join.TestJoinMinZeroScore.testMinKeepsZeroScore(TestJoinMinZeroScore.java:67)

• The contract. ScoreMode.java:25-26: Avg is "Parent hit's score is the average of all child scores"; :34-35: Min is "Parent hit's score is the min of all child scores".
• 0 is legal. AssertingScorable asserts score >= 0 (AssertingScorable.java:36); in the #13697 review a committer wrote "0 is a legal score (but negative values are not legal scores)".
• The two paths are meant to agree. TestJoinUtil uses the multi-valued join at random for single-valued data, "single val can be handled by multiple-vals" (TestJoinUtil.java:1535-1536), against one reference model, JoinScore, which starts Min at +Infinity and counts every score for Avg (:2046-2081). The numeric and global-ordinals joins count a 0 too (the devil's-advocate check ran all four on the same data).
• History shows a leftover marker. In LUCENE-4043 (2012) the score arrays started zeroed and 0 meant "not set yet". LUCENE-6389 (2015) added Min and pre-filled its slots with +Infinity, which only makes sense for a true minimum, but kept the 0 test in front of the switch.
• Why tests never caught it. TestJoinUtil's term-join tests use TermQuery from queries (the random ones on value, TestJoinUtil.java:1524), which are not built to score 0, and no test mixes zero and positive scores for one join value on the single-valued term path. Its only from query meant to produce zeros, testMinMaxScore's score = price with price = random().nextInt(1000) (:508-585, :539), runs the global-ordinals join (:560-562, :572-574), which counts the 0.

The strongest case for "intended": the JoinUtil javadoc says "only the score from the first encountered join value originating from the 'from' side is mapped into the 'to' side" (JoinUtil.java:68-74).

Why it fails: that sentence is about multipleValuesPerDocument=true and a document with several join values, not about adding up the from-documents of one join value, and this bug is on the false path. @lucene.experimental (JoinUtil.java:54) allows the API to change, not wrong arithmetic. "0 means it did not really match" fails too: a FILTER-only document is a match, and the zero is kept when it comes last. Weaker arguments fail too: the 2012 code may have assumed strictly positive scores, but no javadoc says so and Lucene produces exact zeros by design; ScoreMode speaks of parent and child hits because the block join shares it, while JoinUtil says it "Instructs how scores from the fromQuery are mapped to the returned query" (JoinUtil.java:86); exact zeros are rare and the code is old, which explains why nobody noticed, not that it is meant; the multi-valued path is a workaround, not a contract. The cross-model reviews and the independent reproducer rated the intended case weak; the devil's-advocate check and the adjudication found no intended argument that survives. Verdict: a defect.

Two fixes were built and validated; the recommended one is the smallest. Whichever is chosen, add a from query that scores some documents exactly 0 to the join tests (the repro as a new test class is the smaller change; fix-with-test.diff = fix A + both repro tests). Workaround until a fix lands: call createJoinQuery with multipleValuesPerDocument=true on the same SORTED field; the multi-valued path wraps it as a single-valued set (DocValuesTermsCollector.java:49-51, DocValues.java:325-337) and counts every score (executed on the repro's documents: Min 0, Avg 1). Open PR #16805 ("Collector manager join", under review) moves this code into per-slice collectors and keeps the test, so the issue can point there and the fix lands once. Not part of this bug: a from-document without a join value is mapped to the empty term (TermsWithScoreCollector.java:118, :176), a separate bug.

BytesRefHash.add returns an id >= 0 exactly when the join value is new (BytesRefHash.java:314-316). Keep that as firstOccurrence in SV.collect and SV.Avg.collect and test it instead of the score. 1 file, +4/-3.

int ord = collectedTerms.add(value);
final boolean firstOccurrence = ord >= 0;   // new
...
if (firstOccurrence) {                       // was: Double.compare(existing, 0.0d) == 0
  scoreSums[ord] = current;
  scoreCounts[ord] = 1;                      // SV.Avg only
} else {
  ...

Trade-offs

No extra work per document (a boolean instead of a double comparison); no behaviour change, for legal non-negative scores, when no from-document scores 0; the Min/Max ±Infinity fill becomes unnecessary but harmless. No benchmark was run.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: stock, the repro fails 2 of 2 and the order test 2 of 4, with assertions on and off; with the fix both pass, with assertions on and off.
• With the fix: the whole lucene/join test task, repro classes included, 18 classes, 110 tests, 3 skipped, 0 failures (the 16 existing classes: 104 tests; TestJoinUtil 16 of 16); :lucene:join:check -x test (format, ecjLint, forbiddenApis) passes with the new tests in place.
• Standalone runner with the fix shadowed: the independent reproducer's FunctionScoreQuery test 5 of 5, the devil's-advocate sibling-join comparison 4 of 4, the security probe in its strict mode (0 scores too high, against 854 of 1,698 on stock).
• fix.diff and fix-with-test.diff apply cleanly to a9779ef3c193. Not run: the full Lucene test suite, nightly mode, a benchmark.

Drop the first-occurrence branch: Total adds, Min and Max use Math.min / Math.max against the existing ±Infinity fill, Avg adds and counts. fix-b-accumulate.diff, +18/-32.

Trade-offs

SV and MV then share one logic, the one TestJoinUtil's model checks. Larger diff; Math.min and < differ only for NaN, which is not a legal score, and for the sign of a zero.

What the checks validated

Gradle on a9779ef3c193: the two repro classes and TestJoinUtil, 22 of 22 pass with assertions on. Standalone runner with it shadowed: the repro with -da, the reproducer's and the devil's-advocate tests and the strict probe pass. Not run through the whole module task or :check.

Method: code reading (the hunt's non-formal tier), by the hunt unit for the join and block-join module, on main ea57360912ed, on 2026-10-26.

• While reading the TermsWithScoreCollector accumulators for #16378 / #16428 (double-precision accumulation), the unit noticed that the single-valued collectors test the score, not BytesRefHash.add's result, for a first occurrence.
• It wrote a public-API JUnit with the multi-valued join over the same documents as an in-test reference: 2 of 2 fail on main with assertions on and off, on 10.5.2 and on 9.12.3; a fix sketch makes it pass.
• The unit's block-join reference harness does not cover JoinUtil's term join, and TestJoinUtil's random from queries are TermQuery hits that are not built to score 0.