JoinUtil's single-valued joins treat a from-document without a join value as key 0 or the empty string

JoinUtil.createJoinQuery runs a query-time join in two passes: it collects the join values of the "from" documents that match fromQuery, then returns the "to" documents whose to field holds one of those values. A from-document that matches fromQuery but has no value in the from field has nothing to join on, so it should add nothing.

With multipleValuesPerDocument=false, the setting the package docs prescribe for a single-valued from field, it does add something: the term join collects it as the empty term and the numeric join as the number 0. So every to-document keyed "" or 0 comes back as an extra hit in every ScoreMode, and when such a document is a real match its join score folds in the key-less documents' scores. The same call with multipleValuesPerDocument=true on the same single-valued field, the global-ordinals join and TestJoinUtil's own reference collector all skip the key-less document. The cause is four collect sites (three else branches and one long value = 0 initializer) that substitute a default when advanceExact(doc) is false; returning early instead (+10/-18, three files) fixes it. The term join has behaved this way since 4.0.0 (on 3.6.x, the first release with JoinUtil, the same input throws a NullPointerException instead) and the numeric join since its first release, 5.5.0; it is not a regression.

• Query-time join (JoinUtil.createJoinQuery, @lucene.experimental): "The first pass collects all the terms from a fromField that match the fromQuery. The second pass returns all documents that have matching terms in a toField to the terms collected in the first pass" (package-info.java:65-67 on main a9779ef3c193). The term overload reads the from field from SortedDocValues (or SortedSetDocValues); the numeric overload from NumericDocValues (or SortedNumericDocValues) and matches the to field with points.
• multipleValuesPerDocument: "Whether the from field has multiple terms per document" (JoinUtil.java:82); the package example says "Set only to true in the case when your fromField has multiple values per document" (package-info.java:94).
• advanceExact(doc): "return whether target has a value" (DocValuesIterator.java:25). False means the document has no value in that field.

1. Routing. With multipleValuesPerDocument=false the term overload builds GenericTermsCollector.createCollectorSV (JoinUtil.java:106-108): TermsCollector.SV for ScoreMode.None, TermsWithScoreCollector.SV or SV.Avg for the scoring modes (GenericTermsCollector.java:116-128). The numeric overload uses its own single-valued collector (JoinUtil.java:265-293).
2. The default. When advanceExact(doc) is false these collectors substitute a value instead of skipping the document. TermsCollector.SV.collect adds new BytesRef(BytesRef.EMPTY_BYTES) (TermsCollector.java:85-93); TermsWithScoreCollector.SV and SV.Avg do the same (:113-119, :171-177) and then fold the document's score into the empty term's slot. The numeric collector starts from long value = 0;, overwrites it only if advanceExact succeeds, and then adds it to the join set and, when scoring, to the score aggregator (JoinUtil.java:283-292).
3. Second pass. Every to-document whose to field holds a collected value matches, so the one keyed "" or 0 matches, and its join score aggregates the real and the key-less from-documents.
4. Everything else skips. The multi-valued term collectors (TermsCollector.java:64-74, TermsWithScoreCollector.java:218-219, :264-265), the numeric multi-valued collector (JoinUtil.java:236-246), the global-ordinals collectors (GlobalOrdinalsCollectorManager.java:69, GlobalOrdinalsWithScoreCollector.java:116, :145) and TestJoinUtil's reference collector (TestJoinUtil.java:2167-2174) do nothing for a document without a value. With multipleValuesPerDocument=true a single-valued field is read through DocValues.getSortedSet / getSortedNumeric, which wrap it as one value per document (DocValues.java:301-313, :325-337), so true gives the right answer on the same index.

One index: order o1 (customer 7), order o2 (no customer), customers c7 (7) and c0 (0); fromQuery is type:order. Numeric join, Integer or Long: true gives [c7], false gives [c0, c7] in all five ScoreModes. Term join with customers "c7" and "": true gives [c7], false gives ["", c7]. Scores: o1 points at customer 0 with score 1, o2 has no customer and scores 5; customer 0 gets Max 5.0, Avg 3.0, Total 6.0 instead of 1.0 (Min stayed 1.0 only because the extra score was higher; with an extra score of 0.5, Min is 0.5, Avg 0.75 and Total 1.5, and Max stays 1.0).

• Trigger. The term overload, or the numeric overload with Integer or Long (executed; Float and Double +0.0 only in the hunt's bounded-exhaustive harness), with multipleValuesPerDocument=false; a fromQuery that matches at least one document without the from field; and a to-document keyed 0 (numeric) or "" (term).
• Two natural shapes. An optional foreign key (an order with no customer, a comment with no author). Or a same-index join with a content query, which is the package example: multipleValuesPerDocument = false, fromQuery = new TermQuery(new Term("content", searchTerm)), and "toSearcher can be the same as the fromSearcher" (package-info.java:92-100); every matching document that is not a from-document, to-documents included, then contributes the default key.
• Effect. Every to-document keyed 0 or "" is an extra hit, in every ScoreMode (the repro has one; join keys need not be unique, and the second pass returns "all documents that have matching terms", package-info.java:65-67). When such a document is a real match, its score folds in the key-less documents' scores, so Min, Max, Avg or Total can be wrong: Max 5.0, Avg 3.0, Total 6.0 instead of 1.0 in the repro; Min 0.5, Avg 0.75, Total 1.5 with an extra score of 0.5 (executed). No error. Same index or two indexes, with and without assertions, with or without a segment that lacks the field. It is a per-document code path, so it should not depend on index size; only small indexes were run.
• Realistic half. Numeric key 0 (0-based ids or codes) is the main impact; the empty-string half needs an empty key, which is legal but rare.
• Not hit (read from source, not executed): multipleValuesPerDocument=true; the global-ordinals overloads; Solr's term score join, which always passes true; the Elasticsearch and OpenSearch parent-child joins, which use global ordinals. Solr's numeric score join since SOLR-18440 (Solr main and branch_10x, unreleased) passes the from field's multiValued() flag, so a single-valued numeric from field reaches the affected collector. The Lucene.NET 4.8 port has the term half of the same code.
• Side effect, not a separate bug (hunt probe only). TermsQuery.equals compares the fields, fromQuery and the reader id, not the collected terms (TermsQuery.java:107-124, by design since LUCENE-7810). So the false and true joins over the same field compare equal while matching different documents; with a query cache that caches the join, count() of the true join returned the false join's cached 2 instead of 1. Without a cache, or with the fix, both return 1. Main's IndexSearcher has no default query cache (IndexSearcher.java:82-84, GITHUB#14187): the probe ran under the test framework, which installs a cache that caches every segment. A plain run on the same data with no cache, or with LRUQueryCache(1000, 50MB) and the default caching policy, returns 1 (executed by the fact check).
• Reach. JoinUtil is @lucene.experimental; no public report was found in the searched sources. Likelihood: plausible.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: customers joined to the orders expected:<[7]> but was:<[7, 0]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.apache.lucene.search.join.TestNumericJoinFromDocWithoutKey.testFromDocWithoutKeyDoesNotJoinKeyZero(TestNumericJoinFromDocWithoutKey.java:60)

• The contract. JoinUtil.java:64-66: the query retrieves "all documents that have the same terms in the to field that match with documents matching the specified fromQuery and have the same terms in the from field"; the package doc: "The first pass collects all the terms from a fromField that match the fromQuery" (package-info.java:65-67). The numeric overload inherits "All considerations" of the term overload (JoinUtil.java:122-125).
• The flag is a shape hint. "Whether the from field has multiple terms per document" (JoinUtil.java:82); "Set only to true in the case when your fromField has multiple values per document" (package-info.java:94). The upstream random test picks true at random for single-valued data and expects the same results: "single val can be handled by multiple-vals" (TestJoinUtil.java:1535-1536).
• Lucene's own oracle skips missing values. TestJoinUtil's reference collector returns when advanceExact is false (TestJoinUtil.java:2167-2174). It was written to skip them from LUCENE-4043 (2012: if (joinValue == null) return;), but that check could not fire, because DocTerms.getTerm returned an empty BytesRef, never null. The skip took effect in 4.2.0 (BinaryDocValues.MISSING), then docsWithField (LUCENE-5183, 4.5.0). The random test never exercises the difference: every document gets a join value, and "empty strings aren't allowed" (:1701).
• The sibling join skips them. The global-ordinals join (LUCENE-6352, 2015, by the same original author) skipped value-less documents from its first commit.
• No rule states the default. No javadoc, CHANGES or MIGRATE entry describes missing-as-0/empty or requires the term or numeric overloads' fromQuery to match only documents with the from field. That precondition ("the fromQuery can only match \"from\" documents") is written for the global-ordinals overloads only (JoinUtil.java:467-468, :505-506).
• Origin is mechanical. The 2012 single-valued collector called FieldCache.DocTerms.getTerm(doc), which returned "an empty (length=0) BytesRef if the doc did not have this field"; the same API offered exists(docID), unused. Missing numeric doc values read as 0. The iterator migrations LUCENE-7407 and LUCENE-8152 wrote the old defaults out as explicit else branches; their threads do not discuss join semantics.
• Precedent. LUCENE-7871 (#8922), "false positive match BlockJoinSelector[SortedDV] when child value is absent", in the same module, was fixed as a bug in 7.0.

The strongest case for "intended": JoinUtil is @lucene.experimental; the else branches are explicit, so the default looks deliberate; no public report was found in 14 years of releases; the random test says "empty strings aren't allowed"; a caller can add FieldExistsQuery(fromField) to fromQuery; and multipleValuesPerDocument=false could be read as "every from-document has exactly one value". The numeric half has the stronger case: before 7.0 a missing numeric doc value read as 0, and callers may rely on a missing key joining as 0.

Why it fails: the explicit branches are a mechanical carry-over of the old doc-values defaults, not a decision (no doc, no CHANGES entry, no discussion). The test comment is about how the random test generates data, and it does not touch the numeric-0 half. No documented precondition asks callers to filter fromQuery, the package example does not, and the package doc calls fromQuery "usually the user specified query" (package-info.java:73-74). The "exactly one value" reading is contradicted by the package example and by TestJoinUtil.java:1535. For the numeric half, 0 is also a legal key, no join doc names it a sentinel, and since 7.0 advanceExact reports absence, which the multi-valued numeric collector honours on the same data (JoinUtil.java:236-246); the compatibility risk is real, which is why the fix needs a CHANGES entry. Every reviewer concluded the case fails; the reading-only reviews by another model family rated its strength moderate, weak and moderate, and the reading-only cross-model fact check of this doc rated it weak for the term half and moderate for the numeric half.

The recommended fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF); it was also validated earlier with the fixed classes shadowed into the stock jars. The documentation-only option was not run; the user workaround was checked on stock main.

In TermsCollector.SV.collect, TermsWithScoreCollector.SV.collect, TermsWithScoreCollector.SV.Avg.collect and the numeric single-valued collector in JoinUtil, return (or collect nothing) when advanceExact(doc) is false, instead of substituting new BytesRef(BytesRef.EMPTY_BYTES) or 0. This is what multipleValuesPerDocument=true, the global-ordinals join and TestJoinUtil's reference collector already do. It skips the hash insert and score work for documents without a value; performance was not measured.

// TermsCollector.SV
if (docValues.advanceExact(doc)) {
  collectorTerms.add(docValues.lookupOrd(docValues.ordValue()));
}
// TermsWithScoreCollector.SV and SV.Avg (then the score aggregation as before)
if (docValues.advanceExact(doc) == false) {
  return;
}
BytesRef value = docValues.lookupOrd(docValues.ordValue());
// JoinUtil, numeric single-valued collector
if (numericDocValues.advanceExact(doc) == false) {
  return;
}
long value = numericDocValues.longValue();

Trade-offs

• Behaviour change. Callers that relied on a missing value joining as 0 or "" (undocumented) lose those hits; say so in a CHANGES entry (Bug Fixes).
• Coordination. Open PR #16805 ("Collector manager join", active) reworks GenericTermsCollector, TermsWithScoreCollector and the numeric collector for CollectorManager and keeps long value = 0 (it does not touch TermsCollector), and #13747 moves all of them and keeps both defaults; the fix should land in or rebase onto #16805. The hunt's sibling finding about the 0.0 score sentinel in the same SV.collect methods is a different defect and fix line.
• Add a regression test with a value-less from-document and to-documents keyed 0 and ""; the repro is one.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the repro passes 3 of 3 and TestJoinUtil 16 of 16, with assertions on and off; the whole lucene/join module with the repro added: 17 classes, 107 tests, 3 skipped, 0 failures; checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass. Without the fix the same worktree fails the repro 3 of 3 with assertions on and off.
• With the fixed classes shadowed into the stock a9779ef3c193 jars: the repro and the independent reproducer's test pass 3 of 3; upstream TestJoinUtil passes 16 of 16 on seeds DEADBEEF, 1234ABCD, 0BADF00D and 7E57C0DE.
• The hunt's bounded-exhaustive harness, in the two of its 21 scopes that were re-run with the fix (one term, one numeric): the missing-value mismatch class of the single-valued joins disappears and no new class appears. The other 19 scopes were not re-run with the fix.

State in the javadoc that with multipleValuesPerDocument=false a matching from-document without a value joins as "" / 0, and tell callers to restrict fromQuery to documents with the from field.

Trade-offs

Keeps the flag changing results on single-valued data, contradicts the "same terms in the from field" contract and the global-ordinals join, and leaves the package example (a content fromQuery) wrong. Not recommended.

Pass multipleValuesPerDocument=true (correct on a single-valued field: the repro's true checks pass), or add new FieldExistsQuery(fromField) as a FILTER clause to fromQuery: on stock main that returns [c7] for the term and numeric joins in all five ScoreModes (2 of 2 tests pass; they check hit ids only, for the term join and the Integer numeric join; scores, Long, Float and Double were not checked).

Trade-offs

Each caller must know about it; the cost of either variant was not measured.

Method: code reading, then bounded-exhaustive translation validation on the stock jar (the registry labels it non-formal because the first signal was code reading; the finder tiered it semi-formal). No model checker, solver or prover was involved.

• First signals: the term half was noted, unassessed, in the notes of the hunt's sibling join finding ("mapped to the empty term", 2026-10-26); the numeric half was flagged on 2026-10-27 by the hunt's planning critic, an independent reviewer from another model family (OpenAI GPT-5.6), from a source check of long value = 0.
• A query-time-join hunt then built a harness that runs every JoinUtil overload on the real jar against a brute-force join: 1-3 from by 1-3 to documents, join values {"", a, U+FF21, U+10400}, {0, 1, -1, 2} and {+0.0, 1.0, -0.0, 2.0}, every from-query subset, every ScoreMode, one or per-document segments, same and two indexes, deletions. Within those bounds (from-scores 1, 2, 4, 8 and 16), missing-as-default was the only hit-set or score mismatch class of the single-valued term and numeric joins; in the two scopes re-run with the fix (one term, one numeric) it disappears and no new class appears.
• The finder wrote a public-API JUnit (2026-10-27). It was then checked by an independent reproducer, reading-only reviews by a model from another family (OpenAI GPT-5.6), an intended-behaviour devil's advocate, an adjudication, a history sweep, a prior-report search, a security screen and a credibility audit.