ToParentBlockJoinSortField sorts a parent as missing when its block holds a document outside childFilter

ToParentBlockJoinSortField sorts block-join parents by a field of their children, and childFilter selects which non-parent documents of a block take part. Since #15727 (10.5.0) it also applies a missing value when a child lacks a value. To decide that, ToParentDocValues compares the number of childFilter documents that have a value with the number of all non-parent documents in the block. So in a block where at least one childFilter document has a value, any other document outside childFilter, with or without the field, flags the parent, and wherever the missing value wins the min/max the parent sorts as missing: null and first under the default ascending STRING sort, childMissingValue under a numeric sort. In the repro every non-parent document has the field and 10.4.0 sorts correctly; 10.5.0 to 10.5.2 and main do not. No error is raised, and the hits, the hit count and the index are unchanged.

• Block: documents indexed together with IndexWriter.addDocuments, parent last. The join package calls the last document of each block the parent and "all others" child documents (package-info.java:30-31).
• ToParentBlockJoinSortField(field, type, reverse, parentFilter, childFilter) (@lucene.experimental): sorts parents by the smallest (ascending) or largest (descending) value of field among the parent's children selected by childFilter. Two constructors added by #15727 also take a childMissingValue, "The missing value for child documents that lack the sort field".

1. Each comparator passes toIter(childFilter.getBitSet(context)), a BitSetIterator, to BlockJoinSelector.wrap, which builds a ToParentDocValues (ToParentBlockJoinSortField.java:220-229 for STRING, the numeric types alike).
2. ToParentDocValues intersects the children with the doc values (childWithValues, ToParentDocValues.java:216) and takes the min or max over those documents only, counting them. So far only childFilter documents take part, which is correct.
3. It then sets hasChildWithMissingValue = childrenWithValuesCount < totalChildren with totalChildren = docID - prevParentDocID - 1, every non-parent document of the block (nextDoc :265-267, advanceExact :310-311 and :333). For P1: 1 < 2, so true, though no child lacks the field. advanceExact, the path the top-hits comparators use (TermOrdValComparator.java:309-310, LongComparator.java:79-80), returns false before this comparison when no childFilter document of the block has a value (ToParentDocValues.java:316-318, :326-328), so only parents with at least one such document are flagged.
4. The flag replaces the value. STRING: ordValue() returns Math.min(ord, missingOrd) for MIN and Math.max(ord, missingOrd) for MAX (:96-103), with missingOrd -1, or Integer.MAX_VALUE for childMissingValue = STRING_LAST. For P1, Math.min(2, -1) = -1: no value, FieldDoc value null, sorted first. Numeric: longValue() returns Math.min/max(value, childMissingValue) when childMissingValue is set (:187-195); for P1, Math.min(40, 0) = 0.

git blame attributes the count and comparison lines of step 3 (265-267, 311 and 333) to #15727; line 310, which finds the previous parent, dates from 2017. Before it, ToParentDocValues had no missing-value logic, so a document outside childFilter could not affect a parent.

One segment, default codec, CheckJoinIndex passes; query ToParentBlockJoinQuery(type:a, parents, ScoreMode.None).

P1=0 is neither doc 0's 40 nor doc 1's 7: the missing value replaced a real one.

• An excluded sibling without the field: A=null B=c instead of B=c A=z.
• Three-level blocks with childFilter on the child level: every parent that has a grandchild sorts as missing (A=null B=null C=m instead of B=c C=m A=z; run with grandchildren that lack the field).
• The index of upstream's TestBlockJoinSorting.testNestedSorting (every child has field2; rebuilt by verifiers), childFilter filter_1:T, sorted ascending: d=null i=null a=a b=c c=e g=k instead of a=a b=c c=e d=g g=k i=m, which 10.4.0 and 9.12.3 return. Upstream checks only the descending order, where Math.max(ord, -1) hides the defect.

On 10.4.0 and 9.12.3 the verifier's full STRING suite reports 7 tests and 1 failure: testInFilterChildWithoutValueIsMissing expects #15727's fix for a childFilter child without a value (the #15548 bug that #15727 fixed). The cases above pass there.

• Who is exposed. Applications that sort block-join parents with ToParentBlockJoinSortField and a childFilter that does not select every non-parent document of the block: several child types in one block (for example comments and reviews under one product), which the join package documents ("all rows for all joined tables matching a single row of the primary table must be indexed as a single document block", package-info.java:26-28) and which upstream's own testNestedSorting fixture uses; or multi-level blocks with childFilter on one level.
• Affected settings. MIN and MAX are the child-level selection (reverse, or reverseChildren in the 8-argument constructor, ToParentBlockJoinSortField.java:221-222). STRING ascending with missing-first, the default of the 5-argument constructor, and STRING ascending with an explicit childMissingValue = null, which also acts as missing-first (:229) although the javadoc says null skips children without a value (:108, :145); numeric types with a childMissingValue that wins the min/max (for example 0 ascending, Long.MAX_VALUE descending); STRING descending with childMissingValue = STRING_LAST, where a related bug (LFV-223) throws IndexOutOfBoundsException instead.
• Not affected. STRING descending with missing-first and STRING ascending with STRING_LAST (the missing marker cannot win), and numeric sorts with childMissingValue = null.
• Default or opt-in. The class is opt-in and @lucene.experimental, and a selective childFilter is the application's choice. Nothing else is needed: default codec, config and searcher, assertions on or off; CheckJoinIndex accepts the index.
• Scale. None: 5 documents in one segment. Deterministic: every returned parent whose block holds a childFilter document with a value and a document outside childFilter is flagged, and under the affected settings gets the missing value, on every query.
• What you see. No error. A wrong parent order and wrong FieldDoc sort values (null for STRING, childMissingValue for numeric types). The hits, totalHits and the index do not change, and parents whose blocks hold no excluded document keep their values.
• Downstream (read, not run). Solr's sort=childfield(...) builds the 5-argument constructor, so childfield(<string field>) asc would be affected, but Solr main pins Lucene 10.4.0. Elasticsearch and OpenSearch do not use this class. That says nothing about private code.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<P[2=d P1=y]> but was:<P[1=null P2=d]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.join.TestBlockJoinSortFilteredOutSibling.testStringSortIgnoresSiblingOutsideChildFilter(TestBlockJoinSortFilteredOutSibling.java:82)

• The contracts. @param childFilter "Filter that defines which child documents participates in sorting." (ToParentBlockJoinSortField.java:63, :83, :110, :147); @param childMissingValue "The missing value for child documents that lack the sort field. This value participates in the min/max selection among siblings." (:106-108, :143-145).
• BlockJoinSelector agrees. Its wrap methods "select one value per parent among its {@code children}" (BlockJoinSelector.java:91-95, :116-120), where children is the iterator the caller passes; its Bits path decides whether "any of the children of the given parent document has a value" by walking only the supplied children (:51-89).
• CHANGES for #15727 (lucene/CHANGES.txt:1154-1156): "fix incorrect handling of children missing values. Children with missing values now correctly participate in the min/max selection according to the configured missing value." Nothing about documents outside childFilter; #15727's body, 2 comments, 14 reviews and 19 review comments and its issue #15548 never mention them.
• Upstream tests. testNestedSorting (TestBlockJoinSorting.java:45, :284-296) uses a selective childFilter on mixed blocks and expects the excluded siblings' values to be ignored, but checks only the descending order, where Math.max(ord, -1) hides the missing flag, so it cannot show whether excluded documents were meant to affect "missing". TestBlockJoinSelector.testSortedSelector (:113, from 2016) carries missing-value assertions that #15727 added (lines 169-227): :176-179 expect ord -1 for two parents whose only value-less documents are outside its children set. Inside those two blocks no out-of-filter document has a value, so it says nothing about the valued case.
• Regression (STRING sort; the numeric childMissingValue constructors are new in 10.5.0, so that case was wrong from the start). 10.4.0, 10.3.2, 10.0.0 and 9.12.3 sort the valued and the field-less layouts correctly; reverting #15727's three classes over 10.5.0 or main makes the test pass.

The strongest case for "intended" was made by the finder's cross-model refute review (an independent reviewer from another model family, OpenAI GPT-5.6, which reproduced the behaviour and agreed on the root cause) and again by a GPT-5.6 reviewer arguing "works as designed": a split reading of "child". In the refute review's words, "childFilter restricts which existing values are selected, while structural block membership determines whether the parent has any missing child". That reading gives exactly P1=null and P1=0, which is what the code does. In its support: package-info calls every non-parent document of a block a child; #15727 counts children structurally from the PR commit that first adds missing-value detection (e781ce73d4, the third of the PR's 16 commits); testSortedSelector asserts the block-wide count; CHANGES says children with missing values now participate; the class is @lucene.experimental and BlockJoinSelector is @lucene.internal. The finder first accepted this and filed the finding as a documentation question.

Why it fails for the valued case: the split reading treats doc 1, which has s and n, as a child that lacks the sort field, while childMissingValue is documented as "The missing value for child documents that lack the sort field"; it counts doc 1 for the missing check but drops its value. A document outside childFilter then takes part in the sort through its mere presence, against "@param childFilter Filter that defines which child documents participates in sorting." Neither plain reading gives the result: with only childFilter documents as children P1 is y (LONG 40), with every non-parent document as a child P1 would be x (LONG 7). For the STRING sort it is a regression from 10.4.0, also on upstream's own fixture sorted ascending, and nothing in #15727's CHANGES entry, body, comments or reviews records a decision about documents outside childFilter. testSortedSelector has no valued out-of-filter document in the blocks it asserts, and the descending-only testNestedSorting cannot tell either way. @lucene.experimental marks an unstable API, not permission to return wrong results.

What remains open: for an excluded document that lacks the field, upstream may still decide the block-wide count is intended, because testSortedSelector encodes it.

One fix was built and validated in a scratch worktree at ea57360912ed; the other options come from the verifiers, the adjudication and the security assessment and were not run.

ToParentDocValues remembers the BitSet behind the BitSetIterator it is given and counts the set bits of the block with nextSetBit, instead of docID - prevParentDocID - 1, in nextDoc and advanceExact; the advanceExact count moves below its early returns. +24/-2 lines in ToParentDocValues.java. The upstream test TestBlockJoinSelector.testSortedSelector needs docs 1, 11, 13 and 14 (no value) in its children set (+5 lines), which keeps every expectation and passes on stock main too.

this.childBits = children instanceof BitSetIterator bsi ? bsi.getBitSet() : null; // constructor
int totalChildren = countChildren(prevParentDocID, docID); // was: docID - prevParentDocID - 1

private int countChildren(int prevParentDocID, int parentDocID) {
  if (childBits == null) return parentDocID - prevParentDocID - 1; // not a BitSetIterator
  int count = 0;
  for (int doc = prevParentDocID + 1; doc < parentDocID; doc++) {
    doc = childBits.nextSetBit(doc, parentDocID);
    if (doc == NO_MORE_DOCS) break;
    count++;
  }
  return count;
}

Trade-offs

• Cost. Per collected parent, one nextSetBit call per childFilter document of its block plus at most one final call that finds none, where the stock code does one subtraction. In the worst case that is linear in the block size, like the existing walk over the children that have a value, but it also visits childFilter documents without a value, so with sparse doc values it can cost more. No benchmark was run.
• Fallback. Callers that pass a children iterator other than a BitSetIterator keep today's block-wide count. All in-tree callers pass toIter(children) (ToParentBlockJoinSortField.java:229, :256, :283, :313, :352), and BlockJoinSelector is @lucene.internal; option D removes the fallback.
• Safe bounds. The loop keeps nextSetBit's start below its end (a start == end call is a separate bug, LFV-032).
• Behaviour change beyond the valued case. An excluded document without the field no longer forces the missing value either. That is the design question, and the reason testSortedSelector needs its fixture changed.
• Side effect. It removes the widened trigger of the related IndexOutOfBoundsException (LFV-223) when the only 'missing' document is outside childFilter, not that bug itself.

What the checks validated

• Stock: the test fails 2 of 2 with assertions on and off; the modified TestBlockJoinSelector passes 4 of 4.
• Fix alone: the test and TestBlockJoinSorting pass; testSortedSelector fails at line 177 (expected:<-1> but was:<3>), the block-wide expectation.
• Fix plus the fixture change, :lucene:join:test: 17 classes, 106 tests, 0 failures, 3 skipped (@Nightly, in suites that do not use these classes), with assertions on and off; a second seed also passed.
• :lucene:join:tidy left the fix unchanged; :lucene:join:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• Standalone harness with the patched class: the test 2/2 (assertions on and off); the independent verifier's STRING (7/7, including three-level blocks and the upstream fixture ascending), numeric (6/6) and LFV-223 interaction (1/1) tests; the security probe with its fixed expectations (2/2).

Treat a parent as missing only when some document of its block has no value.

Trade-offs

Fixes the valued case and keeps testSortedSelector unchanged, but leaves the field-less case, and with it the one-bit signal for left-out documents without the field (the security assessment prefers option A). Needs a value lookup for block documents outside childFilter, which the current single conjunction does not give, so it is harder to write than A.

State that every non-parent document of the block counts as a child for the missing-value check.

Trade-offs

Contradicts "Filter that defines which child documents participates in sorting", does not explain the valued case (a document whose value is ignored still forces the missing value), and keeps the 10.4 to 10.5 change.

Give ToParentDocValues the BitSet itself through new BlockJoinSelector.wrap overloads, so the count never depends on the iterator's class.

Trade-offs

Removes option A's fallback; changes an @lucene.internal API with five call sites in ToParentBlockJoinSortField.

Method: code reading (the hunt's non-formal tier).

• A hunt unit (join and block join, 2026-10-26). The finder, an AI tool, was writing the reference semantics for the sort part of the unit's bounded-exhaustive harness and noticed that the count of "children" in ToParentDocValues ignores the childFilter iterator it is given. A public-API JUnit on the stock jar confirmed it.
• Afterwards the harness, run with the javadoc's filter-relative reading, reproduced it: 288 mismatches in 9,504 checks over every layout of 2 to 3 documents and the tested sort settings. With this behaviour modelled as implemented, it found nothing else in 617,760 checks over every layout of 2 to 5 documents. Both runs skip STRING with MAX child selection and childMissingValue=STRING_LAST, the combination of the related bug LFV-223. So the signal came from code reading, not from the exhaustive run.
• An independent reviewer from another model family (OpenAI GPT-5.6) then reproduced it but called it intended, and the finder re-filed it as a documentation question; an independent reproducer found the valued case, and further verifiers and an adjudication confirmed it as a bug.