ToParentBlockJoinSortField with a STRING field, max-child selection and childMissingValue = STRING_LAST throws IndexOutOfBoundsException: the missing-last marker is looked up as a term

Since 10.5.0 (#15727), ToParentBlockJoinSortField takes a childMissingValue: what a child without the sort field counts as in the min/max choice among its siblings (STRING_FIRST or STRING_LAST for Type.STRING). For strings the join wrapper ToParentDocValues.SortedDVs represents "a missing child, sorted last" by the ordinal Integer.MAX_VALUE, and with max-child selection (reverseChildren = true, or reverse = true on the 7-argument constructor) that marker wins for every parent whose block has a child with a value and a member without one. TermOrdValComparator.copy() treats only -1 as missing and calls lookupOrd(Integer.MAX_VALUE), and the default doc-values format rejects the ordinal with an IndexOutOfBoundsException. One block of three documents is enough; it fails with assertions on and off on main, 10.5.0, 10.5.2 and 10.6.0 (10.5.1 was run with assertions on only). Nothing is written or corrupted. A one-wrapper fix in the sort field stops the crash and passes the join module's tests; where such a parent should then sort, relative to parents with no valued child, is a design choice for the maintainers.

• Which comparator. ToParentBlockJoinSortField.getStringComparator (ToParentBlockJoinSortField.java:214-232) builds new TermOrdValComparator(numHits, field, missingValue == STRING_LAST, getReverse(), Pruning.NONE) (:215-216, where missingValue is the parent missing value), selects BlockJoinSelector.Type.MAX when reverseChildren (:221-222) and wraps the doc values with sortMissingLast = (childMissingValue == STRING_LAST) (:228-229). The 7-argument constructor passes reverse as both reverseParents and reverseChildren (:120-128).
• The marker. BlockJoinSelector.wrap (BlockJoinSelector.java:96-132, @lucene.internal at :34) reaches ToParentDocValues.SortedDVs, which sets missingOrd = sortMissingLast ? Integer.MAX_VALUE : -1 (ToParentDocValues.java:55). Per parent it keeps the min or max ordinal of the children with a value (:65-78) and records whether the block has more members than valued children (:264-267, :310-311, :333). ordValue() returns Math.min(ord, missingOrd) or Math.max(ord, missingOrd) (:96-104); for MAX with missing-last that is Integer.MAX_VALUE. lookupOrd passes any ordinal straight through (:106-109).
• Executed on the repro's index: at the parent, MAX + missing-last gives ordValue() = 2147483647 with getValueCount() = 1; MIN + missing-first gives -1; the other two combinations give the real ordinal 0.
• The throw. TermOrdValComparator.copy() (TermOrdValComparator.java:343-358) maps only -1 to its missing state (:345-348), asserts ord >= 0 (:350) and calls termsIndex.lookupOrd(ord) (:354). Lucene90DocValuesProducer checks the ordinal in TermsDict.seekExact and throws IndexOutOfBoundsException (:1812-1814); a plain throw, so -da fails too.
• When. TopFieldCollector calls copy() for every hit that enters the top-N queue (TopFieldCollector.java:126, :116); comparisons against the queue bottom use the ordinal without a lookup (TermOrdValComparator.java:324-341). A search fails once an affected parent is copied into its top hits; an affected parent that is never competitive is never copied, so the failure can look data- and top-N-dependent.
• Not affected: MIN selection (Math.min keeps the real ordinal), STRING_FIRST with MAX (Math.max(ord, -1)), a null child missing value, and parents whose children all lack the field (advanceExact returns false, ToParentDocValues.java:316-328).

• Who is hit. ToParentBlockJoinSortField on Type.STRING with reverseChildren = true (or reverse = true on the 7-argument constructor) and childMissingValue = SortField.STRING_LAST, on 10.5.0 or later. Default codec and settings; the SimpleText format also rejects the ordinal. By code reading the index size does not matter; the runs used small indexes.
• Trigger. A parent block with at least one child that has the field and at least one member without it, which is the data the option exists for. A block member outside childFilter also counts as missing in this code (a separate finding, LFV-220), so a block whose filtered children all have the field but which holds another document type also throws.
• What goes wrong. The search throws an unchecked IndexOutOfBoundsException instead of returning results. Nothing is written or corrupted, and the same searcher keeps answering other queries and other sorts.
• Reach. No downstream user of this combination is known (see the downstream note).
• Workaround. childMissingValue = null or STRING_FIRST, or MIN selection, avoid the crash but give a different order.
• How likely. Plausible: only users of a new option combination, but for them the first query that copies an affected parent into its top hits fails, every time.

Copy the test into lucene/join/src/test/org/apache/lucene/search/join/ of a Lucene checkout and run the command below. The default codec is pinned because the exception type is codec-specific. The test makes no random choices; the test framework still randomizes its environment (similarity, locale, time zone), but the outcome does not depend on the seed (DEADBEEF and 5EED1234 fail the same way). The codec's exception has no message, so the test catches only that type and rethrows an AssertionError that states expected vs actual, with the exception as the cause. It asserts the hit count only, because the javadoc does not fix where such a parent sorts relative to others.

It passes :lucene:join:checkGoogleJavaFormat. Under gradle it fails with -Ptests.asserts=true and false on main and passes with the fix; on the standalone runner it fails on main (seeds DEADBEEF and 5EED1234), 10.5.0, 10.5.1, 10.5.2 and 10.6.0.

cp $REPRO/TestBlockJoinSortStringLastReverse.java $LUCENE/lucene/join/src/test/org/apache/lucene/search/join/
cd $LUCENE && ./gradlew :lucene:join:test --tests org.apache.lucene.search.join.TestBlockJoinSortStringLastReverse -Ptests.seed=DEADBEEF -Ptests.asserts=true
# and once more with -Ptests.asserts=false: it fails both ways

# gradle, fresh worktree at main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (the same with false); module prefixes trimmed, ... = frames left out
TestBlockJoinSortStringLastReverse > testStringLastWithReverseChildren FAILED
    java.lang.AssertionError: expected 1 sorted hit, got java.lang.IndexOutOfBoundsException
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:25E4A7E0D9AA9092]:0)
        at org.apache.lucene.search.join.TestBlockJoinSortStringLastReverse.testStringLastWithReverseChildren(TestBlockJoinSortStringLastReverse.java:74)

        Caused by:
        java.lang.IndexOutOfBoundsException
            at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1814)
            at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:1682)
            at org.apache.lucene.search.join.ToParentDocValues$SortedDVs.lookupOrd(ToParentDocValues.java:108)
            at org.apache.lucene.search.comparators.TermOrdValComparator$TermOrdValLeafComparator.copy(TermOrdValComparator.java:354)
            at org.apache.lucene.search.TopFieldCollector$TopFieldLeafCollector.collectAnyHit(TopFieldCollector.java:126)
            ...
            at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:678)
            at org.apache.lucene.search.join.TestBlockJoinSortStringLastReverse.testStringLastWithReverseChildren(TestBlockJoinSortStringLastReverse.java:71)

# 10.5.2 and 10.6.0 release jars: the same message and frames (seekExact:1640 / :1817, copy:342 / :344)

• Contract. childMissingValue: "The missing value for child documents that lack the sort field. This value participates in the min/max selection among siblings. Type constraints are the same as for parentMissingValue", and for Type.STRING "use SortField#STRING_FIRST or SortField#STRING_LAST" (ToParentBlockJoinSortField.java:139-145, and :106-108 for the 7-argument constructor). Validation accepts STRING_LAST for every reverse flag (:178-198). The 10.5.0 CHANGES entry: children with missing values "now correctly participate in the min/max selection according to the configured missing value" (lucene/CHANGES.txt:1187-1189).
• SortedDocValues. The ordinal "is dense, starts at 0" (SortedDocValues.java:41-42); lookupOrd needs "ord ... >= 0 and < getValueCount()" (:51).
• Review history. In the #15727 review the reviewer (romseygeek) asked that the wrapper "still needs to return true from advanceExact() ... And then return an appropriate value from ord()"; the contributor (gmarouli) answered "my initial thought is -1 since any positive number could be confused with an actual ordinal", and the reviewer pointed to TermOrdValComparator's own missingOrd = Integer.MAX_VALUE (2026-02-24). In the comparator that value is internal state, used only for documents without an ordinal (TermOrdValComparator.java:221-228, :345-348); copy() and lookupOrd were not discussed.
• Tests. TestBlockJoinSorting's only STRING_LAST child case is commented "Sort by descending with missing values last in child level" but passes reverse = false (TestBlockJoinSorting.java:563-572), so max-child selection with STRING_LAST never ran through the sort field.
• Loud, but not noise. The failure is a clean exception, but it needs no absurd parameter: three documented, validated settings and ordinary data (a block with a child that lacks the field).

The case: the marker is deliberate. TestBlockJoinSelector pins ordValue() == Integer.MAX_VALUE for MAX with missing-last (:222, :224, :278, :282), the selector is @lucene.internal, and the sort field is @lucene.experimental (ToParentBlockJoinSortField.java:46).

Why it fails: those tests pin the internal selector's value and never call lookupOrd. The public sort field feeds that value, through documented and validated options, to a comparator that cannot handle it. @lucene.experimental allows API changes, not crashes on accepted input. The argument constrains the fix (keep the selector value), not the verdict; the cross-model intended-behaviour review, the devil's advocate and the adjudication all rated it weak.

Three options; A was validated with gradle in a fresh scratch worktree at 645d4fe14a20, B and C were not executed. A stops the crash; B is the complete fix if the maintainers want a parent whose missing child won to keep its own position. Not a fix: clamping the marker to a real ordinal (getValueCount() - 1), which would turn the exception into a silently wrong sort key, the largest real term, in FieldDoc.fields.

In getStringComparator (ToParentBlockJoinSortField.java:228-229), only when max-child selection meets childMissingValue == STRING_LAST, wrap the selector's doc values in a FilterSortedDocValues whose advanceExact returns false when the selected ordinal is Integer.MAX_VALUE. TermOrdValComparator then applies its own missing handling and never looks the marker up. It is the hunt's crash-avoidance sketch, limited to the one case that produces the marker and with an import instead of a qualified name (diff: +15 / -2 lines, one file).

final boolean childMissingLast = childMissingValue == STRING_LAST;
SortedDocValues selected =
    BlockJoinSelector.wrap(sortedSet, type, parents, toIter(children), childMissingLast);
if (type == BlockJoinSelector.Type.MAX && childMissingLast) {
  // The selector reports "a child without a value, sorted last, won the max" as
  // Integer.MAX_VALUE, which is not an ordinal: report such a parent as having no value.
  return new FilterSortedDocValues(selected) {
    @Override
    public boolean advanceExact(int target) throws IOException {
      return in.advanceExact(target) && in.ordValue() != Integer.MAX_VALUE;
    }
  };
}
return selected;

Trade-offs

• No change elsewhere: only MAX with missing-last produces the marker, and no wrapper is added otherwise. The selector keeps returning Integer.MAX_VALUE, so TestBlockJoinSelector is untouched.
• The comparator is built with Pruning.NONE and reads each document's ordinal only through advanceExact then ordValue (TermOrdValComparator.java:309-315), so the filter covers every such read.
• Order: such a parent sorts where parentMissingValue puts parents without a valued child. That matches the javadoc when parentMissingValue = STRING_LAST, not otherwise. Executed with the fix (blocks P1 = [b, missing], P2 = [c], P3 = [missing]; ascending parents, max child, child STRING_LAST): parentMissingValue = STRING_LAST gives P2, P1, P3; null or STRING_FIRST give P1, P3, P2, where the javadoc implies P3, P2, P1. The cross-model reviewer's own two-parent test shows the same with the hunt's sketch (expected [second, first], got [first, second]). The order probe sorts parents ascending only; with reverseParents = true the hit queue multiplies every comparison by -1 (FieldValueHitQueue.java:68, :144), which was not probed.
• What the test covers: the minimal (regression) test asserts only the hit count, so it checks the crash, not the order; the order above comes from the separate order probe.
• By code reading this is what MIN with STRING_FIRST already does: the wrapper returns -1 (executed) and copy() maps -1 to the comparator's missing state (:345-348).

What the checks validated

Gradle in a fresh scratch worktree at 645d4fe14a20 (JDK 25.0.3, -Ptests.seed=DEADBEEF), 2026-10-30:

• without the fix the minimal test fails with -Ptests.asserts=true and false; git apply of the diff succeeds, and with it the test passes both ways; after git apply -R it fails again;
• the whole join module passes with the fix: :lucene:join:test = 17 suites, 105 tests (the new test included), 3 skipped, 0 failures (among them TestBlockJoinSorting 3, TestBlockJoinSelector 4, TestBlockJoin 18);
• :lucene:join:checkGoogleJavaFormat passes with the fix and the test;
• the diagram probe with fix A shadowed (-ea and -da): the repro's search returns 1 hit with fields[0] = null, the 7-argument reverse = true form too, and the other combinations keep their values;
• earlier, the hunt's sketch (the same logic without the MAX guard) passed the test on main (-ea, -da) and TestBlockJoinSorting with TestBlockJoinSelector, 7 of 7.

Keeps the documented order for every parentMissingValue. The raw ordinals cannot carry the third state as they are: within a segment compare (TermOrdValComparator.java:146), compareBottom (:332) and compareTop (:405) subtract one int ordinal from another, and with parentMissingValue null or STRING_FIRST (the cases B is for) the comparator's missing ordinal is -1 (:224-228), so Integer.MAX_VALUE - (-1) overflows to Integer.MIN_VALUE and flips the sign. Executed by the independent fact check (-ea and -da): a stock comparator over the selector, with only lookupOrd(Integer.MAX_VALUE) made non-throwing, one segment, ascending: parents R3 = [missing] then R1 = [b, missing] come back R1, R3, where the marker's meaning gives R3, R1; two other layouts happen to come out right. Only with parentMissingValue = STRING_LAST does no slot hold -1, the case fix A already places correctly. Across segments the slot value is missing too: cross-segment compare and compareValues use values[slot], with null meaning the parent missing value (:149-160, :188-198); FieldDoc.fields come from value(slot) (:183-185); searchAfter maps the top value back through lookupTerm (:230-243). Not executed as a fix.

Trade-offs

A BytesRef has no "after every term" value, so this needs a marker object, overflow-safe comparisons (Integer.compare) and changes to copy, compare, compareBottom, compareTop, compareValues and setTopValue, in TermOrdValComparator or a join-specific comparator. It would also have to settle what MIN with STRING_FIRST (ordinal -1) means when parentMissingValue = STRING_LAST. An API-level design choice for the maintainers.

Return "no value" (or -1) from ToParentDocValues.SortedDVs itself. Avoids the crash for every caller of the public BlockJoinSelector.wrap, with the same order trade-off as A. Not executed.

Trade-offs

Contradicts the TestBlockJoinSelector assertions that pin Integer.MAX_VALUE (:222, :224, :278, :282), which encode the #15727 design.

Method: code reading, then executed tests (the hunt's non-formal tier: the bug was seen in the code, not reported by a model checker).

• First signal (2026-10-26). A unit studying block-join code read the #15727 missing-ordinal handling in ToParentDocValues (MAX selection against missingOrd = Integer.MAX_VALUE) next to how TermOrdValComparator.copy() treats ordinals. A hand-written probe over every reverse-flag x child-missing-value combination and a JUnit on the stock main jars confirmed it. The unit's bounded-exhaustive sort checker (every layout of up to 5 documents) had to exclude this combination because it throws, and is not credited as the finder.
• Checks. An independent reproducer, an executed cross-model refute review and a reading-only intended-behaviour review (OpenAI GPT-5.6), an intended-behaviour devil's advocate and an adjudication confirmed it; a follow-up swept the releases, searched for prior reports and assessed security; the repro was then reduced to one block and one search.