A MUST_NOT clause next to a scoring block-join query is checked on the child documents, so excluded parents come back

Since #16669 (main and branch_10_6, listed under 10.6.0), ReqExclBulkScorer runs MUST x, MUST_NOT y in 4,096-document windows and passes "live and not matching y" to x's bulk scorer as acceptDocs. When x is a ToParentBlockJoinQuery with ScoreMode Avg, Max, Min or Total, its BlockJoinBulkScorer forwards that set unchanged to the child scorer (ToParentBlockJoinQuery.java:541, since #13697) and never checks the parents it returns. So a parent excluded by MUST_NOT comes back whenever one of its matching children is allowed ([p1, p2] instead of [p1] in a 6-document repro, while count() says 1), a MUST_NOT that matches only children removes parents, and a block that straddles a window boundary, with a matching child before it, makes the search throw. Default codec and searcher, no deletions needed; every release through 10.5.2 is correct (five run, the others by a source scan).

1. Which scorer runs. With a MUST_NOT clause and a required side costing more than 0.3% of the segment, and exactly one MUST clause and no SHOULD clause, BooleanScorerSupplier takes that clause's own bulk scorer and wraps it in ReqExclBulkScorer (BooleanScorerSupplier.java:203-207, :235-236, :256-265).
2. What it passes down. Per window [windowMin, windowMin + 4096) it loads the MUST_NOT matches into a bit set, flips it, ANDs in the live docs and calls req.score(collector, windowBits, windowMin, windowMax) (ReqExclBulkScorer.java:59-86). WindowBits.get answers only inside its window: assert index >= windowBase && index < windowEnd (:113-116), although its length() reports the live docs' length, or the caller's max when there are no deletions, not the window end (:85, :119-121).
3. What the join does with it. A scoring ToParentBlockJoinQuery uses BlockJoinBulkScorer (ToParentBlockJoinQuery.java:166-174; ScoreMode.None uses the default bulk scorer). Its score maps the parent range to the child range [prevParent + 1, lastParent + 1) and calls childBulkScorer.score(wrappedCollector, acceptDocs, prevParent + 1, lastParent + 1) (:533-541). The wrapping collector emits a parent when the next block starts or the batch ends (:586-591, :609-612); nothing tests the parent against acceptDocs.

• Six documents: children 0-1 and parent p1 (doc 2), children 3-4 and parent p2 (doc 5, status:hidden); skill:java on docs 0 and 3.
• Window [0, 4096): the bit set allows docs 0-4 and rejects doc 5. The child scorer asks about docs 0 and 3, both allowed, so p1 and p2 are collected. Doc 5 is never asked about.
• A MUST_NOT skill:java (children only) rejects docs 0 and 3 instead, so both parents disappear: [] instead of [p1, p2].

In a 6,000-document segment of 3-document blocks whose children all match skill:java, the block of docs 4,095-4,097 straddles the window boundary. In the window [4096, 8192), prevParent is 4,094, so the child range starts at 4,095, one below the window. The child scorer asks acceptDocs only about the children it matches (DocAndFloatFeatureBuffer.java:57); doc 4,095 matches, so WindowBits.get(4095) fails its assertion with -ea, and with -da reads mask bit -1 (ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 64). With only doc 4,095 changed to skill:go, the block still straddles the boundary but nothing below the window is asked: the search returns 2,000 hits, the 1,000 hidden parents included, where count() says 1,000. In large random segments with 2% of children matching, 18 of 20 layouts returned the excluded parents without an exception and 2 threw; with 50% matching, all 20 threw.

count(), ConstantScoreQuery and ScoreMode.None build the join with ScoreMode.None (ToParentBlockJoinQuery.java:98), whose default bulk scorer tests acceptDocs on the documents it collects, the parents (Weight.java:296-298). Before #16669, ReqExclBulkScorer passed only the live docs, which agree on children and parents in a valid block-join index, so the forwarding added by #13697 was harmless.

• Trigger. A scoring join that, after rewrite, is the only required clause, with no SHOULD clause; a MUST_NOT clause, or a pure-negative filter such as FILTER(*:* -acl:deny), which BooleanQuery.rewrite turns into a MUST_NOT (it drops the *:* FILTER, BooleanQuery.java:387-392, and inlines the rest, :530-560); a join cost above 0.3% of the segment's maxDoc (BooleanScorerSupplier.java:205; the join's cost is its child query's cost, ToParentBlockJoinQuery.java:177-180; the threshold exists only since #16715, and at #16669's own commit any join cost took the bulk path); a search that needs scores, such as the default search(q, n). Default codec, IndexWriterConfig and IndexSearcher, a CheckJoinIndex-valid index, no deletions.
• Not affected: count(), ConstantScoreQuery, ScoreMode.None and a positive FILTER next to the join (executed); a join used as FILTER (built without scores, so ScoreMode.None) and a second MUST clause next to the join (a conjunction scorer runs instead), by code reading; ToChildBlockJoinQuery (no custom bulk scorer; its MUST join plus MUST_NOT checks in the bounded-exhaustive model, up to 7 documents (8 without deletions), show no mismatch).
• Effect. Silent wrong results: excluded parents come back with scores and can be the top hit (a search(q, 10) test failed 10 of 10 randomized iterations, each time at an excluded parent that ranks first because all scores tie). At scale: 300 blocks in 6 segments, query cache on, returned 260 parents per scoring mode, 89 of them excluded; count() gave the correct 171. Where a block straddles a 4,096-document window boundary and one of its children before the boundary matches the child query, the request fails instead; without such a match it returns the wrong parents silently. A child-only MUST_NOT removes parents or changes their scores (p1's Total score halved in one test).
• Versions. 10.6.0 (released 2026-10-29; the RC1 bits), main and branch_10_6; branch_10x has the same code by reading.
• Likelihood: plausible. A natural nested-search shape, deterministic once the join is not very selective; a positive filter or second MUST clause next to the join, or ScoreMode.None, avoids it.
• Not run: concurrent (executor) search, intra-segment partitions, branch_10x builds.

The test pins the default codec (the test framework randomizes it) and turns the query cache off (the framework randomizes caching), checks the index with CheckJoinIndex, and runs +ToParentBlockJoinQuery(skill:java, ScoreMode.Avg) -status:hidden. The seed only picks the framework's random directory, locale and settings, which the test does not depend on.

The same root cause has two more symptoms, shown by the hunt's three-test class (not needed to file): a MUST_NOT skill:java that matches only children returns [] instead of [p1, p2], and a 6,000-document segment whose children all match and whose block of docs 4,095-4,097 straddles doc 4,096 throws AssertionError at ReqExclBulkScorer$WindowBits.get(ReqExclBulkScorer.java:114) with -ea and ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 64 with -da (main 645d4fe14a20).

Regression test: this same test; it passes with the recommended fix.

cp $REPRO/TestBlockJoinMustNotExcludesParent.java $LUCENE/lucene/join/src/test/org/apache/lucene/search/join/
cd $LUCENE && ./gradlew :lucene:join:test --tests TestBlockJoinMustNotExcludesParent -Ptests.seed=5EED0222
# assertions off: add -Ptests.asserts=false (same failure)

# main 645d4fe14a20, gradle, -Ptests.seed=5EED0222 (the same with -Ptests.asserts=false)
TestBlockJoinMustNotExcludesParent > testMustNotExcludesParent FAILED
    java.lang.AssertionError: expected:<[p1]> but was:<[p1, p2]>
        ...
        at org.apache.lucene.search.join.TestBlockJoinMustNotExcludesParent.testMustNotExcludesParent(TestBlockJoinMustNotExcludesParent.java:81)
:lucene:join:test (FAILURE): 1 test, 1  failure

# standalone runner: main 645d4fe14a20 -ea / -da: fail 1/1; 10.6.0 RC1 jars -ea / -da: fail 1/1 (same message); 10.5.2: pass 1/1
# with the fix: the test passes; :lucene:join:test (SUCCESS): 105 tests, 3 skipped

• The class's own rule. "NOTE: acceptDocs applies (and is checked) only in the parent document space" (ToParentBlockJoinQuery.java:141-142, from 2017). The class javadoc: "You can then use this Query as a clause with other queries in the parent document space" (:56-57); the join package doc: "The resulting query can then be used as a clause in any query that matches parent" (package-info.java:37-38).
• The contracts. BulkScorer.score: acceptDocs "represents the allowed documents to match" (BulkScorer.java:70-71), and this scorer's matches are parents. Occur.MUST_NOT: clauses that "must not appear in the matching documents ... do not contribute to the score of documents" (BooleanClause.java:58-60); the parent-level case returns documents acceptDocs rejects, and in the child-only case a clause that matches no parent still removes parents and changes their scores.
• Consistency. The same query gives the right set through count(), ConstantScoreQuery and ScoreMode.None, and on every release through 10.5.2.
• The crash. WindowBits.length() reports the full length, but get answers only inside its window (ReqExclBulkScorer.java:113-121), so a lookup that Bits allows asserts or reads index -1.
• Precedent. LUCENE-6588 (#7646, 2015, reported with a patch by Christoph Kaser): ToChildBlockJoinQuery skipped a parent's score when the first child of a block was not in acceptDocs, an acceptDocs that was not block-consistent. jpountz first replied "I don't think we should try to support fine-grained deletion of children documents", then, reminded that the acceptDocs came from a filter, committed the fix as a bug ("I had just forgotten that some child documents could seem deleted because of FilteredQuery"). TestBlockJoin.testParentScoringBug still says "delete documents to simulate FilteredQuery applying a filter as acceptDocs" (TestBlockJoin.java:1539-1541).
• No test expects it. TestBlockJoin uses MUST_NOT only inside the query a join wraps, never as a clause next to a join; TestReqExclBulkScorer has no join; the #16669 discussion never mentions joins.

The strongest case: in #14067 (2024), jpountz wrote "Block-joins require deletes to be consistent across parent and child documents. So I wouldn't call it a bug, the implementation should be free to check deletes on child documents or parent documents depending on whichever is more convenient/efficient." So checking children was an accepted design; the join is @lucene.experimental; and a child-only MUST_NOT at the parent level is odd input.

Why it fails: that reply assumed acceptDocs are deletions, which are block-consistent. Since #16669 they also carry MUST_NOT matches, which are not; the BulkScorer javadoc never limited acceptDocs to live docs, the class's own comment says it is checked in parent space, and in LUCENE-6588 the same objection (per-child deletions are unsupported) was dropped once it was clear that a filter applied as acceptDocs need not be block-consistent. @lucene.experimental is about API stability, not wrong results. Verdict: not intended.

Option A was built and run; B and C are untested alternatives.

In ToParentBlockJoinQuery.BlockJoinBulkScorer, give the child scorer null acceptDocs and pass acceptDocs to the wrapping collector, which tests each parent before in.collect(currentParent) (in collect and in endBatch). Every parent collected lies in [min, max), so the lookup stays inside the WindowBits window.

// BlockJoinBulkScorer.score
BatchAwareLeafCollector wrappedCollector = wrapCollector(collector, acceptDocs);
childBulkScorer.score(wrappedCollector, null, prevParent + 1, lastParent + 1);

// wrapCollector(LeafCollector collector, Bits acceptDocs): in collect() and endBatch()
if (currentParent >= 0 && (acceptDocs == null || acceptDocs.get(currentParent))) {
  in.collect(currentParent);
}

Trade-offs

• Children of excluded parents are still scored before the parent is dropped, so #16669's push-down does not help this query; no benchmark was run.
• The child scorer no longer sees the live docs: in an index whose deletions are not whole blocks (unsupported, per #14067) deleted children would count, as they already do on the non-scoring path, whose default bulk scorer checks only the parent (Weight.java:296-298, by code reading).

What the checks validated

• Scratch worktree of 645d4fe14a20, JDK 25: the minimal test fails on stock and passes with the fix, assertions on and off.
• The hunt's three tests, including the 6,000-document crash case, pass with the fix (assertions off).
• The whole lucene/join module with the fix and the test: 105 tests in 17 suites, 3 skipped, 0 failures, assertions on and off; :lucene:join:check -x test (format, ecjLint, forbiddenApis) passes.
• The bounded-exhaustive model with the fix: 0 mismatches for the ToParent family on every layout up to 5 documents (248,936 checks; stock: only the MUST_NOT mismatches). Earlier: an independent verifier's own fix and the hunt's other repros pass; 9 upstream join test classes (45 tests) pass.
• Not run: the full Lucene suite, nightly tests, benchmarks.

Pass the child scorer a Bits that accepts a child when its parent, parents.nextSetBit(child), is accepted. Every such parent lies inside the window.

Trade-offs

Keeps skipping children of excluded parents; costs a nextSetBit per lookup; the same deleted-child semantics as A. Not built.

Pass only the caller's acceptDocs to the required clause and apply the window's exclusion mask to what it collects, through a collector wrapper.

Trade-offs

Could address any bulk scorer that maps doc spaces (not built or validated), but gives up #16669's push-down for every query. Making WindowBits answer below its window would remove only the crash, not the wrong results (by code reading).

Method: bounded-exhaustive translation validation of the block-join queries (BlockJoinTV), on the stock main jar.

• Every CheckJoinIndex-valid single-segment block layout up to 7 documents with whole-block deletions (79,338,044 checks, assertions on and off) and up to 8 without deletions (3,773,664 checks), times every child and parent match set, every ScoreMode and a plain or two-phase child iterator, against an independent brute-force join.
• The only failing kind for ToParentBlockJoinQuery was check C8 (BooleanQuery MUST join + MUST_NOT one document): 20,026 result mismatches per scoring ScoreMode, plus score mismatches; ScoreMode.None was clean. 5 seeded mutants were all killed. The same runs also flag other join queries (ToChildBlockJoinQuery, ParentChildrenBlockJoinQuery and ParentsChildrenBlockJoinQuery, in checks C4-C7); those are not this bug.
• The root cause was then read from ReqExclBulkScorer; the window-straddle crash was found by reasoning about its windows and confirmed by a test. A second hunt unit, modelling ReqExclBulkScorer and scanning the in-tree bulk scorers, found the same bug independently at about the same time (merged into this record).