ParentsChildrenBlockJoinQuery refuses a second scorer for the same segment, so explain() of a BooleanQuery that contains it throws

ParentsChildrenBlockJoinQuery (new in 10.4.0, @lucene.experimental) protects itself against intra-segment concurrency with a Weight-wide Set<LeafReaderContext> seenContexts (ParentsChildrenBlockJoinQuery.java:151 at 645d4fe14a20): the first statement of scorerSupplier throws IllegalStateException("ParentsChildrenBlockJoinQuery does not support intraSegment concurrency. Context ... was already seen.") when a segment is requested a second time (:191-197). The Weight#scorer and Weight#scorerSupplier javadoc says a scorer (supplier) for the same LeafReaderContext "may be requested multiple times as part of a single search call" (Weight.java:115-117, :138-140), and Lucene makes such a second request itself, on one thread: BooleanWeight#explain explains every clause (the join's explain pulls a scorer, ParentsChildrenBlockJoinQuery.java:172) and then pulls its own scorer for the same segment to compute the score (BooleanWeight.java:121-126). So IndexSearcher.explain of a BooleanQuery that keeps the join as one of 2+ clauses throws for every matching document (as a non-scoring FILTER clause only when no query cache serves it), on a default IndexSearcher with no executor, with assertions on and off; the same happens for FunctionScoreQuery explain and the test-framework check QueryUtils.checkBulkScorerSkipTo. Search, count and explain of the join on its own work and returned correct results in every test. It has been there since the class was added (#14728, first released in 10.4.0) and is not fixed upstream. A candidate fix that checks the searcher's slices once in createWeight instead of remembering segments was tested: the repro and the whole lucene/join suite pass, including the upstream intra-segment test; the design is upstream's choice.

1. The guard. ParentsChildrenBlockJoinWeight holds private final Set<LeafReaderContext> seenContexts = new HashSet<>(); (ParentsChildrenBlockJoinQuery.java:151), and scorerSupplier(context) starts with:

if (!seenContexts.add(context)) {                                        // :192
  throw new IllegalStateException(                                       // :193
      "ParentsChildrenBlockJoinQuery does not support intraSegment concurrency. Context "
          + context + " was already seen.");
}

LeafReaderContext does not override equals/hashCode, so the key is the context instance, and the set lives as long as the Weight: after one scorerSupplier call for a segment, every later call for that segment on the same Weight throws, from any thread or partition.

1. The join's own explain pulls a scorer. explain(context, doc) starts with scorer(context) (:172), which calls scorerSupplier(context) (Weight.java:129-134): request 1.
2. BooleanWeight#explain pulls a second one. It calls explain on every clause (BooleanWeight.java:72-75), decides match or no-match from those (:76-119), and for a match pulls scorer(context) (:126) "to make sure that explanations have the same value as the score" (:121-125). BooleanWeight.scorerSupplier asks every clause for its scorer supplier (:300): request 2, and the guard throws.
3. Why search and count work. IndexSearcher.searchLeaf asks for one scorer supplier per segment partition (IndexSearcher.java:842), and a searcher without intra-segment partitions has one partition per segment. Explain of the join alone makes only request 1.
4. What the guard is for. The introducing commit: "Note that this query type will not work with intrasegment concurrency, as it may split a block, leading to overcollection of child documents." With intra-segment concurrency, each partition of a split segment calls scorerSupplier for the same LeafReaderContext on the shared Weight (IndexSearcher.java:760-788, :814, :842). The seen-set detects that, but also every sequential repeat. (The over-collection itself was not executed here; the guard rejecting a split segment loudly was.)

bq = BooleanQuery(SHOULD join, SHOULD color:blue); color:blue matches nothing and only keeps the query at two clauses, because a one-clause BooleanQuery rewrites to its clause (BooleanQuery.java:278-288).

• A document matched only by the other clause; and, on the verifiers' own indexes, BooleanQuery(MUST join, FILTER type:child) and BooleanQuery(MUST color:blue, FILTER join) (the independent reproduction, whose join selects color:blue children). On the index above the equivalent is MUST color:red, FILTER join, which throws on main's default searcher.
• FunctionScoreQuery explain around the join (FunctionScoreQuery.java:176, :181: inner.explain, then inner.scorer for the same segment).
• QueryUtils.checkBulkScorerSkipTo (QueryUtils.java:605, :613: scorerSupplier, then bulkScorer on one Weight).
• A second Weight.scorer / scorerSupplier call on a segment, even after the first scorer was fully consumed.

The independent reproduction's class fails 4 of 6 and the intended-behaviour check's probe 5 of 9 on stock main; both pass in full with the fix.

Query cache. A FILTER join throws only when no query cache serves it: only non-scoring clauses go through the cache (BooleanWeight.java:56-61, IndexSearcher.java:979-980), and once the cache holds the join for a segment it answers both requests without asking the join's Weight. On the index above, explain(MUST color:red, FILTER join, 0) throws with setQueryCache(null) but returns a match (0.13076457) with an always-caching LRUQueryCache, on main and on 10.6.0, and also matched under the test framework's random default cache at seed DEADBEEF, so a FILTER-shaped regression test would be flaky. A scoring clause is never cached: the repro's SHOULD join, SHOULD color:blue throws even with the always-caching cache. Main's default searcher has no query cache (IndexSearcher.java:82-84, #14187); every release from 10.4.0 to 10.6.0 defaults to an LRUQueryCache that caches only on segments of 10,000+ documents and, under the default policy, only a filter used several times.

• Who is hit. Code that combines the join with another clause and explains a hit: BooleanQuery (SHOULD+SHOULD, the join as MUST with a FILTER clause, the join as FILTER when no query cache serves it), FunctionScoreQuery around it, and test code that runs QueryUtils checks on it. The issue that motivated the class (#14565) shows BooleanQuery(MUST ToChildBlockJoinQuery, MUST childQuery), the workaround this class replaces by taking the child query itself; combining a query with other clauses, such as filters, is ordinary Lucene usage.
• Default vs opt-in. Default IndexSearcher (no executor), default codec and settings, assertions on or off; opt-in only in that the query class is experimental and new. The study's code search (default branches) found no use of the class in Solr, Elasticsearch or OpenSearch (a bounded search, not proof that nobody uses it).
• Effect. One IllegalStateException per explain call, for every document the compound query matches; a document that matches no clause gets a normal no-match explanation. The message is misleading: it blames intra-segment concurrency on a searcher that has none. The search and count calls tested (default searcher, no partitions) returned correct results.
• Workaround. Explain the join query on its own. No simple public-API workaround that keeps the normal explanation of the compound query was identified.

The try/catch only turns the IllegalStateException into an AssertionError that states expected vs actual; the exception is chained as the cause. The test framework randomizes, per seed, the default codec, similarity, locale and time zone (seed DEADBEEF ran with codec=Asserting(Lucene104), sim=Asserting(RandomSimilarity(...)), locale=en-NA, timezone=Asia/Bishkek) and installs a default query cache with a random caching policy for every test. None of this changes the outcome: the test asserts only isMatch(), and the join is a scoring (SHOULD) clause, which the query cache never serves. 20 of 20 iterations of --iters 20 (seeds derived from a random root seed) also failed.

Add the Apache license header when you put the file into Lucene. Suggested regression test: this same test, as its own class or as one method in TestParentsChildrenBlockJoinQuery.

# main 645d4fe14a20, gradle, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestParentsChildrenBlockJoinExplain > testExplainOfBooleanQueryWithJoinClause FAILED
    java.lang.AssertionError: expected a match explanation for doc 0, got java.lang.IllegalStateException: ParentsChildrenBlockJoinQuery does not support intraSegment concurrency. Context LeafReaderContext(_0(11.0.0):c2 docBase=0 ord=0) was already seen.
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:4D917E5C3C561CE1]:0)
        at org.apache.lucene.search.join.TestParentsChildrenBlockJoinExplain.testExplainOfBooleanQueryWithJoinClause(TestParentsChildrenBlockJoinExplain.java:44)

        Caused by:
        java.lang.IllegalStateException: ParentsChildrenBlockJoinQuery does not support intraSegment concurrency. Context LeafReaderContext(_0(11.0.0):c2 docBase=0 ord=0) was already seen.
            at org.apache.lucene.search.join.ParentsChildrenBlockJoinQuery$ParentsChildrenBlockJoinWeight.scorerSupplier(ParentsChildrenBlockJoinQuery.java:193)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:300)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.Weight.scorer(Weight.java:130)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.BooleanWeight.explain(BooleanWeight.java:126)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.IndexSearcher.explain(IndexSearcher.java:967)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.IndexSearcher.explain(IndexSearcher.java:943)
            at org.apache.lucene.search.join.TestParentsChildrenBlockJoinExplain.testExplainOfBooleanQueryWithJoinClause(TestParentsChildrenBlockJoinExplain.java:41)
:lucene:join:test (FAILURE): 1 test, 1  failure

# 10.6.0 release jars, -da: the same AssertionError, caused by the same IllegalStateException
#   (Context LeafReaderContext(_0(10.6.0):c2 ...)) at ParentsChildrenBlockJoinQuery.java:193 <- BooleanWeight.explain:126
# with the candidate fix on main: 1 test, 0 failures (assertions on and off)

• The contract allows repeats, with no exception: Weight#scorer "A scorer for the same LeafReaderContext instance may be requested multiple times as part of a single search call." (Weight.java:115-117), the same for scorerSupplier (:138-140) and bulkScorer (:167-168). MIGRATE.md:1678-1684: "Weight implementations that rely on the assumption that a scorer, bulk scorer or scorer supplier for a given LeafReaderContext is requested once per search need updating."
• The guard's purpose is partitions only: its message, the introducing commit ("may split a block, leading to overcollection of child documents") and the upstream test testIntraSegmentConcurrencyNotSupported, which uses a Concurrency.INTRA_SEGMENT searcher and asserts the exception only when there is more than one slice (TestParentsChildrenBlockJoinQuery.java:406-478). Here there is no executor and one slice.
• Older users of the contract: BooleanWeight#explain has pulled a second scorer since LUCENE-8008 (d27ddcb409d, 2017-11-29); QueryUtils.checkBulkScorerSkipTo asks one Weight twice per segment.
• Never discussed: the #14728 conversation and inline reviews do not mention explain, compound queries or repeated requests; the class javadoc (ParentsChildrenBlockJoinQuery.java:43-51), CHANGES and MIGRATE state no such restriction. The error text and seenContexts occur nowhere else in Lucene, and the sibling ToChildBlockJoinQuery handles a repeated supplier and a compound explain.

Strongest case (from the intended-behaviour checker, the cross-model reviewers and the adjudication; all rated it moderate): the guard is deliberate and an upstream test asserts it; the class is @lucene.experimental; the "may be requested multiple times" wording and the MIGRATE note came with intra-segment concurrency (#13542), so a query that opts out of intra-segment search might opt out of that clause; explain "should not be displayed with every hit" (IndexSearcher.java:937-938); and every other upstream test of the class that searches uses a Concurrency.NONE searcher with the query on its own ("Do not use Concurrency.INTRA_SEGMENT as that would break childLimitPerParent counting", TestParentsChildrenBlockJoinQuery.java:185, :246, :369; testIntraSegmentConcurrencyNotSupported uses Concurrency.INTRA_SEGMENT, :464, and testInvalidChildLimit builds no searcher).

Why it fails: the guard rejects a case its own purpose does not cover (one thread, one slice, no partitions) with a message that says otherwise; the repeat clause is unconditional and its users predate the class; a sequential repeat made after the first scorer is fully consumed also throws, so it is not about concurrent state; and a tested candidate fix that keeps the protection for split segments passes the upstream intra-segment test.

The candidate fix was applied to a scratch worktree of 645d4fe14a20 and run through gradle, and the independent fact check re-ran those steps. It was not tested with an executor running slices in parallel or with custom slice implementations beyond the probe's, and how to detect partitions is upstream's call. Alternatives B and C were not run. Deleting the guard is not a fix.

In ParentsChildrenBlockJoinQuery.createWeight, walk searcher.getSlices() and throw IllegalStateException("ParentsChildrenBlockJoinQuery does not support intraSegment concurrency. Segment ... is split into partitions.") if any LeafReaderContextPartition has minDocId > 0 or maxDocId < maxDoc. Then delete the seenContexts field, its imports and the check at the top of scorerSupplier. The condition matches how IndexSearcher builds partitions: a whole segment is [0, NO_MORE_DOCS), a split segment's parts end at or before maxDoc (IndexSearcher.java:404-416, :1097-1111).

// createWeight(): call checkNoSegmentIsSplit(searcher) first, then build the Weight
private static void checkNoSegmentIsSplit(IndexSearcher searcher) {
  for (IndexSearcher.LeafSlice slice : searcher.getSlices()) {
    for (IndexSearcher.LeafReaderContextPartition p : slice.partitions) {
      if (p.minDocId > 0 || p.maxDocId < p.ctx.reader().maxDoc()) {
        throw new IllegalStateException(
            "ParentsChildrenBlockJoinQuery does not support intraSegment concurrency. Segment "
                + p.ctx + " is split into partitions.");
      }
    }
  }
}
// and remove seenContexts and the check at the top of scorerSupplier()

Trade-offs

• Sequential repeats (explain, FunctionScoreQuery, QueryUtils) work; real partitions are still rejected, now up front and deterministically, instead of when a second partition happens to reach the Weight.
• On a searcher that splits segments, every use of the query fails in createWeight, including explain of the join alone, which works there today (search, and count unless a query cache answers it, already throw there: on stock main with a 4-partition searcher and no cache, count(join) and search(join) throw the guard's exception and explain(join, 0) is a match). Consistent with "does not support intraSegment concurrency", but a behaviour change for such searchers.
• Code that partitions a segment without getSlices() (a subclass calling the protected search(LeafReaderContextPartition[], Weight, Collector) with its own ranges) is no longer caught; today's guard catches it when two ranges of one segment reach the same Weight.
• Likewise no longer caught: a custom slices() that puts the same whole segment into two slices. getSlices() rejects a repeated leaf only within one slice (IndexSearcher.java:549-575) and a whole-segment partition passes the new check; today's guard throws when the second slice asks the same Weight (read, not run).
• Removes the Weight's mutable HashSet, which today is written from scorerSupplier, a method that concurrent slice tasks call on the same Weight (concurrent access was not tested).
• Custom slices that split segments exactly at block boundaries stay rejected, as today.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20 with the repro test added, seed DEADBEEF: the repro fails on stock and passes with the fix, with -Ptests.asserts=true and false.
• TestParentsChildrenBlockJoinQuery with the fix: 8 tests, 0 failures, including testIntraSegmentConcurrencyNotSupported; 80 tests, 0 failures with -Ptests.iters=10.
• The whole :lucene:join:test with the fix: 17 classes, 105 tests, 0 failures, 3 skipped (@Nightly), assertions on and off. The same suite on stock: 105 tests, 1 failure, the repro.
• checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain of lucene/join pass with the fix.
• With the fix shadowed into the standalone runner: the independent reproduction's class passes 6 of 6 (stock: 4 fail; its explain checks assert explanation value == search score for every hit) and the intended-behaviour check's probe 9 of 9 (stock: 5 fail; BooleanQuery explain on three shapes, FunctionScoreQuery explain, QueryUtils.checkBulkScorerSkipTo).
• A fix-validation probe with a searcher that splits the one segment into four one-document partitions: searching the join still throws "ParentsChildrenBlockJoinQuery does not support intraSegment concurrency." with the fix, as on stock; explain of the join alone there gives a match on stock and the new exception with the fix.
• Not tested: an executor running slices in parallel, and custom slice implementations beyond the probe's.

Compute the same getSlices() condition in createWeight, pass it to the Weight, and skip the seenContexts check when no segment is split.

Trade-offs

Smallest behaviour change: searchers without intra-segment partitions (the default, and inter-segment concurrency) are fixed, and segment-splitting searchers behave exactly as today. But it keeps the identity-based check, so explain of a compound query still fails on segment-splitting searchers, and it keeps the mutable set. Not run.

Instead of rejecting every split, reject only partitions that cut a block (minDocId > 0 and doc minDocId - 1 is not a parent), the constraint BlockGroupingCollectorManager documents for block grouping (BlockGroupingCollectorManager.java:36-43).

Trade-offs

More permissive and precise, but it needs the parent bit set per partition, and it changes what the upstream test asserts: testIntraSegmentConcurrencyNotSupported expects the exception whenever there is more than one slice, while IndexSearcher.slices puts partition boundaries at multiples of maxDoc / numSlices (IndexSearcher.java:398-416), which can fall on a block boundary. The lower-boundary condition is enough only while the partitions of a segment are contiguous and cover it, as IndexSearcher builds them; arbitrary custom partitions also need the matching upper-boundary check (maxDocId == maxDoc, or doc maxDocId - 1 is a parent). Not run.

Removing seenContexts and its check makes every repeated-request test pass. (The study's root-cause check only: the repro tests pass, the upstream test fails 1 of 8.)

Trade-offs

It fails the upstream testIntraSegmentConcurrencyNotSupported (8 run, 1 failed), and by the introducing commit's rationale intra-segment search would then over-collect children silently instead of failing.

Method: non-formal (a hand-written probe, then a directed JUnit), 2026-10-26, main ea57360912ed. The hunt unit's bounded-exhaustive translation-validation harness compares explain with the search score; it builds ParentsChildrenBlockJoinQuery only standalone and reported no exception for it. While triaging that harness's other findings for the class, the finder explained the query inside a BooleanQuery, hit the exception, checked the Weight javadoc, and wrote a public-API JUnit on the stock main jar (2 of 2 fail, -ea and -da). A second hunt unit later hit the same exception on its own and deferred to this finding. A minimisation step replaced the finder's test with the one-method test shown here.