SpanWithinQuery's Matches report the contained span's position but the enclosing span's offsets and terms, so the default UnifiedHighlighter marks the enclosing terms

SpanWithinQuery(big, little) matches the little spans that lie inside a big span. For each match, SpanWeight#matches takes the positions from the spans, which for this query are the little spans, but the offsets and the sub-matches from Spans#collect, and ContainSpans#collect collects the big span's terms as well. So one MatchesIterator reports the contained term's position with the enclosing span's offsets and terms: for ac inside ab … bc, positions [1,1] with offsets [0,8) and the sub-matches ab@0, ac@1, bc@2. The default UnifiedHighlighter (weight matches on since 9.0.0) therefore marks the enclosing terms, on fields with and without offsets, and MatchRegionRetriever returns [0,8) on a field with offsets but [3,5) on the same text indexed with positions only. Nested span queries are affected the same way. Which documents the query matches and its own scores are unaffected. The interval counterpart had the same inconsistency and was fixed in 9.1.0 (LUCENE-10229); spans were never changed. Present since span queries got Matches in 7.5.0.

• SpanWithinQuery(big, little): "Keep matches that are contained within another Spans"; it matches "spans from little that are inside of big" (SpanWithinQuery.java:28, :31-35). SpanContainingQuery(big, little) is the mirror image: it keeps the big spans that contain a little one.
• Weight#matches(context, doc) returns Matches; getMatches(field) gives a MatchesIterator with, per match, its start and end position, its start and end offset, and getSubMatches(), "the positions and offsets of individual terms within the current match" (MatchesIterator.java:47-84). Highlighters use it: the UnifiedHighlighter by default since 9.0.0, MatchRegionRetriever and MatchHighlighter since 9.0.0.
• Spans#collect(SpanCollector) visits the leaf terms (postings) of the current span; payload scoring and the UnifiedHighlighter's older phrase path use it too.

1. SpanWeight#matches (SpanWeight.java:209) returns spans.startPosition() and spans.endPosition() - 1 as the match positions (:252-260).
2. Its startOffset() and endOffset() are the first and last entries of innerTerms (:262-276), and getSubMatches() iterates innerTerms (:278-323). collectInnerTerms() fills innerTerms with spans.collect(termCollector) and sorts them by position (:330-334).
3. For SpanWithinQuery the spans are new ContainSpans(big, little, little) (SpanWithinQuery.java:90): positions come from the third argument, the little spans (ContainSpans.java:28-47).
4. But ContainSpans#collect calls bigSpans.collect(collector); littleSpans.collect(collector); (ContainSpans.java:54-58). At a match the big spans sit on the enclosing span, so ab@0 and bc@2 join ac@1; after the sort the offsets run from ab's 0 to bc's 8.
5. The UnifiedHighlighter marks every leaf sub-match (OffsetsEnum.java:171-240) and re-analyzes a positions-only field into a MemoryIndex that stores offsets (MemoryIndexOffsetStrategy.java:48), so it is wrong on both fields. MatchRegionRetriever reads the match offsets on a field with offsets (OffsetsFromMatchIterator.java:53-54) but maps the match positions to offsets on a field without them (OffsetsFromPositions), so the two fields disagree.

SpanContainingQuery uses new ContainSpans(big, little, big) (SpanContainingQuery.java:91): its positions are the big span's, so its offsets and terms agree with them. The interval counterpart was fixed by keeping only the small source: "the only sub source we care is the \"small\" source" (ContainedByIntervalsSource.java:69-75, #521).

• Highlighting. The UnifiedHighlighter uses Weight#matches by default since 9.0.0 (LUCENE-9431; opt-in from 7.5.0, LUCENE-8286): it marks <b>ab</b> <b>ac</b> <b>bc</b> on a field with offsets and on a plain positions-only TextField (executed). MatchRegionRetriever and MatchHighlighter (9.0.0, LUCENE-9463) return [0,8) with offsets and [3,5) without (executed). Custom code that reads MatchesIterator offsets or sub-matches gets the wide values (executed).
• Nested queries. SpanNear([SpanWithin(...), zz]) reports offsets [0,11) instead of [3,11) and the terms ab, bc besides ac, zz (executed). SpanOrQuery, FilterSpans and the other composite spans forward collect the same way (read).
• No special settings. Nothing beyond using SpanWithinQuery and highlighting: the mechanism depends only on the span tree and the indexed positions and offsets, not on codec or IndexWriterConfig settings (read). The JUnits ran under the test framework's randomized codec, IndexWriterConfig and similarity, and the highlighter with all its defaults (executed); no field with offsets is needed for the UnifiedHighlighter.
• How often. On every SpanWithinQuery match whose enclosing span has leaves outside the contained one, those leaves become sub-matches, and with ordinary (non-stacked) tokens the match offsets widen to them; that is typical when the big clause is wider than the little one, though no usage data was gathered. Deterministic.
• How visible. The extra marks are query terms, so they look plausible; SpanWithinQuery is a little-used legacy query (intervals are the modern replacement, and #16820 proposes deprecating spans). Registry likelihood: rare; priority: medium (both qualitative; maintainers could rate it lower).
• Downstream. Elasticsearch exposes the query as span_within; a maintainer closed elastic/elasticsearch#42715 (2024) as fixed with output that highlights the enclosing terms, shown as evidence that highlighting works; the extra marks were not discussed (read, not executed).
• What stays right. Which documents a SpanWithinQuery matches, counts and its own scores: span scoring never calls Spans#collect, whose only callers are Matches, the payload queries, the UnifiedHighlighter's PhraseHelper, the sandbox PayloadSpanUtil and the composite spans that forward it (read). Executed: identical hits and bit-identical scores for top-level SpanWithinQuerys with and without fix option B; fix option A changes payload-function scores over a SpanWithinQuery (see the fixes). The SpanContainingQuery and Intervals.containedBy controls on the same input pass (executed).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<3> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.queries.spans.TestSpanWithinMatchesOffsets.testMatchOffsetsAreThoseOfTheContainedSpan(TestSpanWithinMatchesOffsets.java:63)

• The contract. MatchesIterator.java:47-52 "The start position of the current match"; :61-66 "The starting offset of the current match"; :68-73 "The ending offset of the current match"; :75-84 getSubMatches "iterates over the positions and offsets of individual terms within the current match". If ab and bc were part of the match, its positions would be [0,2]; if they were not, its offsets would be [3,5) and they would not be sub-matches.
• What the query says it matches. SpanWithinQuery.java:28 "Keep matches that are contained within another Spans"; :31-35 "matching spans from little that are inside of big"; :75-78 "Return spans from little ... The payload is from the spans of little".
• Precedent. LUCENE-10229 (#11265, by @dweiss) "Match offsets should be consistent for fields with positions and fields with offsets" quotes @romseygeek, who led the Matches API work, from #11259: "It's because it returns the position of the inner match, but the offsets of the outer ... the current behaviour is kind of the worst of both worlds". #521 (@zhaih, 9.1.0, CHANGES.txt:4469-4470) made containedBy report only the small source; #803 (9.2.0) followed up. Spans were not mentioned or changed.
• Design intent. LUCENE-8306 described getSubMatches() as "another MatchesIterator over the individual term positions within the current match"; LUCENE-8286's CHANGES entry says weight-matches highlighting "will highlight more accurately than any other highlighter" (CHANGES.txt:7161-7162).
• Internal disagreement (executed): MatchRegionRetriever [0,8) with offsets vs [3,5) without; the classic Highlighter gives ab <B>ac</B> bc; Intervals.containedBy gives [3,5) and ac only.
• No test or discussion endorses it. No upstream test checks the positions, offsets, sub-matches or highlights of SpanWithinQuery or SpanContainingQuery Matches: TestSpanWithinQuery and TestSpanContainQuery reach the test framework's CheckHits.checkMatches, which asserts only that Matches is non-null for every hit (CheckHits.java:353-359). The LUCENE-6083, -6706, -8306, -8404 and -10229 threads say nothing in favour of reporting the enclosing span.

The strongest works-as-designed case (the intended-behaviour verifier and a reading-only review by a model from another family): a containment match depends on both clauses, and ContainSpans#collect has visited both on purpose since LUCENE-6706 (2015, payload scoring), so "every leaf that caused the match" is one consistent reading of the sub-matches, and the Matches class javadoc says it reports "all matching terms in a query" (Matches.java:24-25); Matches and MatchesIterator are @lucene.experimental and SpanWeight is "Expert-only"; the UnifiedHighlighter's older phrase path (PhraseHelper, its own Spans#collect call) has highlighted the enclosing terms for years; and an Elasticsearch maintainer closed #42715 as fixed with such output. The #11259 comment quoted above also says "It's not obvious to me what the correct response is here" and goes on: "perhaps we should change it so that you get offsets of the inner match as standard, and then the outer match is returned as part of the sub matches"; the expectation that the sub-matches are ac only follows what #521 implemented, not that comment.

Why it fails: it explains the collector, not one MatchesIterator whose positions and offsets name different regions (even the #11259 suggestion gives the match the inner offsets, [3,5) here); "experimental" lets the API change, not contradict its own javadoc; and Lucene already ruled on the identical inconsistency for intervals. The verifiers rejected it (moderate strength).

Option A was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF); option B (the finder's sketch) was run on the same JUnits; option C was not built.

Override collect in the anonymous ContainSpans that SpanWithinWeight#getSpans returns (+6 lines in SpanWithinQuery.java), so that Spans#collect visits only littleSpans. This fixes the top-level case, nested span queries (composite spans forward collect), big-span terms stacked on the little span's position, and also the UnifiedHighlighter's non-Matches phrase path (PhraseHelper calls Spans#collect itself). It gives the same small-source-only result as #521, which made containedBy report only the small source, but unlike #521 (an intervals Matches change only) it also changes the other Spans#collect consumers. SpanContainingQuery is unchanged.

        @Override
        public void collect(SpanCollector collector) throws IOException {
          // the match is the little span: collect its leaves only, not the big span's
          littleSpans.collect(collector);
        }

Trade-offs

• Payloads change for SpanWithinQuery. Every Spans#collect consumer sees only the contained span: PayloadScoreQuery over a SpanWithinQuery scores only its payloads (probe, payloads ab=1, ac=2, bc=4: min/avg/max 1.0/2.33/4.0 before, 2.0/2.0/2.0 after), SpanPayloadCheckQuery checks only those, so it can accept or reject different spans and therefore match different documents (SpanPayloadCheckQuery.java:164-169), and the sandbox PayloadSpanUtil returns only those (read).
• That matches the existing SpanWithinWeight#getSpans javadoc ("The payload is from the spans of little", SpanWithinQuery.java:76-77, LUCENE-6083) but reverses what LUCENE-6706 (5.3.0) made collect do; that change targeted payload scoring in general, its thread never discusses SpanWithinQuery, and no upstream test covers SpanWithinQuery payloads.
• A direct SpanContainingQuery is unchanged, but a SpanContainingQuery or other composite with a SpanWithinQuery clause changes the same way.
• It would need a CHANGES entry that names the payload change.
• The adjudication and the registry notes describe the complete fix as a Matches-only leaf collection that keeps payload behaviour (option C). A is recommended here because it is the only option built and validated that fixes the nested and stacked cases and the PhraseHelper path, and because it matches the getSpans javadoc; if payload behaviour must not change, C is the choice, and maintainers may prefer it.

What the checks validated

• Applied in a clean worktree at a9779ef3c193, gradle, seed DEADBEEF: both JUnits pass, 4 of 4 and 4 of 4, with assertions on and off.
• The whole lucene/queries test task (53 suites, 518 tests, 5 skipped) and lucene/highlighter (36 suites, 727 tests) pass; that includes this doc's 2 JUnits and 2 probes, so 85 upstream suites with 1,234 tests.
• With -Ptests.nightly=true: TestPayloadScoreQuery (including the @Nightly testSpanContainingQuery), TestPayloadCheckQuery and TestPayloadSpans pass, 22 tests.
• Probe: the stacked case becomes [1,1] [3,5) with the sub-match ac only; the UnifiedHighlighter gives ab <b>ac</b> bc with weight matches on and off.
• checkGoogleJavaFormat passes for both modules with the fix and the two JUnits.
• Not run: the rest of ./gradlew check (forbidden APIs, ecj, javadoc lint), other modules (for example lucene/sandbox, whose PayloadSpanUtil calls Spans#collect, and lucene/monitor), other seeds (all runs used DEADBEEF), benchmarks.

In SpanWeight#matches collectInnerTerms(), keep only collected terms whose position lies in [startPosition, endPosition) (+14 lines; the finder's sketch). No change to payloads or scoring.

Trade-offs

• Leaves nested queries wrong: big-span terms inside the outer match positions survive (bc@2 in SpanNear([SpanWithin, zz])), so the default UnifiedHighlighter still marks bc there: ab <b>ac</b> <b>bc</b> <b>zz</b> on "ab ac bc zz", where option A gives ab <b>ac</b> bc <b>zz</b> (executed by the fact check).
• Leaves stacked tokens wrong: a big-span term at the little span's position survives (xy@1 in the probe).
• The UnifiedHighlighter's non-Matches phrase path is unchanged.

What the checks validated

• Same worktree and seed, assertions on: TestSpanWithinHighlighting passes 4 of 4, but TestSpanWithinMatches still fails 1 of 4: the nested test, "Unexpected term match: 2[6->8]" (bc@2).
• Probe: the stacked case keeps the sub-match xy@1 [3,8) (offsets become [3,5)); the UnifiedHighlighter without weight matches still marks all three words; payload scores unchanged.
• In the hunt, the same diff passed the finder's and the independent reproduction's tests and changed no other result in a run of 437 highlighter and span tests (435 upstream tests plus the finder's 2-test repro; only the repro changed).

Give Spans a second collection path used only by SpanWeight#matches (for example a method that defaults to collect, overridden by the within-ContainSpans and forwarded by every composite Spans). It would fix nested and stacked cases without touching payloads.

Trade-offs

Touches every composite Spans class and adds API surface to a module that may be deprecated (#16820). Not built or run.

Method: bounded-exhaustive translation validation on the stock jars (the hunt's semi-formal tier), hunt unit "highlighter offset sources".

• The harness ran the real UnifiedHighlighter offset sources, MatchRegionRetriever and the Matches API of the stock main jars over every document of a small enumerated document language (holes, stacked tokens, multi-valued fields with position gaps 0 and 1) and a 121-query catalogue, and compared them with an exact reference semantics and with each other.
• Relation R7b (MatchRegionRetriever on a field with offsets equals the same on a positions-only field) failed on the multi-valued tiers, which have no stacked tokens with distinct offsets, only for SpanWithinQuery(SpanNear([ab, bc], 1, ordered), ac): 44 of 1,815 span checks at gap 1 and 56 at gap 0 locally, 3,280 in the larger gap-0 batch job.
• The root cause was then found by reading SpanWeight#matches and ContainSpans#collect, and reduced to a public-API JUnit.
• No model checker, SMT solver or prover was involved.