UnifiedHighlighter re-analysis under the default WEIGHT_MATCHES flag drops payloads, so payload-checking queries are highlighted wrongly or make highlight() throw

When the UnifiedHighlighter cannot read offsets from the index (a plain TextField, every highlightWithoutSearcher call, or a field with offsets in postings whose query has a multi-term part or another part that makes it switch), it re-analyzes the stored text into a one-document MemoryIndex and evaluates the query there. MemoryIndexOffsetStrategy stores payloads in that copy only when the span PhraseHelper is position-sensitive, and under WEIGHT_MATCHES, the default since 9.0.0, the helper is always PhraseHelper.NONE. So the copy never has payloads, yet the whole query runs on it with Weight#matches: SpanPayloadCheckQuery gets no highlight, a payload exclusion in a SpanNotQuery gets an extra one, and a payload-filtered Intervals.term makes highlight() throw IllegalArgumentException. Search is correct. With offsets in postings the same highlighter highlights all three forms correctly; withWeightMatches(false) fixes the span forms, but the interval form then ignores the payload filter and highlights every terms (an older, separate limit). The fix is one condition: also store payloads when WEIGHT_MATCHES is set.

• Offset source: to mark a match, the UnifiedHighlighter needs the character offsets of the matching tokens. It reads them from postings indexed with offsets, from term vectors, or, as the fall-back, by re-analysis: it runs the analyzer over the stored text again and indexes the tokens into a one-document MemoryIndex, a private in-memory copy of the document.
• WEIGHT_MATCHES: the highlighter's default mode since 9.0.0. It asks the query itself where it matches (Weight#matches) on the offset source's reader; with re-analysis that reader is the MemoryIndex copy.
• Payload-checking queries: queries whose match depends on the bytes a token filter attached to each position: SpanPayloadCheckQuery, Intervals.term(term, payloadFilter), or a payload check nested in another span query, such as the exclusion of a SpanNotQuery.

1. WEIGHT_MATCHES is on by default (UnifiedHighlighter.java:110; getFlags adds it at :518-528 because its three required flags are defaults too).
2. The highlighter re-analyzes when the field has neither offsets in postings nor term vectors (getOffsetSource, :696-709), for every highlightWithoutSearcher call (:1078-1081), and also for a field with offsets when mtqOrRewrite holds (getOptimizedOffsetSource, :1257-1261, switch at :1269-1273): the query has a multi-term part, its span phrase helper needs a rewrite, or it has a part the highlighter does not recognize. Re-analysis then uses MemoryIndexOffsetStrategy (:1294-1302), because the default PASSAGE_RELEVANCY_OVER_SPEED or WEIGHT_MATCHES is set (either is enough; so the withWeightMatches(false) baselines use it too).
3. Under WEIGHT_MATCHES, getPhraseHelper returns PhraseHelper.NONE, "will be handled by Weight.matches which always considers phrases" (:1219-1224).
4. MemoryIndexOffsetStrategy.java:47-48 stores payloads only if that helper is position-sensitive: storePayloads = components.phraseHelper().hasPositionSensitivity(); // might be needed. NONE is not, so the copy stores no payloads (MemoryIndex.java:1033-1042) and reports none (:1888-1891).
5. FieldOffsetStrategy then evaluates the whole query with Weight#matches on that copy (FieldOffsetStrategy.java:72-74, :114-147; with a searcher, through an OverlaySingleDocTermsLeafReader that takes only the highlighted field's terms from the copy, MemoryIndexOffsetStrategy.java:116-119). Every payload there is null: a payload check fails, a payload exclusion excludes nothing, and PayloadFilteredTermIntervalsSource.matches throws because the field "has no indexed payloads" (PayloadFilteredTermIntervalsSource.java:163-167).

With WEIGHT_MATCHES off, the helper is position-sensitive for span queries, the copy stores payloads, and the span query is highlighted correctly.

One document, body = both terms. only terms., every token with the payload p + its position (both p0, terms p1, only p2, terms p3); each query matches the document in search. Executed on main a9779ef3c193, assertions on and off:

• Who. Applications that highlight payload-checking queries with the UnifiedHighlighter on its default flags: SpanPayloadCheckQuery (and its PayloadType / MatchOperation variants), Intervals.term(term, payloadFilter), and payload checks nested in other span queries such as a SpanNotQuery exclusion. No Lucene query parser builds these queries, so the application or a downstream product does.
• Which fields. Every field highlighted by re-analysis: fields without offsets in postings or term vectors (a plain TextField), every highlightWithoutSearcher call, and fields with offsets in postings when the query also has a multi-term part (executed) or another part that makes the highlighter switch, such as a span rewrite or an unrecognized query part (read). Correct sources: offsets in postings when the highlighter keeps that source (executed for the plain span, exclusion and interval cases), and term vectors that store positions, offsets and payloads (checked for a span query without a multi-term part). Term vectors stored without payloads cannot answer a payload check in either mode (read); that is a configuration limit, not this bug, and the fix does not change that strategy.
• Default vs opt-in. Default from 9.0.0 (LUCENE-9431); in 7.5.0 to 8.11.4 only when the application enables WEIGHT_MATCHES by overriding getFlags (executed on ports of the test). Downstream, Solr's unified highlighter turned it on by default earlier: SOLR-12754 added hl.weightMatches, default false in Solr 7.6.0 and true from 8.0.0, which also ships the payload_check parser, so Solr users of that highlighter were exposed by default before Lucene 9.0 (read in the history, not run).
• Effect. Silent wrong highlights in both directions: a matching occurrence is not marked, or an occurrence that a payload condition excludes is marked. The interval form makes highlight() throw for a query that searched without error, so the whole highlight call fails; the same highlighter instance still highlights later queries correctly. Search hits, counts and scores are not affected, and the index is not touched.
• Scale. Deterministic at any index size: one document is enough. On the copy every payload is null, so the highlight is wrong whenever the right answer depends on the payload values: a check that requires a payload never matches, and a check for "no payload" (a null entry) matches every occurrence (executed by the reach probe); the interval form throws. A query whose answer does not depend on the discarded payloads is highlighted correctly.
• How likely. Rare: payload-checking queries are an expert feature, and the prior-report search (213 queries, 2026-10-26) found no report in the nearly five years since WEIGHT_MATCHES became the default. Solr looks like the main downstream channel (read, not run).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[<b>word</b>]> but was:<[word]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.uhighlight.TestUnifiedHighlighterReanalysisPayloads.testSpanPayloadCheckQueryOnPositionsOnlyField(TestUnifiedHighlighterReanalysisPayloads.java:57)

• Re-analysis is a documented, interchangeable offset source. Class javadoc (UnifiedHighlighter.java:68-71): "A Highlighter that can get offsets from either postings (...), term vectors (...), or via re-analyzing text." getOffsetSource (:689): "Fall-back: OffsetSource#ANALYSIS is returned." highlightWithoutSearcher (:1078-1081): "Naturally, the mode of operation will be OffsetSource#ANALYSIS."
• The tests treat the sources alike. UnifiedHighlighterTestBase runs the suites over postings, term-vector, postings-with-term-vector and re-analysis field types (UnifiedHighlighterTestBase.java:44-63, :105-118). No UnifiedHighlighter test uses a payload query, so nothing pins the current behaviour.
• WEIGHT_MATCHES promises more accuracy. Builder#withWeightMatches (:340-348): "It's more accurate to the query"; CHANGES for LUCENE-8286: "This mode will highlight more accurately than any other highlighter". Here the flag turns a correct highlight into a wrong one or an exception.
• The same code stores payloads without the flag. The line's comment says payloads "might be needed"; with WEIGHT_MATCHES off the phrase helper is position-sensitive for span queries, payloads are stored and the highlight is right (executed).
• History: an unhandled interaction, not a decision. The condition was written in 2016 for the span-only phrase-helper path (LUCENE-7438, moved by LUCENE-7526). LUCENE-8286 (2018) added the PhraseHelper.NONE branch and a WEIGHT_MATCHES term to the re-analysis routing (the default PASSAGE_RELEVANCY_OVER_SPEED already chose this strategy), but only renamed the accessor in the condition. Neither its threads nor those of LUCENE-9431 (which made the flag the default) mention payloads.
• Payload highlighting is a module capability. TestHighlighter#testPayloadQuery: "We can highlight based on payloads. It's supported both via term vectors and MemoryIndex since Lucene 5."
• Legal trigger. Public analyzer API (PayloadAttribute), public payload queries, the default highlighter.

The verifiers and reading-only reviews by a model from another family built the strongest case: the three-argument highlight() javadoc says the field must "be indexed with offsets" and documents an IllegalArgumentException otherwise (UnifiedHighlighter.java:734-746); "// might be needed" reads as a memory trade-off; the classic QueryScorer keeps MemoryIndex payloads opt-in (setUsePayloads); the UnifiedHighlighter is @lucene.experimental; payload queries are expert.

Why it fails: the @throws clause is stale, since the highlighter does not throw for a field without offsets but falls back to re-analysis as getOffsetSource documents; the bug is also reached on a field that does have offsets and through highlightWithoutSearcher, which has no such precondition. The cost argument would justify a switch, but the UnifiedHighlighter has none and documents no limitation, and the cost is per highlight call and grows with the kept tokens (one int per kept token, the payload bytes with their offsets, and one 32 KiB block per highlighted field and call; not measured). @lucene.experimental is about API stability, not wrong output or an exception escaping highlight(). The reviews rated the intended argument moderate and agreed on the root cause; all verifiers judged it real, the devil's advocate with high confidence that it is not intended.

What the rebuttal does not settle: no UnifiedHighlighter API promises that re-analysis keeps payloads, and the withWeightMatches javadoc hedges: the snippets "can be a little different for phrases", the passage relevancy "can be different (maybe worse?)" and it is "slower when highlighting many fields" (UnifiedHighlighter.java:340-347). Those hedges are about markup and passage scoring, not about matches that are lost or invented, but together with the memory cost they make a gated fix or a documented limit a reasonable choice for the maintainers.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle; the other options were not run.

Workarounds today: withWeightMatches(false) (span payload checks then highlight correctly; the interval form then ignores the payload filter); index offsets in postings (helps only when the highlighter keeps that source: no multi-term part, span rewrite or unrecognized query part); or store term vectors with positions, offsets and payloads (correct in a check with a span query without a multi-term part).

A separate, known item to expect after the fix: a payload-filtered term inside positional interval combinations (ordered, unordered, maxgaps, atLeast, ...) still throws UnsupportedOperationException from getQuery() in WEIGHT_MATCHES highlighting even with payloads present (another registry row, partly covered by upstream #10074). The fix here does not touch that; a single payload-filtered term passes with it.

In MemoryIndexOffsetStrategy's constructor, add || components.highlightFlags().contains(UnifiedHighlighter.HighlightFlag.WEIGHT_MATCHES) to the storePayloads condition (+7/-1 with the comment). It keeps the line's intent (its comment is // might be needed) of storing payloads whenever evaluating the query might need them: under WEIGHT_MATCHES the consumer is the query's own Weight#matches. A second diff adds the repro as a regression test; a PR also needs a CHANGES entry.

// Payloads might be needed: by span queries in the PhraseHelper, or, with WEIGHT_MATCHES, by
// any payload-aware query, because Weight#matches then evaluates the whole query here.
boolean storePayloads =
    components.phraseHelper().hasPositionSensitivity()
        || components
            .highlightFlags()
            .contains(UnifiedHighlighter.HighlightFlag.WEIGHT_MATCHES);
memoryIndex = new MemoryIndex(true, storePayloads); // true==store offsets

Trade-offs

• Cost (read, not measured), for each copy built under WEIGHT_MATCHES: one extra int per kept token position (-1 when the token has no payload); each non-empty payload copied into a BytesRefArray with one offset int per payload; and that array's first 32 KiB byte block, allocated when the MemoryIndex is created, i.e. once per highlighted field per highlight call (reset() keeps it for the next document). Only tokens kept by the query-term pre-filter count. On the searcher paths the stored text is cut to maxLength (default 10,000 chars); highlightWithoutSearcher re-analyzes the whole string the caller passes, as it already does for the rest of the copy. The same costs are already paid with WEIGHT_MATCHES off whenever the phrase helper holds span queries.
• Payload-free fields pay these costs too; option B avoids them at the price of plumbing.

What the checks validated

• Stock main a9779ef3c193: the repro fails 5 of 5 with assertions on and off and 50 of 50 over 10 iterations, every failure at the default-flags assertion; the whole highlighter module has 727 tests with 5 failures, exactly the repro.
• With the fix: the repro passes 5 of 5 with assertions on and off, and 50 of 50 over 10 iterations.
• With the fix, the whole lucene/highlighter module (classic, unified, match and vector highlighters, incl. the repro): 35 classes, 727 tests, 0 failures.
• ecjLint (main and test), checkGoogleJavaFormat and renderJavadoc for the highlighter module pass; both diffs apply cleanly to a clean a9779ef3c193 tree.
• Earlier, by the hunt, the same condition shadowed in: the finder's, the independent reproducer's, the devil's advocate's, the review-stage check's and the reach probe's tests all pass; shadowed into 7.5.0 and 8.1.0 it makes the history ports pass.
• Not run: other modules' suites (the change is in lucene/highlighter only); no benchmark.

Use FieldInfo#hasPayloads() of the indexed field, as Solr's hl.payloads does for the original Highlighter, and always store them for highlightWithoutSearcher, which has no FieldInfo.

Trade-offs

Saves these costs for payload-free fields, but MemoryIndexOffsetStrategy only receives UHComponents, so getOffsetStrategy or UHComponents must carry the information.

Store payloads only when the query reads them.

Trade-offs

No generic marker exists for "this query reads payloads" (span payload checks, payload-filtered intervals, custom queries), so it is fragile and misses third-party queries.

State that payload-checking queries are not supported by re-analysis under WEIGHT_MATCHES.

Trade-offs

No code change, but the silent wrong highlights and the exception stay for anyone who misses the sentence.

Method: bounded-exhaustive translation validation on the real jars, then code reading (semi-formal tier), in a hunt unit on highlighter offset sources.

• An exact reference semantics written from the javadoc, plus the Matches API as a second oracle, was compared with every UnifiedHighlighter offset source (postings, term vectors, both, re-analysis) and flag set, and with the other highlighters, on every document of a small enumerated document language and a 121-query catalogue: about 2.6e8 checks on the stock main jars. It flagged the re-analysis MemoryIndex as an unfaithful copy of the document (the class of the known #10466).
• The model's document language has no payloads, so it could not express this case. Reading MemoryIndexOffsetStrategy's storePayloads condition next to getPhraseHelper found it, and a JUnit on the stock jar witnessed it end to end.
• Found 2026-10-26 on main ea57360912ed; then checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6) and an adjudication.