MatchRegionRetriever's default OffsetsFromTokens strategy ignores consumeTermsMatching, so multi-term queries get no hit regions on fields without positions

For a field indexed without positions (DOCS, DOCS_AND_FREQS or DOCS_AND_CUSTOM_FREQS, for example a stored StringField), MatchRegionRetriever and the MatchHighlighter built on it use OffsetsFromTokens by default: it re-analyzes the stored values and keeps the tokens whose terms the matching query reports to a QueryVisitor. Its visitor overrides only consumeTerms. Constant-score multi-term queries report their terms through consumeTermsMatching (an automaton): PrefixQuery, WildcardQuery, RegexpQuery and TermRangeQuery with their default rewrite whenever their automaton accepts more than one term (a degenerate one that names a single term reports it through consumeTerms and highlights), and TermInSetQuery with two or more distinct terms. So the term set stays empty and the field gets no hit region, although the document is a hit and Weight#matches reports a match on that field. The outcome depends on how a query reports its terms, not on what matched: a one-term TermInSetQuery is highlighted and a two-term one is not; PrefixQuery with SCORING_BOOLEAN_REWRITE is highlighted and with the default rewrite it is not. Search results are correct; only the hit regions are missing. Depending on its chain, MatchHighlighter then omits the field, shows it unhighlighted or throws. Present since the package was added (9.0.0). The fix collects the consumeTermsMatching automata for the field too, once per distinct query.

• Hit regions: MatchRegionRetriever returns, per document and field, the character ranges (OffsetRange) that the query matched; MatchHighlighter turns them into snippets. For each field it picks an offset strategy from the field's index options (MatchRegionRetriever.java:488-505).
• OffsetsFromTokens: the default strategy for fields indexed DOCS, DOCS_AND_FREQS or DOCS_AND_CUSTOM_FREQS (no positions; every StringField is DOCS). It re-analyzes the stored values and keeps the tokens whose terms the query "hinted at when passed a QueryVisitor" (:498-501).
• QueryVisitor's two channels: consumeTerms is "Called by leaf queries that match on specific terms"; consumeTermsMatching is "Called by leaf queries that match on a class of terms" and passes an automaton (QueryVisitor.java:33-51).

1. highlightDocument asks weight.matches(...) (MatchRegionRetriever.java:405), skips a field only when its MatchesIterator is null or MATCH_WITH_NO_TERMS (:411-416), calls the strategy (:434), and drops the field when the strategy returns no ranges (:435-437).
2. With their default rewrite (CONSTANT_SCORE_BLENDED_REWRITE, PrefixQuery.java:32-34; also the classic query parser's default, QueryParserBase.java:83-84), PrefixQuery, WildcardQuery, RegexpQuery, TermRangeQuery and TermInSetQuery match through a constant-score wrapper whose matches() returns DisjunctionMatchesIterator.fromTermsEnum(context, doc, q, ...) (AbstractMultiTermQueryConstantScoreWrapper.java:203-212): every sub-match's getQuery() is the multi-term query itself (DisjunctionMatchesIterator.java:94, 132, TermMatchesIterator.java:73-75).
3. Visited, that query reports through consumeTermsMatching: AutomatonQuery.visit delegates to CompiledAutomaton.visit (AutomatonQuery.java:150-154), which uses consumeTermsMatching for NORMAL and ALL automata and consumeTerms only for a SINGLE term (CompiledAutomaton.java:365-382); TermInSetQuery uses consumeTerms for 1 distinct term and consumeTermsMatching for 2 or more (TermInSetQuery.java:182-187). So a degenerate automaton query that accepts exactly one term (WildcardQuery(lucene) without wildcard characters, RegexpQuery(lucene), TermRangeQuery [lucene TO lucene]) does highlight, while the match-all WildcardQuery(*) does not (executed).
4. OffsetsFromTokens.get visits each match's getQuery() with a visitor that overrides only consumeTerms (OffsetsFromTokens.java:56-70) and keeps a token only if matchTerms.contains(...) (:82). The set stays empty, no token is kept, and step 1 drops the field.

Why the controls work: a TermQuery reports consumeTerms; SCORING_BOOLEAN_REWRITE and FuzzyQuery's default rewrite end in TermQuery clauses; a field with positions uses OffsetsFromPositions (MatchRegionRetriever.java:492-493), which uses the match positions instead of the term set.

One document, stored StringField tag = lucene, solr (indexed DOCS), plus a positional twin tag_pos; re-analysis with KeywordAnalyzer. Executed on main a9779ef3c193, assertions on and off:

In the hunt's bounded-exhaustive harness (1,295 enumerated documents, a 121-query catalogue, stock main jars) the region list was empty in 8,720 of 11,280 multi-term checks, for all 7 constant-score queries; the fuzzy and term queries all agreed.

• Who. Users of the matchhighlight package (MatchRegionRetriever with its default strategies, or MatchHighlighter) who highlight a field indexed without positions, typically a keyword-style StringField (the upstream test calls such fields "often useful for multivalued 'keyword-like' fields", TestMatchRegionRetriever.java:686), and run a prefix, wildcard, regexp, term-range or multi-term set query on it.
• Default vs opt-in. Nothing to opt into: the default-strategy constructors (MatchRegionRetriever.java:107-109, MatchHighlighter.java:187-192), a field without positions, and a multi-term query with its default rewrite, which is also what the classic query parser builds for tag:luc* (read). TermInSetQuery with two or more distinct terms reaches this defect from 9.6.0 (before that, sets of up to 16 terms were rewritten into term clauses and highlighted, while larger sets lost their regions by another route; see History), DOCS_AND_CUSTOM_FREQS fields from 10.5.0.
• Effect. Silent missing highlights; hits, counts and scores are correct and the Matches API reports the match. In MatchHighlighter the field takes the application's no-match path: omitted with skipRemaining(), shown as leading text with maxLeadingCharacters(...), and with no no-match highlighter in the chain the call throws RuntimeException: No field highlighter could be matched to field (MatchHighlighter.java:301-308), as stock Lucene already does for a queried field without a match (executed on main and 10.5.2).
• Scale. Deterministic, with assertions on and off, at any index size: one document is enough, and it fails on every document such a query returns.
• How likely. Rare: GitHub code search found the package used in none of Solr, Elasticsearch and OpenSearch (partly rate-limited; that says nothing about use elsewhere); TextField, the usual full-text field, is indexed with positions (TextField.java:36, 40) and is not affected; the searches made found no report since 9.0.0.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<{f=[[from=4, to=7]]}> but was:<{}>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.matchhighlight.TestOffsetsFromTokensPrefixQuery.testPrefixQueryOnFieldWithoutPositions(TestOffsetsFromTokensPrefixQuery.java:61)

• Contract. Package javadoc: MatchRegionRetriever "can be used to retrieve hit areas for a given Query" (package-info.java:22-23); class javadoc: "compute a list of 'match regions' for a given query" (MatchRegionRetriever.java:57-58); OffsetsFromTokens "works for fields where we know the match occurred" (OffsetsFromTokens.java:35-36); the default-strategy comment keeps the terms "that the query hinted at when passed a QueryVisitor" (MatchRegionRetriever.java:498-501); consumeTermsMatching is for "leaf queries that match on a class of terms" (QueryVisitor.java:42).
• The channel decides, not the match. On the same document, a 1-term TermInSetQuery highlights and a 2-term one (equivalent to a TermQuery disjunction that does highlight) does not; the default and the scoring rewrite of the same PrefixQuery disagree (executed). TermInSetQuery moved to consumeTermsMatching for a performance reason (LUCENE-9349: "to enable queries that build large automata to provide them lazily", CHANGES.txt:5695-5698).
• The Matches change was made for this path. LUCENE-9462, "Fields without positions should still return MatchIterator" (CHANGES.txt:4700-4701; 6244383e0d5, same day and authors as the component, an ancestor of it), removed the hasPositions() == false fallback from TermQuery, SynonymQuery and the multi-term constant-score wrapper, so that these queries, multi-term ones included, return real matches iterators on these fields.
• Sibling and goal. The UnifiedHighlighter in the same module consumes consumeTermsMatching (uhighlight/MultiTermHighlighting.java:82-85). The umbrella issue #10501 / LUCENE-9461 asked that "highlights should be accurate for all sorts of queries: fuzzy, phrase, prefix, Boolean, spans, etc."
• Tests and history. Upstream tests run prefix, range and fuzzy queries only on fields with positions (TestMatchRegionRetriever.java:219-291) and the no-positions field only with a TermQuery (:683-707). No javadoc, test, CHANGES or MIGRATE entry, or design thread (#10479 / LUCENE-9439, apache/lucene-solr#1721, LUCENE-9461 to 9464) says multi-term queries are unsupported here; consumeTermsMatching (2019) predates the component (2020).
• Legal trigger. Public API, the default-strategy constructor, a query rewritten against the searcher, an analyzer that reproduces the indexed tokens (as the constructor javadoc requires, MatchRegionRetriever.java:115-117), ordinary queries with their default rewrites.

The strongest "works as designed" case (the cross-model review, the intended-behaviour check, the refute review): the OffsetsFromTokens javadoc says it returns tokens "that are also returned by the query's term collector" (OffsetsFromTokens.java:38-39), and QueryVisitor.termCollector collects consumeTerms only (QueryVisitor.java:86-98); consumeTermsMatching is @lucene.experimental; MatchHighlighter calls itself "An example highlighter" and advises assembling your own (MatchHighlighter.java:38-43); the default-strategy comment says "possibly" and names alternative strategies (MatchRegionRetriever.java:498-504); in the design review the author wanted the no-positions path minimal, a corner case.

Why it fails: the javadoc phrase does not name termCollector and the code does not call it; termCollector's own javadoc promises "all terms that may match a query" (QueryVisitor.java:87), so its blindness to automata is a gap, not a documented exclusion. The strict reading makes the output depend on the rewrite method and on the number of terms in a set, which points to an accidental dependency on the reporting channel rather than a design choice (no design statement says either way). @lucene.experimental is about API stability, and the example status of MatchHighlighter does not cover MatchRegionRetriever's default strategy. No reviewer found the intended case convincing: the adjudication and the cross-model lenses rated it moderate, the intended-behaviour check found that it does not hold up, and the independent reproduction noted only that maintainers might file the fix as an improvement.

What it does not settle: maintainers may still label the fix an improvement rather than a bug fix (judge confidence medium). Framing it as "the default fallback ignores consumeTermsMatching" and leading with the two inconsistency pairs pre-empts the "term collector" reading.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle; options C to E were not run.

Workarounds today: give the multi-term query MultiTermQuery.SCORING_BOOLEAN_REWRITE (it then highlights; subject to the boolean clause limit and scoring cost); index the field with positions (a custom FieldType instead of StringField), so OffsetsFromPositions is used; or supply a custom OffsetsRetrievalStrategySupplier, for example OffsetsFromValues, which marks every stored value of the field in full (not run).

In OffsetsFromTokens.get, also override consumeTermsMatching and collect the automaton for the field; visit each distinct getQuery() once (an identity set), because a multi-term query returns itself for every matching term; keep a re-analyzed token when the term set contains it or any automaton accepts it. The class javadoc changes in the same diff. A second diff adds the repro as a regression test; a PR also needs a CHANGES entry. On 10.x the visitor method takes Supplier<ByteRunAutomaton> instead of Supplier<ByteRunnable> (#15961 changed it on main), so the backport of the fix changes only that type. The repro test uses unnamed _ lambda parameters (Java 22 or later); branch_10x compiles for Java 21, so a 10.x copy of the test needs named parameters.

Set<Query> visited = Collections.newSetFromMap(new IdentityHashMap<>());
while (matchesIterator.next()) {
  Query q = matchesIterator.getQuery();
  if (visited.add(q) == false) continue; // a multi-term query repeats itself per term
  q.visit(new QueryVisitor() {
    // consumeTerms(...) unchanged: adds exact terms to matchTerms
    @Override
    public void consumeTermsMatching(Query query, String queryField, Supplier<ByteRunnable> automaton) {
      if (field.equals(queryField)) matchAutomata.add(automaton.get());
    }
  });
}
// per re-analyzed token:
if (matchTerms.contains(token) || acceptedByAny(matchAutomata, token)) { ranges.add(...); }

Trade-offs

• Cost. Each re-analyzed token of the field is tested against the distinct automata of the query on that field (one per multi-term clause), the same order as the UnifiedHighlighter's multi-term highlighting (read; not benchmarked with many distinct clauses). The security probe's cost case, one document with 20,000 values in a DOCS field, half matching, WildcardQuery(a*z) over 32-character tokens: stock 51 ms, this fix 59 ms (best of 3, shared loaded host, orders of magnitude only).
• Limits (read). The deduplication works within one get(...) call, so once per document and field; nothing is cached across documents. An automaton query's supplier returns the automaton it already compiled (CompiledAutomaton.java:369, 486-492), but TermInSetQuery's builds a new automaton over the whole term set on every call (TermInSetQuery.java:186, 190-195, "pretty heavy-weight" in its own comment), so a large set pays that once per highlighted document. Deduplication is by object identity: Lucene's wrapper passes one query object to every term match, but a custom MatchesIterator returning new, equal query objects would not be deduplicated. The visitor keeps the default acceptField; overriding it would skip other fields' clauses earlier.
• Why the deduplication matters. The hunt's first draft added the automaton on every matchesIterator.next() (once per matching term) and ran every copy against every non-matching token: 112,571 ms on that same document. Do not propose the draft as is.
• Tokens are still matched only if the re-analysis analyzer reproduces the indexed terms, as before.

What the checks validated

• Stock main a9779ef3c193 (gradle): the repro fails 6 of 6 with assertions on and off and 60 of 60 over 10 iterations, every failure at a test's last assertion; the whole highlighter module has 728 tests with 6 failures, exactly the repro.
• With the fix: the repro passes 6 of 6 with assertions on and off, and 60 of 60 over 10 iterations.
• With the fix, the whole lucene/highlighter module (classic, unified, match and vector highlighters, incl. the repro): 35 suites, 728 tests, 0 failures.
• ecjLint (main and test), checkGoogleJavaFormat and renderJavadoc for the highlighter module pass; tidy leaves the fix unchanged; both diffs apply cleanly to a clean a9779ef3c193 tree.
• With the fix shadowed into the main jars: the finder's (5 of 5), the independent reproducer's (10 of 10) and the devil's advocate's (5 of 5) tests pass, and the security probe's regions equal a positional twin field's exactly (150 of 150, 0 extra, 0 missing).
• Not run: other modules' suites (the change is in lucene/highlighter only); a backport in a 10.x source tree; no benchmark beyond the probe's cost case, in particular none with a large TermInSetQuery, many distinct multi-term clauses, or custom and composite queries.

The same visitor change, adding the automaton once per match.

Trade-offs

Same regions, but the cost grows with tokens x matching terms per document: 112.6 s for one 20,000-value document in the measurement above.

Make the constant-score wrapper's per-term matches return a TermQuery for the matched term from getQuery().

Trade-offs

consumeTerms would then see the matched terms, but it changes what getQuery() means for every Matches API consumer: much wider than this bug. In exchange it would report only the terms that actually matched, with no automaton to build per document.

Use OffsetsFromValues when the query reports an automaton.

Trade-offs

Simpler, but OffsetsFromValues ignores the matches and returns a range for every stored value of the field, including values that did not match (OffsetsFromValues.java:43-67), unlike the term-query behaviour on the same field.

State in OffsetsFromTokens and computeOffsetRetrievalStrategies that multi-term queries get no regions on fields without positions.

Trade-offs

No code change, but the silent missing highlights stay for anyone who misses the sentence.

Method: bounded-exhaustive translation validation on the real jars (semi-formal tier), in a hunt unit on highlighter offset sources.

• A harness compared the highlighters' offset sources with an exact reference semantics and the Matches API on every document of a small enumerated document language (1,295 single-valued documents of up to 4 positions) and a 121-query catalogue, on the stock main jars. One relation compared MatchRegionRetriever's default strategy on a DOCS-only field with every occurrence of the query's positive terms.
• That relation failed in 8,720 of 11,280 multi-term-query checks, for every constant-score multi-term query in the catalogue (a prefix, 2 wildcard, 2 regexp and 2 term-range queries), while the 2 fuzzy queries and all term queries agreed. The finder traced it to the visitor and wrote a public-API JUnit.
• Found 2026-10-26 on main ea57360912ed; then checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6) and an adjudication.