MatchHighlighter.highlight(topDocs, q1, q2, ...) can silently drop the highlights of a field that only a later query matches (regression in 10.0.0)

MatchHighlighter.highlight(TopDocs, Query...) highlights several queries over the same hits, for example the title and body clauses of a search, so that each query can get its own markers. It runs one MatchRegionRetriever per query (9.x did too); since 10.0.0 each retriever loads from stored fields only the always-fetched fields plus the fields its own query touches. Each document's DocHit is created by the first query's callback and copies that query's field values only; later queries add their match ranges but not their values, and the output is built from the values alone. So a field that only the second or a later query matches is silently missing from DocHighlights.fields (unless it is always fetched or the first query also touches it), the result depends on the order of the queries, and a document that the first query does not match can come back with no fields. Hits, counts and scores are unaffected. 9.x loaded every field that any query matched. Introduced by #12881 (755de5aae33, 2023-12-07), first released in 10.0.0, unfixed through 10.6.0 (10.6.0 by reading) and main; the fix keeps the union of the values every query loaded (+12/-1).

• MatchHighlighter: the highlighter of the matchhighlight package, built on the Matches API. highlight(TopDocs, Query...) returns one DocHighlights per hit, a map from field name to formatted values, built by a chain of field highlighters such as FieldValueHighlighters.highlighted(...) (format a field that has matches) and skipRemaining() (omit the rest).
• Several queries at once: you pass the clauses of the search query separately, highlight(topDocs, q1, q2); every match range remembers its query (QueryOffsetRange.query), so a custom formatter can give each query its own markers. The class's own test does this with the sub-clauses of its search query.
• MatchRegionRetriever: runs one query over the hits, reads each hit's stored fields and computes the match ranges per field. It hands each document to a callback together with a FieldValueProvider, the field values it loaded.

1. highlight builds one retriever per query, with fieldsToLoadUnconditionally = fieldsAlwaysReturned::contains (MatchHighlighter.java:211, :224-231); 9.x had the same loop.
2. Each retriever loads a stored field only if it is always-fetched or one of the fields its own query visits (MatchRegionRetriever.java:180-192, :200-204). It reads the stored document of every hit, matched or not (:357-376), and calls the callback for each (:382-388).
3. The first query's callback finds no DocHit yet and creates one, new DocHit(docId, fieldValueProvider) (MatchHighlighter.java:240-244); the constructor copies that provider's fields (:170-175).
4. Every callback then calls docHit.addMatches(q, hits) (:245, :177-184): later queries add their ranges, and their providers' values are dropped.
5. computeDocFieldValues iterates docHit.fieldValues only (:257). body's ranges from body:bar sit in matchRanges, but body has no values, so no field highlighter is ever asked about it.

Before #12881 (9.x), DocHit loaded the document once, after all queries, with a visitor that accepted matchRanges.containsKey(field) || needsField.test(field): every field that any query matched.

Doc 0 {title: foo, body: bar}, doc 1 {title: baz, body: bar}, queries title:foo and body:bar, the stock highlighted then skipRemaining chain:

An executed probe drove the second query's own retriever: for doc 0 it reports loaded=[body] and hits={body=[[from=0, to=3]]}. The second query loads the field and finds the match; MatchHighlighter keeps the match range but drops the loaded value, so the range is never used.

• Who. Applications that call MatchHighlighter.highlight(TopDocs, Query...) with two or more queries that touch different fields, for example the title and body clauses of the search query passed separately to tell their markers apart (the pattern of the class's own test). Fields with offsets in postings and positions-only fields are both hit (executed).
• Default vs opt-in. MatchHighlighter itself is opt-in; nothing else needs a non-default setting: default codec, IndexWriterConfig, searcher and the stock FieldValueHighlighters.
• Hidden when. The field is always fetched: alwaysFetchFields(...), or a maxLeadingCharacters or verbatimValue highlighter in the chain that covers it (their alwaysFetchedFields are added to the always-fetched set); or the first query also touches that field. Queries on the same field are correct (executed controls).
• Effect. Silent missing highlight fields; the result depends on the argument order; a document that the first query does not match can come back with no fields. highlight only reads topDocs, so hits, counts and scores do not change (read from the code). A bounded probe (64 documents, 3 fields, 400 ordered lists of 2 or 3 queries, 4 stock highlighter chains, one segment; 9,600 highlight calls, 187,200 document comparisons, with and without assertions) only ever found fields missing: never a field beyond the combined query's, a value different from the same separate queries with every field force-loaded, or a changed document order. It did not check scores.
• Scale. Deterministic, any index size, one document is enough. In the hunt's bounded-exhaustive model, "separate queries give the same fields as their disjunction" failed in 12,221 of 20,130 document comparisons at the small scope (183 documents) and in 136,554 and 132,594 of 180,510 at the full scope (1,641 documents, two position gaps).
• How likely. Rare: the package is far less used than the UnifiedHighlighter (code search found it in none of Solr, Elasticsearch and OpenSearch), the multi-query form is a niche within it, and nothing has been reported since 10.0.0 (2024-10). The class's author wrote on dev@ in 2025-11 that they "use it with Solr a lot" and that "what was missing for me was highlighting multiple queries at once".

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<{title=[>foo<], body=[>bar<]}> but was:<{title=[>foo<]}>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.matchhighlight.TestMatchHighlighterMultiQueryFields.testLaterQueryOnAnotherFieldIsHighlighted(TestMatchHighlighterMultiQueryFields.java:58)

• The method collects ranges for each query. MatchHighlighter.java:223: "Collect match ranges for each query and associate each range to the origin query."
• Field highlighter contract. FieldValueHighlighters.highlighted: "Highlights fields matching predicate matchFields only if they contained query matches" (FieldValueHighlighters.java:89). FieldValueHighlighter: highlighters "are probed whether they are applicable to a particular combination of (field, hasMatches) pair", with hasMatches "true if the field has a non-empty set of match regions" (MatchHighlighter.java:53-67). body has match regions and is never probed.
• The class's own test uses the pattern. "Note the query passed to the highlighter are individual sub-clauses of the Boolean query used to fetch documents" (TestMatchHighlighter.java:757-763); QueryOffsetRange.query exists to tell the queries apart (MatchHighlighter.java:150-157).
• Regression. 9.x loaded every field with matches from any query; 9.0.0, 9.9.0, 9.9.1, 9.10.0 and 9.12.3 pass (executed).
• No announced change. #12881's CHANGES entry (10.0.0 API Changes) and MIGRATE note describe performance work and "minor API changes"; its PR body, comments and reviews discuss performance only.
• Dropping the values saves no reads. The later query's retriever has already read the stored document and loaded the field before it is dropped, so dropping it saves no stored-field I/O. #12881's narrowing itself (each retriever loads only the fields its query affects) is real performance work, and the recommended fix keeps it.
• Order dependence. highlight(topDocs, title, body) and highlight(topDocs, body, title) return different fields for the same set of queries.
• Author's intent. The class javadoc says "fully featured, ready-to-use component"; on dev@ (2025-11) its author wrote "This one - while not properly documented - is fully functional" and named highlighting multiple queries at once as what had been missing for them.

The strongest works-as-designed case (the GPT intended-lens review and the devil's-advocate verifier): the class javadoc calls it "An example highlighter" and advises assembling your own from the building blocks; highlight(TopDocs, Query...) has no javadoc, so nothing promises that separate queries equal their disjunction; the only upstream multi-query test calls itself "interesting, if not always super-practical" and keeps both queries on one field; #12881 narrowed field loading on purpose ("so that stored fields of a document are accessed once"); and alwaysFetchFields exists.

Why it fails: the same javadoc sentence calls the class "fully featured, ready-to-use"; it is public and not @lucene.experimental. An order-dependent result for a varargs set of queries cannot be a specification. The narrowing is correct inside MatchRegionRetriever, whose FieldValueProvider documents which fields it exposes; the loss is only in MatchHighlighter's composition, which keeps the later query's ranges but drops the values that query already loaded. alwaysFetchFields is a workaround that loads the field for every hit and is nowhere documented as required. Every verifier judged the bug real and the adjudication confirmed it with high confidence.

What it does not settle: the contract is implied, not stated word for word, and maintainers could choose to document a restriction instead; the 9.x behaviour, the author's stated use case and the small fix make that unlikely.

Option A was applied in a scratch worktree at a9779ef3c193 and run through gradle; option B was run with shadowed sources on main; C was not run.

Workarounds today: call alwaysFetchFields(...) for every field the queries can highlight (it loads them for every hit, matched or not), or pass one combined query (which loses the per-query markers).

In MatchHighlighter, DocHit gets an addFieldValues(provider) that adds each loaded field with putIfAbsent; the constructor calls it, and the callback calls it for every later query that reaches an existing DocHit (+12/-1). It uses values every retriever has already loaded: no extra stored-field access, one putIfAbsent per loaded field per later query. All retrievers read the same stored document, so the kept values are identical, and the union is the field set a retriever for the combined query loads. A second diff adds the repro as a regression test; a PR also needs a CHANGES entry.

    DocHit(int docId, MatchRegionRetriever.FieldValueProvider fieldValueProvider) {
      this.docId = docId;
      addFieldValues(fieldValueProvider);
    }

    void addFieldValues(MatchRegionRetriever.FieldValueProvider fieldValueProvider) {
      for (var fieldName : fieldValueProvider) {
        fieldValues.putIfAbsent(fieldName, fieldValueProvider.getValues(fieldName));
      }
    }
    // in highlight(): if (docHit == null) { ... } else { docHit.addFieldValues(fieldValueProvider); }

Trade-offs

Field order: DocHighlights.fields is a LinkedHashMap, and with this fix its order follows the queries: highlight(topDocs, body, title) gives {body=[>bar<], title=[>foo<]}, while the combined query and 9.12.3 give stored order, {title=[>foo<], body=[>bar<]} (executed probe). Fields and values are the same. If maintainers want the stored order kept, use option B.

What the checks validated

• Stock main, gradle: the repro fails 2 of 2 with assertions on and off; the whole highlighter module: 724 tests, 2 failed (only the repro).
• Fix, gradle: the repro passes 2 of 2 with assertions on and off and 20 of 20 over 10 iterations.
• Fix, the whole lucene/highlighter module (classic, unified, match and vector highlighters, with the repro): 35 classes, 724 tests, 0 failures.
• ecjLint, checkGoogleJavaFormat, renderJavadoc for lucene/highlighter: pass. Both diffs apply cleanly to a clean a9779ef3c193 tree.
• Earlier, by the hunt: its equivalent change (the same code, a shorter comment) shadowed into main makes the finder's and the independent reproducer's tests pass, the 48 upstream matchhighlight tests pass, and in the security probe all 187,200 document comparisons get the combined query's field set, with no extra field or value.
• Not tested: a custom field highlighter that also formats fields without matches and declares no alwaysFetchedFields. By reading, the fix then also emits fields that a later query touches but does not match in that document; the combined query does the same on main, 9.x did not (it loaded only fields with matches). The same holds for option B.
• The diffs also apply cleanly (git apply --check) to MatchHighlighter.java on branch_10x and branch_10_6; not compiled or tested there. Those branches build with Java 21, so the test's _ -> true must become f -> true.
• Not run: other modules' suites (the change is in lucene/highlighter only); no benchmark.

Rewrite all queries first, collect the highlighted fields they visit (the same QueryVisitor rule as MatchRegionRetriever), and pass "always fetched or affected by any query" as fieldsToLoadUnconditionally to every retriever (+24/-3).

Trade-offs

For the tested chain (highlighted, then skipRemaining()), it gives the 9.12.3 and combined-query output including the stored field order ({title=[>foo<], body=[>bar<]} in both argument orders, executed probe). Costs more: every retriever copies every query's fields out of the stored document, so with n queries on n fields about n times the field values of option A (each retriever already reads the stored document once per hit). Not measured. Not tried: cheaper designs that also keep stored order, such as loading the stored fields once after all queries (as 9.x did) or letting only the first retriever preload the fields any query affects.

What the checks validated

Run with the standalone JUnit runner and shadowed sources on main (not gradle): the repro passes 2 of 2, the 48 upstream matchhighlight tests pass.

State that separate queries must stay on the first query's fields, or that callers must alwaysFetchFields every highlighted field.

Trade-offs

No code change; the order-dependent silent loss stays for anyone who misses the sentence, and 9.x users upgrading lose highlights without notice. Not run.

Method: bounded-exhaustive translation validation on the real jars (semi-formal tier), in a hunt unit on overlapping and nested match regions in the matchhighlight package.

• Reference semantics written from the javadoc and class comments were compared with the stock classes on every input up to a small scope: a vocabulary of three words, one- to four-word values, 21 queries, every offsets strategy, PassageSelector and PassageFormatter, and MatchHighlighter end to end.
• Metamorphic relation E5: highlight(topDocs, q1 on field A, q2 on field B) must give the same highlighted fields as highlight(topDocs, q1 OR q2). The finder spotted the defect by code reading while modelling MatchHighlighter, and E5 witnessed it: 12,221 of 20,130 document comparisons differ at the small scope (183 documents, with assertions), 136,554 and 132,594 of 180,510 at the full scope (1,641 documents, two position gaps).
• The finder then reduced it to a deterministic public-API JUnit; an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6) and an adjudication checked it (2026-10-26).