DocValuesRewriteMethod's Weight#matches looks for doc-values terms in the postings, so on a doc-values-only field newSlowSetQuery reports no match for its own hits

Queries rewritten with DocValuesRewriteMethod (SortedSetDocValuesField.newSlowSetQuery, SortedDocValuesField.newSlowSetQuery, and any MultiTermQuery with MultiTermQuery.DOC_VALUES_REWRITE) find their hits from doc values, as the rewrite's javadoc promises for "an unindexed docvalues field". But the weight's matches() hands the matching doc-values terms to DisjunctionMatchesIterator.fromTermsEnum, which looks them up in the field's postings. A doc-values-only field has none, so Weight#matches returns null for every hit, although its javadoc reserves null for documents that do not match and says a DocValues query match returns MATCH_WITH_NO_TERMS. A BooleanQuery with such a MUST or FILTER clause then has null Matches for all its hits, so the UnifiedHighlighter, which uses Weight#matches with its default flags since 9.0.0, highlights nothing; as a MUST_NOT clause the same null never excludes anything. Hits, counts and scores are correct. The code dates from LUCENE-8229 (7.4.0); newSlowSetQuery has been affected since it first shipped in 9.6.0.

• Doc values vs postings: a field can be stored as doc values (a per-document column, used for sorting, faceting and the "slow" doc-values queries) and/or as postings (the inverted index, term to documents with positions). SortedSetDocValuesField alone writes only doc values.
• DocValuesRewriteMethod (MultiTermQuery.DOC_VALUES_REWRITE): runs a term, set, prefix, wildcard, regexp, range or fuzzy query against the doc values, "against an unindexed docvalues field". SortedSetDocValuesField.newSlowSetQuery and SortedDocValuesField.newSlowSetQuery use it.
• Weight#matches(context, doc): for one document, returns Matches (which fields and term positions matched) or null if the document does not match. A match without positions, such as a DocValues query, returns MATCH_WITH_NO_TERMS. The UnifiedHighlighter (by default since 9.0.0), MatchHighlighter and monitor's HighlightsMatch use it.

1. newSlowSetQuery is new TermInSetQuery(MultiTermQuery.DOC_VALUES_REWRITE, field, values) (SortedSetDocValuesField.java:130-132, SortedDocValuesField.java:126-128 on main a9779ef3c193), rewritten to a ConstantScoreQuery over DocValuesRewriteMethod's wrapper (DocValuesRewriteMethod.java:38-40).
2. Hits come from doc values: the weight's scorerSupplier reads the SORTED_SET doc values and iterates the ordinals of the query's terms (DocValuesRewriteMethod.java:149-184). Doc 1 is found.
3. matches() comes from the postings: the override (DocValuesRewriteMethod.java:84-92) builds the same doc-values terms enum and passes it to DisjunctionMatchesIterator.fromTermsEnum, which opens Terms.getTerms(reader, field), the postings (DisjunctionMatchesIterator.java:86). For a doc-values-only field that is Terms.EMPTY (Terms.java:40-46), every seekExact fails (:90), it returns null (:100), and MatchesUtils.forField turns that into null Matches (MatchesUtils.java:113-116). The default Weight#matches, which advances the scorer and returns MATCH_WITH_NO_TERMS (Weight.java:78-95), is never used.
4. Boolean queries pass it on: a required clause with null Matches makes the whole query's Matches null (BooleanWeight.java:147-151); a prohibited clause excludes a document only when its Matches is non-null (:142-146), so a MUST_NOT clause of this kind never excludes anything. A SHOULD clause counts only when its Matches is non-null (:153-158): it is left out of its hits' Matches, the whole query's Matches is null for a hit that no other clause matches (MatchesUtils.java:63-66), and with minimumNumberShouldMatch it is not counted (:160) (read, not executed).
5. The highlighter trusts it: WEIGHT_MATCHES is on by default (UnifiedHighlighter.java:110; it is added when MULTI_TERM_QUERY, PHRASES and PASSAGE_RELEVANCY_OVER_SPEED are on, as they are by default, and the caller has not turned it off, :518-527), and FieldOffsetStrategy collects no offsets when Weight#matches of the whole query is null (FieldOffsetStrategy.java:143-150).
6. Why other shapes work: newSlowExactQuery / newSlowRangeQuery have no matches() override and get the scorer-based default. KeywordField.newSetQuery builds an IndexOrDocValuesQuery whose matches() delegates to the doc-values TermInSetQuery (IndexOrDocValuesQuery.java:154-157, TermInSetQuery.java:104-110), so it runs this very override, and works only because a KeywordField also has postings (DOCS only: terms, no positions).

3 documents in one segment, body = "fruit" (StringField), tags doc values only (apple, banana, cherry). tags:(banana), +body:fruit #tags:(banana) and tags:ban* with DOC_VALUES_REWRITE each hit doc 1, and Weight#matches is null for doc 1; +body:fruit -tags:(banana) hits docs 0 and 2, and Weight#matches is non-null for doc 1. CheckHits.checkMatches fails: "Unexpected null Matches object in doc1 for query tags:(banana)". newSlowExactQuery on the same field passes all of it. Highlighting "a ripe fruit in a basket" for +body:fruit #tags:(banana) gives no highlight; with a newSlowRangeQuery filter it gives "a ripe <b>fruit</b> in a basket".

Wider form, same root cause: when one field name has postings and doc values with different contents (a tokenized TextField plus a raw SortedDocValuesField for sorting), the postings decide matches(). With doc 0 "red apple" and doc 1 "red": for newSlowSetQuery("title", ["red apple"]) the hit doc 0 gets null Matches, and for ["red"] the non-hit doc 0 gets non-null Matches, because its tokenized postings contain "red".

• Who. Applications with a SortedSetDocValuesField or SortedDocValuesField that has no postings under the same name, queried with newSlowSetQuery (9.6.0 on) or with any MultiTermQuery using DOC_VALUES_REWRITE / DocValuesRewriteMethod (7.4.0 on), and that read Weight#matches.
• Highlighting. The UnifiedHighlighter uses Weight#matches with its default flags since 9.0.0 (opt-in WEIGHT_MATCHES from 7.5.0 to 8.11.4). Any query with such a MUST or FILTER clause gets no highlights, silently, with offsets in the postings and with analysis-time offsets (executed for a newSlowSetQuery filter on main, and for a PrefixQuery filter with DocValuesRewriteMethod in the hunt's 8.11.4 and 9.0.0 runs; the wildcard, regexp, range and fuzzy forms go through the same matches() override, read). MatchHighlighter / MatchRegionRetriever and NamedMatches get the same null (read, not executed).
• Monitor. HighlightsMatch.MATCHER reports a stored query as matching exactly when Weight#matches is non-null (HighlightsMatch.java:44-62). A stored +body:fruit -label:(restricted) with the MUST_NOT clause on a doc-values-only field is reported as matching a document labelled "restricted" (QueryMatch.SIMPLE_MATCHER correctly reports 0); with a FILTER clause instead, a real match is missed. Executed on main and 10.5.2. The MUST_NOT direction rarely shows in highlighters, which ask Weight#matches only about hits they already have (TopDocs or caller-supplied ids, MatchRegionRetriever.java:216-275, 405); it matters where non-null Matches is itself the match decision, as here.
• Default vs opt-in. Default IndexWriterConfig, codec, searcher and highlighter settings. The opt-in part is the layout and the query: a doc-values-only field queried through the slow doc-values set or multi-term path.
• Scale. Every hit, at any index size, deterministic: the hunt's independent test failed the same 5 of its 7 tests in each of 10 derived seeds with randomized codecs and IndexWriterConfigs.
• Not affected. Search hits, counts, scores and scorer-based filters; fields whose values are also in the postings (KeywordField.newSetQuery, IndexOrDocValuesQuery over an indexed field); newSlowExactQuery and newSlowRangeQuery; the numeric newSlowSetQuery methods, which use a different class.
• How likely. Rare: the newSlowSetQuery javadoc steers users to pair it with a postings query, and the recorded prior-report search found no report. Downstream engines route some doc-values-only keyword queries through DOC_VALUES_REWRITE; whether any highlights them through Weight#matches was not checked.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected non-null Matches for hit doc 0 of tags:(apple), got null
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertNotNull(Assert.java:713)
       org.apache.lucene.search.TestDocValuesSetQueryMatches.testMatchesOfHitOnDocValuesOnlyField(TestDocValuesSetQueryMatches.java:47)

• The contract. Weight.java:67-77: "Returns Matches for a specific document, or null if the document does not match the parent query. A query match that contains no position information (for example, a Point or DocValues query) will return MatchesUtils#MATCH_WITH_NO_TERMS."
• The rewrite advertises this layout. DocValuesRewriteMethod.java:28-34: "This can be used to perform these queries against an unindexed docvalues field." MultiTermQuery.java:111-121 sends users without doc values to "the other rewrite methods that rely on postings alone".
• Lucene's own oracle. CheckHits.checkMatches (CheckHits.java:352-373, MatchesAsserter :640-688), run by QueryUtils.check(random, q, searcher) (QueryUtils.java:148), requires non-null Matches for every hit; it fails here.
• Siblings comply. newSlowExactQuery / newSlowRangeQuery on the same field return MATCH_WITH_NO_TERMS. TestMatchesIterator#testPointQuery (TestMatchesIterator.java:444-468) asserts that a required point/doc-values clause next to a TermQuery still yields the term's positions.
• Why tests missed it. TestDocValuesRewriteMethod has a doc-values-only field (_with-skip, :70) but compares hit lists only (:117-133); its QueryUtils.check(a1) (:157) checks equals/hashCode. TestMatchesIterator has no DocValuesRewriteMethod case. Adding CheckHits.checkMatches to TestDocValuesRewriteMethod fails on stock main ("Unexpected null Matches object in doc11 for query _with-skip://") and passes with the fix once that test's random deletions are turned off.
• History. The override came with Weight#matches in LUCENE-8229 (2018), whose discussion proposed a Matches object meaning "a match in this doc, but no term hits" whenever the scorer matched (now MATCH_WITH_NO_TERMS); nobody discussed this layout. The same commit gave the postings-based MultiTermQueryConstantScoreWrapper a fallback to super.matches() for fields indexed without positions; this override got none.
• Regression for users (9.6.0). On 9.5.0 the sandbox DocValuesTermsQuery, which had no matches() override, passes CheckHits.checkMatches on a doc-values-only field; #12129 (@rmuir, merged 2023-02-06) graduated it to newSlowSetQuery on a new SortedSetDocValuesSetQuery, and #12175 (@gsmiller, merged 2023-03-01) moved newSlowSetQuery onto DOC_VALUES_REWRITE. Both first shipped in 9.6.0, and on 9.6.0 it fails. #12175 calls the change removing "mostly redundant code" and says nothing about Matches.

The strongest works-as-designed case, built by the intended-behaviour verifier and a review by a model from another family: Weight#matches, Matches and DocValuesRewriteMethod are @lucene.experimental; Matches is a positions API, so a field without postings has no matches; and the newSlowSetQuery javadoc recommends wrapping it in an IndexOrDocValuesQuery with a postings query.

Why it fails: experimental status allows the API to change, not to break its own documented contract. The javadoc names DocValues queries explicitly and reserves null for non-matching documents, while "no positions for this field" is what Matches#getMatches(field) == null expresses. The IndexOrDocValuesQuery note is speed advice, and the same note offers the other remedy, ANDing it "with a selective query", which is exactly the failing FILTER shape. Every verifier and the cross-model reviews rated the intended argument weak.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF); the plain super.matches() variant was run only as a root-cause probe in the hunt, the other options were not run.

Replace the body of the weight's matches() in DocValuesRewriteMethod (+13/-5): return null when the scorer-based super.matches() is null; otherwise return the postings-based term Matches if there are any, else the MATCH_WITH_NO_TERMS from super.matches(). For live documents matches() is then non-null exactly for the hits (like every Weight#matches it does not look at deletions), so this fixes the doc-values-only, MUST_NOT and mixed-field forms at once, and fields that also have postings keep their postings-based term Matches (KeywordField.newSetQuery, whose DOCS-only postings have terms but no positions; fields indexed with positions keep their positions). A second diff adds the two repro tests as new tests in lucene/core and lucene/highlighter (+279/-5 in total).

final Matches docValuesMatch = super.matches(context, doc);
if (docValuesMatch == null) {
  return null;
}
final SortedSetDocValues values = DocValues.getSortedSet(context.reader(), query.field);
final Matches termMatches =
    MatchesUtils.forField(
        query.field,
        () -> DisjunctionMatchesIterator.fromTermsEnum(
            context, doc, query, query.field, getTermsEnum(values)));
return termMatches != null ? termMatches : docValuesMatch;

Trade-offs

• Each matches() call now also runs the scorer path: it enumerates every query term that is in the segment's doc-values dictionary into an ordinal set (an array grown as needed, then a LongHashSet unless the ordinals are contiguous; DocValuesRangeIterator.java:151-183) and advances a fresh doc-values iterator to the document; for a hit it then enumerates the terms a second time for the postings lookup. Weight#matches is normally called per highlighted hit; no benchmark was run, and #13391 already reports matches() of large TermInSetQuerys as slow.
• When postings and doc values differ, the term matches still come from whichever query term the postings have for the document; only the match decision is fixed.

What the checks validated

• Stock main a9779ef3c193: the repro fails 5 of 6 with assertions on and off; the highlighter test fails 2 of 4.
• With the fix: the repro passes 6 of 6 with assertions on and with assertions off; the highlighter test passes 4 of 4.
• With the fix, the closest suites pass: in core TestDocValuesRewriteMethod, TestTermInSetQuery, TestMatchesIterator, TestIndexOrDocValuesQuery, TestPrefixQuery, TestWildcardQuery, TestRegexpQuery, TestTermRangeQuery, TestFuzzyQuery, TestDocValuesQueries, TestMultiTermQueryRewrites, TestKeywordField, TestSortedSetDocValuesSetQuery; all of uhighlight and matchhighlight; all of lucene/monitor: 62 classes, 848 tests, 3 skipped, 0 failures.
• ecjLint and checkGoogleJavaFormat pass for core and highlighter with both repro tests in place.
• Earlier, with the fix shadowed onto the stock jars: the finder's tests, the independent reproducer's tests including the mixed-field form, the GPT-stage test and the MUST_NOT plus monitor probe (10 of 10) pass.
• Not run: the full core and highlighter suites; no benchmark.

Let the scorer-based default answer: MATCH_WITH_NO_TERMS or null.

Trade-offs

Simplest and always consistent with the hits (for live documents), but every field that also has postings gets MATCH_WITH_NO_TERMS with no term detail, and fields indexed with positions lose their positions; that includes KeywordField.newSetQuery, whose IndexOrDocValuesQuery asks this weight (its postings are DOCS only). Whether this changes any highlighter's output was not tested. Run only as a root-cause probe in the hunt (7 of 7 pass, membership only).

If terms(field) is null, return super.matches(); otherwise keep today's code.

Trade-offs

Smaller, but postings still decide the match, so the mixed-field form (postings with different contents than the doc values) stays wrong.

Call CheckHits.checkMatches in TestDocValuesRewriteMethod#assertSame for the doc-values queries, and/or add a DocValuesRewriteMethod case to TestMatchesIterator.

Trade-offs

Probed: the added checks fail on stock main ("Unexpected null Matches object in doc11 for query _with-skip://") and pass with the recommended fix, but only with the test's random deletions turned off. As written the test deletes documents, and the non-hit half of checkMatches then fails on stock main too ("Unexpected non-null Matches object in non-matching doc121", on the field that also has postings, with or without the fix), so the check needs an index without deletions or a separate test method.

Method: bounded-exhaustive translation validation on the stock jar (semi-formal tier), in a hunt unit on multi-term query rewrites.

• The harness gives every MultiTermQuery (term range, prefix, set, wildcard, regexp, fuzzy) an independent reference semantics for its term language and its hits, generated from the document layout rather than read from the index, and checks it on real multi-segment indexes with a field that has postings plus SORTED_SET doc values and a field with doc values (and a skip index) only, under every rewrite method.
• One of its checks (MATCH: Weight#matches non-null if and only if the document is a reference hit, for every live document) fired only for DOC_VALUES_REWRITE on the doc-values-only field: 511,327 cases at each of the three maxClauseCount settings, and no other rewrite or field.
• The hypothesis was first seen while reading DocValuesRewriteMethod during modelling and checked with a scratch probe before the harness ran, so this is a hypothesis formed while modelling and witnessed by a tool, not the output of a model checker, SMT solver or prover.
• Found 2026-10-26 on main ea57360912ed; then checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6), an adjudication and a skeptical-committer audit.