Payload-filtered Intervals.term reports Weight#matches for documents it does not match, so a MUST_NOT clause hides its own query's hits from highlighting

Intervals.term(String|BytesRef, Predicate<BytesRef>) builds a source that matches a term only at positions whose payload passes the filter. Searching with it is correct: the scorer applies the filter. But its matches(...) returns an iterator as soon as the document contains the term and applies the filter only inside next(). For a document where the filter rejects every position, IntervalQuery's Weight#matches therefore returns a non-null, empty Matches for a document the query does not match.

Code that reads Weight#matches trusts the non-null value. A BooleanQuery that excludes such a clause with MUST_NOT returns null Matches for its real hits, so the default UnifiedHighlighter returns them unhighlighted; Intervals.or, a required clause and a SHOULD clause with minimumShouldMatch over-report; the monitor module's HighlightsMatch.MATCHER and NamedMatches report matches that do not exist. Hits, counts, scores and explain() are correct. Every release from 8.1.0 (2019) to 10.6.0 and main is affected; only applications that use the payload-filter overload, which no Lucene query parser builds, are hit.

1. The factory. Intervals.term(String, Predicate<BytesRef>) (Intervals.java:82-84) delegates to the BytesRef overload, which returns new PayloadFilteredTermIntervalsSource(term, payloadFilter) (:90-92).
2. matches hands out an iterator whenever the term is in the document. matches(field, ctx, doc) (PayloadFilteredTermIntervalsSource.java:151-174) seeks the term and calls the private matches(te, doc) (:181), whose only null return is if (pe.advance(doc) != doc) { return null; } (:183-185). The iterator starts with upto = pe.freq() (:198) and applies the filter only in next(): } while (filter.test(pe.getPayload()) == false); (:202-212). If every position is rejected, the first next() returns false, but the iterator is already out.
3. Non-null means "hit". IntervalQuery's Weight#matches returns null only when the source does (IntervalQuery.java:170-186), and MatchesUtils.forField calls the supplier "eagerly to work out if we have a hit or not", returning a Matches for any non-null iterator (MatchesUtils.java:106-117).
4. The scorer is right. The scoring iterator applies the filter in nextInterval() (PayloadFilteredTermIntervalsSource.java:117-126) and IntervalScorer's two-phase check rejects a document with no first interval (IntervalScorer.java:80-81), so hits, counts and scores are correct.
5. Where it spreads. BooleanWeight.matches returns null when a prohibited clause has non-null Matches (BooleanWeight.java:142-146), keeps a required clause's (:147-152) and counts a SHOULD clause's towards minimumShouldMatch (:153-162). DisjunctionIntervalsSource.matches keeps every non-null sub-iterator (DisjunctionIntervalsSource.java:86-97), and two more wrappers hand the child's iterator on unchanged: fixField (FixedFieldIntervalsSource.java:44-47) and the not* sources (notContaining, notContainedBy, nonOverlapping, notWithin) when the subtrahend is absent from the document (DifferenceIntervalsSource.java:58-61); on main, fixField(f, PF(dog)) and notContaining(PF(dog), term(cat)) give non-null Matches for the non-hit. Monitor's HighlightsMatch.MATCHER adds a match whenever Weight#matches is non-null (HighlightsMatch.java:44-62); NamedMatches names a clause whenever its Matches is non-null (NamedMatches.java:105-110). The UnifiedHighlighter uses Weight#matches by default (DEFAULT_ENABLE_WEIGHT_MATCHES = true, UnifiedHighlighter.java:110).
6. Why conjunctions hide it. Composite sources that can drop their children's intervals check for a first interval before returning: IntervalMatches.asMatches (IntervalMatches.java:34-36; used by maxwidth/maxgaps, extend, OffsetIntervalsSource behind before/after, and the not* sources when the subtrahend is present), ConjunctionIntervalsSource.java:86-88, MinimizingConjunctionIntervalsSource.java:72-74, MinimumShouldMatchIntervalsSource.java:88-89, and core's PhraseWeight (PhraseWeight.java:115-117); on main, maxwidth(1, PF(dog)), extend(PF(dog), 0, 0) and notContaining(PF(dog), term(dog)) give null Matches for the non-hit. This leaf, the only leaf source that filters positions, can drop every position but does not check. The iterator skeleton was copied from TermIntervalsSource.matches (TermIntervalsSource.java:164-194), where laziness is safe because every position of a plain term is an interval.

A plain main() probe without the test framework (only lucene-core and lucene-queries, production default codec Lucene104); doc 0 = big|adj dog|verb, doc 1 = big|adj dog|noun (the reverse of the step-by-step example's numbering); PF(dog) = "dog with payload noun"; main 645d4fe14a20, assertions on and off:

Also executed:

• Highlighting: +f:red -IntervalQuery(f, PF(fox)) over "red fox" (fox a verb) and "red cat", offsets in the postings, UnifiedHighlighter builder defaults: expected <b>red</b> fox and <b>red</b> cat, got red fox and <b>red</b> cat (main 645d4fe14a20, 10.5.2, 9.12.3).
• minimumShouldMatch: (f:red PF(fox))~2 hits {a, c} but has Matches for {a, b, c}, where b has fox only as a verb (main ea57360912ed, 10.5.2, 9.12.3).
• Monitor: for a stored IntervalQuery(f, PF(dog)) and the document big|adj dog|verb, QueryMatch.SIMPLE_MATCHER reports 0 matches and HighlightsMatch.MATCHER 1; the same as a required clause, inside or(), in a two-document batch and with Presearcher.NO_FILTERING; as a MUST_NOT clause a real match is missed. NamedMatches gives [big, dogAsNoun] instead of [big] for a hit.
• Lucene's own oracle: CheckHits.checkMatches fails on the lone query and on the MUST_NOT query.
• Controls: maxwidth(1, PF(fox)) returns null Matches for the non-hit; a document with an accepted payload gets correct Matches.

• What goes wrong. For any document whose term occurs only with rejected payloads, the very case a payload filter exists for: false Matches for the lone source, or(), a required clause and SHOULD with minimumShouldMatch; lost Matches, and so lost highlights, for the hits of a query that excludes the source with MUST_NOT; false and missed matches in monitor's HighlightsMatch.MATCHER; a wrong clause name from NamedMatches.
• Silent, narrow. No exception or warning. Hits, counts, scores and explain() are correct, and nothing is written to the index.
• No special setup. Default IndexWriterConfig, codec and searcher; the plain-Java probe fails with the production default codec and no test framework.
• Who is exposed. The payload-filter overload is opt-in and Java-only. LUCENE-8694 describes the source as "the intervals equivalent of SpanPayloadCheckQuery", and open issue #16820 proposes deprecating span queries in favour of intervals, so span-payload users may move to it.
• Highlighter note. Without offsets in the postings, the UnifiedHighlighter re-analyses the text into a MemoryIndex without payloads, and payload-filtered interval queries then fail with a separate bug's IllegalArgumentException ("no indexed payloads") before this one shows.
• Likelihood: rare. A visible symptom needs a Matches consumer, and the public searches found no report since the API shipped in 2019. The intervals package "contains experimental classes" (package-info.java:21), and Weight#matches is @lucene.experimental (Weight.java:76).

The token is built with the test framework's Token and CannedTokenStream, so only the queries module is needed. assertEquals(0, searcher.count(query)) is the precondition (the document is not a hit); the failing line is the assertNull. It fails the same way with seeds DEADBEEF and 5EED1234, and in 20 of 20 iterations with a third seed; a plain-Java probe without the test framework shows the same result with the production default codec.

The file fails unchanged on 9.1.0 to 10.6.0; releases before 9.1.0 need only the old test-framework import paths (and, for 8.1.x, the sandbox package org.apache.lucene.search.intervals).

TestPayloadFilteredIntervalsMatches > testNoMatchesWhenFilterRejectsEveryPosition FAILED
    java.lang.AssertionError: Weight#matches for non-matching doc 0 expected null, but was:<org.apache.lucene.search.MatchesUtils$3@6fff5f99>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:E83CA871939DB2FA]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotNull(Assert.java:756)
        at junit@4.13.2/org.junit.Assert.assertNull(Assert.java:738)
        at org.apache.lucene.queries.intervals.TestPayloadFilteredIntervalsMatches.testNoMatchesWhenFilterRejectsEveryPosition(TestPayloadFilteredIntervalsMatches.java:53)

• Two javadoc contracts. IntervalsSource#matches: "Returns null if no intervals exist in the given document and field" (IntervalsSource.java:47-57). Weight#matches: "Returns Matches for a specific document, or null if the document does not match the parent query" (Weight.java:67-78). Matches#getMatches: "or null if there are no matches in that field" (Matches.java:35-37). The factory defines the source's intervals as the term's positions "filtered by the value of the term's payload at each position" (Intervals.java:78-92).
• Older than the class. Both javadoc sentences were already in place at the introducing commit 5ca25249274.
• Test oracles. CheckHits.MatchesAsserter asserts "Unexpected non-null Matches object in non-matching doc" (CheckHits.java:640-682), and TestIntervals.checkIntervals asserts "Expected null matches iterator on doc" (TestIntervals.java:180). Both fail on this source.
• Siblings and scorer. Every composite source that can drop its children's intervals, and core's PhraseWeight, return null when there is no first match; pass-through wrappers (or(), fixField, the not* sources when the subtrahend is absent from the document) cannot drop intervals, hand the child's iterator on unchanged and rely on the leaf's null. This is the only leaf source that filters positions, so the only one that can drop every position, and the source's own scorer rejects the document.
• Why tests missed it. TestPayloadFilteredInterval, the only upstream test of the class, calls matches() only on a document with accepted positions (TestPayloadFilteredInterval.java:79-87), and no other test in the tree uses the payload-filter overload. The LUCENE-8694 issue never mentions Matches, and no CHANGES entry weakens the contract.

The strongest case for "intended": IntervalIterator says "it is possible for a document to return NO_MORE_INTERVALS on the first call to nextInterval()" (IntervalIterator.java:35-36), so empty iterators are a known state; Weight#matches and Matches are @lucene.experimental; the document does contain the term; and a lazy iterator avoids reading positions up front.

Why it fails: that note is about the approximation the scorer then filters, while IntervalsSource#matches reserves null for exactly this case; "experimental" warns about API changes, not a waived return value; MatchesUtils.forField and BooleanWeight.matches read non-null as "hit", so the empty iterator changes other queries' answers; and the eager check reads no more positions than the scorer already reads to decide the hit. Verdict: not intended, a defect (the independent reproduction, the cross-model review, the intended-behaviour check, the adjudication and the tie-break).

Two fixes were built and run through gradle in a scratch worktree of 645d4fe14a20. Both return null from matches when the filter rejects every position and stay lazy after the first accepted position. The asMatches fix is smaller and reuses an existing idiom of the package; the look-ahead fix, the hunt's, keeps the check inside the leaf.

In PayloadFilteredTermIntervalsSource.matches(String, LeafReaderContext, int), replace return matches(te, doc); (PayloadFilteredTermIntervalsSource.java:173) with the lines below. IntervalMatches.asMatches advances the wrapped iterator to the first interval and returns null when there is none (IntervalMatches.java:26-36); FilteredIntervalsSource.matches (FilteredIntervalsSource.java:111-125) and OffsetIntervalsSource.matches (OffsetIntervalsSource.java:137-144) use the same pair of helpers.

    IntervalMatchesIterator mi = matches(te, doc);
    if (mi == null) {
      return null;
    }
    // the payload filter is applied lazily in next(): return null if it rejects every position
    return IntervalMatches.asMatches(IntervalMatches.wrapMatches(mi, doc), mi, doc);

Trade-offs

• Positions are now read eagerly up to the first accepted one: the same work the scorer's two-phase check does to decide the hit. For a document with only rejected positions it reads all of them, as the scorer does.
• Two small wrapper objects per matches call: the IntervalIterator from wrapMatches and the iterator that asMatches returns (IntervalMatches.java:37, :99-100).
• The wrapper's getQuery() still delegates to the leaf's iterator, which throws UnsupportedOperationException (a separate issue, LFV-463, that neither fix changes); IntervalQuery overrides getQuery() itself (IntervalQuery.java:179-183).

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: the repro fails on stock and passes with the fix, assertions on and off.
• org.apache.lucene.queries.intervals.* (including the repro): 118 tests, 0 failures.
• The whole :lucene:queries:test: 513 tests, 5 skipped, 0 failures.
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest of :lucene:queries pass with the fix and the repro test.
• Shadowed into the 645d4fe14a20 jars: the independent reproduction's 7 tests, the highlight test, the 9 monitor and NamedMatches tests and the tie-break probe all pass.
• The finder's test and the intended-behaviour check's 3 oracle tests, shadowed with the fix into the 645d4fe14a20 jars, assertions off: fail on stock and pass with the fix (re-run by the fact check).

In the private matches(te, doc), read positions until the first one whose payload passes the filter before building the iterator, return null if there is none, and let the iterator report that position first and then continue lazily. Bounded memory: an earlier draft buffered all positions, and the cross-model review's O(freq) note replaced it with this one-position look-ahead.

Trade-offs

No wrapper object and the check stays inside the leaf, but more code to review; the first position is carried in two extra fields. The same eager read as the asMatches fix.

What the checks validated

• Gradle, the same scratch worktree: the repro passes; org.apache.lucene.queries.intervals.*: 118 tests, 0 failures; checkGoogleJavaFormat and ecjLintMain pass.
• Earlier: the finder's, the independent reproduction's, the intended-behaviour check's, the adjudication's, the highlight and the monitor/NamedMatches tests pass with it, and so do the 117 upstream intervals tests (main ea57360912ed); the minimal repro and the tie-break tests pass with it on 645d4fe14a20.

For example in MatchesUtils.forField or DisjunctionIntervalsSource.matches. Not built.

Trade-offs

forField serves many queries and would have to consume the first match and replay it; fixing or() alone leaves the lone source and BooleanQuery wrong. The source is the one place that breaks the contract.

Method: non-formal (a hunt unit, intervals builder and multi-term sources).

• The unit's harnesses cross-check that Weight#matches is non-null if and only if the scorer matched. They do not cover payload-filtered leaves, so the finder applied the check by hand in a probe: two documents big|adj dog|noun and big|adj dog|verb, the payload-filtered term under 12 parent operators, plus +big -<source>.
• The probe showed Matches for the non-matching doc 1 of the lone source and null Matches for the hit doc 1 of +big -<source>; hit counts were correct under all 12 operators.
• The finder, an AI tool, then wrote the public-API JUnit; the one-test repro was cut from it on 2026-10-29. This is testing, not a bounded-exhaustive or model-checking run.