Payload-filtered Intervals.term sources compare equal whatever their filter, so ordered, or, BooleanQuery and the query cache merge them

Intervals.term(String|BytesRef, Predicate<BytesRef>) returns a source that matches the term only at positions whose payload passes the filter. Its hashCode is Objects.hash(term) and its equals compares only the term, so two sources on the same term with different filters are equal, and so are the IntervalQuery objects built from them.

Lucene removes "duplicate" parts by equality while it builds and rewrites queries, so the two filters are merged silently: ordered() and unordered() turn the pair into a repeat of the first filter, Intervals.or keeps one of them, BooleanQuery collapses two FILTER (or MUST) clauses into one and rewrites MUST x + MUST_NOT y to MatchNoDocsQuery, DisjunctionMaxQuery uses the first filter for both disjuncts, and a query cache serves one filter's hits for the other. Results are wrong in both directions, with default settings and any index size; the smallest case is one document on which ordered(a|x, a|y) finds 0 hits instead of 1.

The term-only equality dates from LUCENE-8694 (8.1.0, 2019), written with a wrong cast, so up to 9.8.0 the same calls threw ClassCastException. #12827 (9.9.0) fixed the cast only, which made the wrong equality reachable. Affected: 9.9.0 to 10.6.0 and main. Only applications that use the payload-filter overload with two different filters on one term are hit; no Lucene query parser builds these sources.

1. The filter decides which positions match. Intervals.term(BytesRef, Predicate<BytesRef>) returns new PayloadFilteredTermIntervalsSource(term, payloadFilter) (Intervals.java:86-92; the String overload at :78-84 delegates to it). Its interval iterator skips every position whose payload fails the filter: } while (filter.test(pe.getPayload()) == false); (PayloadFilteredTermIntervalsSource.java:117-126; the matches iterator at :202-212).
2. Equality ignores the filter. hashCode() is Objects.hash(term) (:256-259) and equals ends in return Objects.equals(term, that.term); (:261-267). toString() is PAYLOAD_FILTERED(a) for both (:269-272), so the merge is invisible in a printed query too.
3. ordered() turns equal neighbours into a repeat. OrderedIntervalsSource.deduplicate groups consecutive sources with current.get(0).equals(source) and replaces each group with RepeatingIntervalsSource.build(current.get(0), n) (OrderedIntervalsSource.java:29-57). ordered(a|x, a|y) becomes "two occurrences of a|x"; the document has one, so 0 hits. Repeats exist so that "multiple iterators do not match on a single term" (RepeatingIntervalsSource.java:33-41), which is right only for sources that match the same positions.
4. The same equality is used everywhere else. unordered() counts sources in a LinkedHashMap (UnorderedIntervalsSource.java:44-57); Intervals.or collects them in a HashSet (DisjunctionIntervalsSource.java:43-68); IntervalQuery.equals/hashCode delegate to the field and source (IntervalQuery.java:134-146); BooleanQuery keeps FILTER and MUST_NOT clauses in HashSets (BooleanQuery.java:126-140), drops duplicates on rewrite (:343-362), returns MatchNoDocsQuery("MUST clause also in MUST_NOT") (:364-381), removes a FILTER equal to a MUST (:387-394), merges duplicate MUST clauses (:450-455, deduplicateClauses at :714) and duplicate SHOULD clauses only when minimumShouldMatch is at most 1 (:442-447); two SHOULD clauses with minimumShouldMatch 2 are first rewritten to two MUST clauses (:577-588) and then merged. DisjunctionMaxQuery keeps its disjuncts in a Multiset (DisjunctionMaxQuery.java:47) that keeps the first of two equal keys with a count of 2 (Multiset.java:41-66, :79-84), so its weights use the first filter twice (DisjunctionMaxQuery.java:115-117). A query cache looks entries up by equality.

Adding the filter to equals/hashCode, and nothing else, makes the repro and every other executed shape pass, so the equality is the cause.

The hunt's 7-test JUnit (payload pos: N at position N; documents first = a x, second = x a, both = a a; a@0 and a@1 filter for pos: 0 and pos: 1; each filter alone returns its correct documents), on every release from 9.9.0 to 10.5.2, and on main with assertions on and off:

Also executed by the verifiers: unordered(a@0, a@1) returns nothing; two MUST clauses, and two SHOULD clauses with minimumShouldMatch 2, over-match; a DisjunctionMaxQuery of the two returns [both, first], missing second. By reading, atLeast is affected only when minShouldMatch equals the number of sources (it then delegates to unordered, Intervals.java:566-569); with fewer it builds a MinimumShouldMatchIntervalsSource, which does not de-duplicate (:577). atLeast was not executed.

• Silent, both directions. Missing hits (ordered, unordered, or, MUST + MUST_NOT, DisjunctionMaxQuery) and extra hits (two FILTERs, two MUSTs, SHOULD with minimumShouldMatch 2, the cache), with no exception.
• Default settings, any size. The in-query shapes need no cache and no configuration: default IndexWriterConfig under Lucene's randomized test framework, a plain IndexSearcher, one document. The minimal test fails in 20 of 20 iterations with derived seeds in one run (a random directory each time, one class-level codec) and with seed DEADBEEF under another codec.
• The cache shape. 9.9.0 to 10.6.0 ship a default LRUQueryCache; it was executed on stock 9.9.0, 9.12.3 and 10.5.2 searchers (a 12,000-document segment, the query seen 8 times), and 10.6.0 has the same default (read). By reading 10.5.2, the default cache skips segments under 10,000 documents and its policy caches such a query after 5 sightings. On main caching is off by default (IndexSearcher.java:82-84, GITHUB#14187), so the application must install a cache.
• Who is exposed. The payload-filter overload is opt-in. A natural trigger is or(term("bank", isNoun), term("bank", isVerb)), or the same term filtered per request. The flexible query parser's interval functions use only the one-argument Intervals.term, and no in-tree code outside tests calls the overload. A read-only public code search (2026-10-26) found only one-argument Intervals.term calls in Elasticsearch, OpenSearch and Solr. The #12827 author wrote that they could not wrap payload-filtered sources in a MinimumShouldMatchIntervalsSource because UnorderedIntervalsSource.deduplicate() called this equals, so such combinations do occur; whether their filters differed is not stated.
• Likelihood: rare. The recorded public searches found no report of wrong results since 9.9.0. The intervals package "contains experimental classes" (package-info.java:21).

The test builds the two tokens with the test framework's Token and CannedTokenStream, so it needs only the queries module. The filters are new BytesRef("x")::equals and new BytesRef("y")::equals. On main a plain IndexSearcher has no query cache, so the failure comes from ordered() alone. It fails in 20 of 20 iterations with derived seeds in one run (a random directory each time, one class-level codec) and with seed DEADBEEF under another codec. That each filter alone finds the document is not asserted, to keep one assertion; the hunt's control test for that passes on every version.

The same file compiles and fails unchanged on the 9.9.0, 9.12.3, 10.5.2 and 10.6.0 jars; on 9.8.0 it throws ClassCastException from PayloadFilteredTermIntervalsSource.equals, the pre-#12827 form.

TestPayloadFilteredTermOrdered > testOrderedOfSameTermWithDifferentPayloadFilters FAILED
    java.lang.AssertionError: hits of ordered(a|x, a|y) expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:80F198F3455C249A]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.queries.intervals.TestPayloadFilteredTermOrdered.testOrderedOfSameTermWithDifferentPayloadFilters(TestPayloadFilteredTermOrdered.java:49)

• The query contract. Query#equals is "required so that QueryCache works properly", and "Typically a query will be equal to another only if it's an instance of the same class and its document-filtering properties are identical" (Query.java:95-107).
• The factory contracts. term(term, payloadFilter) is "filtered by the value of the term's payload at each position" (Intervals.java:78-92); ordered "Returns intervals in which the subsources all appear in the given order" (:418); or returns "the disjunction of a set of sub-sources" (:118-152). The results contradict these for legal inputs.
• History. LUCENE-8694 (#9740, romseygeek, 2019) wrote equals/hashCode as a copy of TermIntervalsSource's, with a leftover cast to TermIntervalsSource, so for two distinct instances equals threw and never returned true. #12827 (eumus, 2023) changed only that line. Commit-level A/B: 9.8.0 jars plus the #12827 class give the 9.9.0 wrong results; 9.9.0 jars with #12827 reverted give 6 ClassCastExceptions.
• Precedents. LUCENE-8694 calls the class "the intervals equivalent of SpanPayloadCheckQuery", whose equals/hashCode compare its payload criteria and its wrapped query (SpanPayloadCheckQuery.java:264-280). There, LUCENE-7808 (#8859, 2017) fixed "incomplete .equals and .hashCode methods, causing caching issues": the wrapped span query had not been compared. FunctionMatchQuery compares its DoublePredicate with Objects.equals (FunctionMatchQuery.java:119-129), and FilteredIntervalsSource compares its name (FilteredIntervalsSource.java:142-154).
• Nothing documents it. No javadoc says that filters are ignored by equality or that two filters on one term must not be combined.

The strongest case for "intended": the upstream test testPayloadFilteredTermIntervalsSourceEquals (#12827) asserts that two separately written (_) -> true filters give equal sources (TestPayloadFilteredInterval.java:93-101); Java lambdas have no value equality, so for lambda filters a term-only equals is what lets rebuilt queries hit the cache and be recognised as repeats (a Predicate class with a value equals could do it too); repeat detection deliberately relies on equality (romseygeek in LUCENE-10140, #11177); and the package is experimental.

Why it fails: the #12827 test covers only two filters that behave the same, and #12827 never discussed filters that select different positions; equality of two distinct instances never worked before #12827 (it threw), so no earlier release relied on it; repeat detection is right only for sources that match the same positions; "experimental" warns about API changes, not wrong matches. Verdict: not intended, a defect (the independent reproduction, the cross-model review, the intended-behaviour check and the adjudication, high confidence).

One fix was built and validated through gradle on 645d4fe14a20. Because Java lambdas have no value equality, any fix that compares the filter must change the upstream test from #12827.

+2/-2 in PayloadFilteredTermIntervalsSource.java, plus the #12827 test testPayloadFilteredTermIntervalsSourceEquals changed to share one Predicate instance and to assert that a different filter is not equal (+8/-2 in TestPayloadFilteredInterval.java). For a lambda Objects.equals is identity; a Predicate class with value equals keeps value equality. FunctionMatchQuery compares its predicate the same way.

  @Override
  public int hashCode() {
    return Objects.hash(term, filter);
  }

  @Override
  public boolean equals(Object o) {
    if (this == o) return true;
    if (o == null || getClass() != o.getClass()) return false;
    PayloadFilteredTermIntervalsSource that = (PayloadFilteredTermIntervalsSource) o;
    return Objects.equals(term, that.term) && Objects.equals(filter, that.filter);
  }

Trade-offs

• Two separately built but identical lambdas are no longer equal: rebuilt queries miss the query cache and are not de-duplicated (no wrong result).
• They are also no longer recognised as a repeat: on documents single = a x and double = a a, unordered(term(a, f), term(a, f')) with two separately built "payload is pos: 0" filters returns [double, single] with the fix and [] on main; with one shared always-true Predicate instance (p -> true), main and the fix both give [double], as plain unordered(term(a), term(a)) does (measured by the intended-behaviour check on main ea57360912ed). That is how unordered treats any two different sources that can match the same position ("The subsources may overlap", Intervals.java:432), but it changes behaviour for callers who rebuild the same filter; the CHANGES entry should say "reuse one immutable Predicate instance".
• The fix is only as sound as the predicate's own equals: a Predicate class with a wrong value equals, or a predicate whose behaviour changes after the query is built, can still merge different filters or serve stale cache entries.
• The upstream test from #12827 must change: with the source fix alone it fails (2 tests, 1 failure).
• No in-tree code outside tests uses the overload.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: the repro fails on stock and passes with the fix, assertions on and off.
• org.apache.lucene.queries.intervals.* (8 suites including the repro): 118 tests, 0 failures.
• The whole :lucene:queries:test: 513 tests, 5 skipped, 0 failures.
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest of :lucene:queries pass with the fix and the repro test.
• Earlier, the same source change made the finder's 7 tests, the independent reproduction's 8, the cross-model review's 5 extra paths and the security probe pass on main ea57360912ed.

An overload such as Intervals.term(term, String filterName, Predicate<BytesRef>) whose name goes into equals/hashCode, as FilteredIntervalsSource does with its name (FilteredIntervalsSource.java:77-89, :142-154). Not built.

Trade-offs

Keeps cache hits and repeat detection for rebuilt filters with the same name; callers must name distinct filters distinctly. An API addition; the existing overload still needs A or deprecation.

A factory that takes the accepted payload values, as SpanPayloadCheckQuery takes its payload list and match operation and compares them in equals. Not built.

Trade-offs

Exact equality, but covers fewer cases than a Predicate; an API change.

Say in the javadoc that filters are ignored by equality.

Trade-offs

It would document wrong results for ordered, or and BooleanQuery. Until a fix ships, users can keep two payload-filtered sources on the same term out of one query and out of a shared query cache (for example a QueryCachingPolicy that never caches IntervalQuery).

Method: semi-formal (a hunt unit, query object contracts).

• A bytecode scan of the stock main jars (JDK 25 ClassFile API) compared, for each query component, the fields read by its matching code with the fields read by equals/hashCode. It reported EQ-MISS PayloadFilteredTermIntervalsSource.filter, read by the interval iterators' nextInterval and next but not by equals.
• A bounded-exhaustive equality-congruence harness then tried every ordered pair of each query family over a small parameter domain on a fixed 12-document index, and witnessed equal IntervalQuery pairs with different matched documents (payload pos: 0 vs pos: 1: documents 0 1 3 6 10 vs 1 4 8).
• The finder, an AI tool, wrote the 7-test public-API JUnit; the one-test repro was cut from it on 2026-10-29. The finder tagged the find "formal"; the registry records it as semi-formal because no TLC, SMT or proof output produced the signal.