Intervals.term(term, payloadFilter) throws for the whole search when one segment has no payloads for the field

Intervals.term(term, payloadFilter) returns the positions of a term whose payload passes a filter. Its source checks, once per segment, whether the segment's field has payloads, and throws IllegalArgumentException when it has none, before it looks up the term. That flag is per-segment data: a newly flushed segment gets it only if one of its own documents indexed a non-empty payload for the field; segments may differ on it, and a merge ORs it. So when a flush, commit or refresh happens to hold only documents without payloads in that field, a search or count of the query fails for the whole index, even if that segment does not contain the term, until a merge folds the segment into one with payloads; Weight#matches throws for any document of that segment. Two documents, "dog" with payload noun and "dog" without, one segment each: count throws; in one segment the count is 1. Default writer, codec and searcher, assertions on or off; it fails on the 9 releases run, from 8.1.0 to 10.6.0, and on main, and the releases in between have the same checks by a tag sweep.

1. Indexing sets the flag per segment, from the data: FreqProxTermsWriterPerField.writeProx sets sawPayloads only for a payload with length > 0 (FreqProxTermsWriterPerField.java:88-93), and finish() calls setStorePayloads() only then (:67-73). FieldInfo.hasPayloads(): "Returns true if any payloads exist for this field" (FieldInfo.java:642).
2. Segments may differ on it: FieldInfo.verifySameSchema compares index options, norms, term vectors, doc values, points and vectors, not payloads (FieldInfo.java:244-269). A merge ORs the flags (FieldInfos.java:778-780).
3. PayloadFilteredTermIntervalsSource.intervals() throws IllegalArgumentException when terms.hasPayloads() is false (PayloadFilteredTermIntervalsSource.java:57-62), before seekExact(term) (:63-66); matches() does the same (:163-168, before :169-172).
4. Nothing catches it: IntervalWeight.scorerSupplier calls the source (IntervalQuery.java:189-190) from IndexSearcher.searchLeaf (IndexSearcher.java:842), so search and count fail for the whole index; IntervalWeight.matches calls matches() (IntervalQuery.java:171-175).
5. In a segment that has payloads, a position without one reads as null (Lucene104PostingsReader.java:1338-1343) and goes to the filter (PayloadFilteredTermIntervalsSource.java:124). So the same document is searched fine next to a payloaded one and fails the whole search alone in its own segment.

• Two documents, one segment each ("dog" with payload noun, "dog" without), count(IntervalQuery("f", Intervals.term("dog", p -> p != null))): throws. The same two documents after forceMerge(1): 1.
• DelimitedPayloadTokenFilter text, big|adj dog|noun, dog walks, hot|adj dog|noun, one segment each, filter "payload is noun": throws; after forceMerge(1): 2.
• The payload-less segment does not contain the queried term: still throws, because the check comes before the term lookup.
• Default IndexWriterConfig and merge policy, a plain flush() and an NRT reader: count throws.
• Weight#matches (used for highlighting) throws the same exception from matches() for the document of the payload-less segment; for doc 0, in the segment with payloads, it returns a match.

• Who is hit. Callers of the payload-filter overload of Intervals.term on a field with sparse payloads: DelimitedPayloadTokenFilter output where some documents have no delimited tokens, or a NumericPayloadTokenFilter that sets a payload only on tokens of one type (NumericPayloadTokenFilter.java:49). Any flush, commit or NRT refresh that holds only such documents creates the state, with the default IndexWriterConfig, codec and merge policy.
• Who is not. Applications whose analyzer gives every token a payload, like SimplePayloadFilter in the only upstream test (TestPayloadFilteredInterval.java:42). No Lucene query parser builds this source, and by public code search neither do Elasticsearch, OpenSearch or Solr.
• Default vs opt-in. Default writer, codec and searcher; the trigger is the opt-in payload-filter overload in the intervals package, which its package-info.java calls experimental.
• Scale and timing. One payload-less segment fails the whole index, even when it does not contain the queried term. It lasts until a merge folds that segment into one with payloads, and the next such flush brings it back, so the failure comes and goes with the merge schedule.
• How it fails. Loudly and deterministically: an IllegalArgumentException, assertions on or off. No silent wrong results, nothing written to the index. Likelihood: plausible; the public searches found no report since the API shipped in 2019.

NoMergePolicy and a commit per document only fix the segment layout; a default IndexWriterConfig with an ordinary flush and an NRT reader reaches the same state. CannedTokenStream tokens set the payload directly, so no analysis module is needed. Deterministic, single thread, a plain IndexSearcher. The catch only turns the exception into an expected-versus-actual message and keeps it as the cause. This test was run on main and 10.6.0. On 9.0.0 and older LuceneTestCase is in org.apache.lucene.util and CannedTokenStream and Token are in org.apache.lucene.analysis (on 8.1.x the intervals package is also org.apache.lucene.search.intervals). The older releases ran the finder's DelimitedPayloadTokenFilter test instead, which fails there with only its LuceneTestCase import (9.0.0 and older) and its package line (8.1.x) changed.

TestPayloadFilteredIntervalsSegmentWithoutPayloads > testSegmentWithoutPayloads FAILED
    java.lang.AssertionError: expected count 1 (the doc whose 'dog' has a payload), got java.lang.IllegalArgumentException: Cannot create a payload-filtered iterator over field f because it has no indexed payloads
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:171531F2198284EE]:0)
        at org.apache.lucene.queries.intervals.TestPayloadFilteredIntervalsSegmentWithoutPayloads.testSegmentWithoutPayloads(TestPayloadFilteredIntervalsSegmentWithoutPayloads.java:52)
        Caused by:
        java.lang.IllegalArgumentException: Cannot create a payload-filtered iterator over field f because it has no indexed payloads
            at org.apache.lucene.queries.intervals.PayloadFilteredTermIntervalsSource.intervals(PayloadFilteredTermIntervalsSource.java:58)
            at org.apache.lucene.queries.intervals.IntervalQuery$IntervalWeight.scorerSupplier(IntervalQuery.java:190)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:874)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.IndexSearcher.searchLeaf(IndexSearcher.java:842)
            ...
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.IndexSearcher.count(IndexSearcher.java:528)
1 test completed, 1 failed

# the same with -Ptests.asserts=false; with the recommended fix the test passes

• The contract. The factory javadoc says only "filtered by the value of the term's payload at each position" (Intervals.java:79-80, :87-88) and documents no exception. IntervalsSource.intervals: "Returns {@code null} if no intervals for this field exist in this segment" (IntervalsSource.java:39). The multiterm factories, by contrast, document their per-segment limit as @throws IndexSearcher.TooManyClauses (Intervals.java:366-367).
• A schema check copied onto data. The payload check sits right under the hasPositions() check (PayloadFilteredTermIntervalsSource.java:51-56). Positions follow the field's IndexOptions, which verifySameSchema keeps equal in every segment (FieldInfo.java:246); payloads are per segment. TestTermVectors.testMergeWithPayloads builds one segment with and one without payloads as a legal state (TestTermVectors.java:128, :137-171).
• Null payloads are already normal. In a segment that has payloads, a position without one reaches the filter as null, and Lucene's own DelimitedPayloadTokenFilter produces such positions by design. Removing the check gives the filter no new kind of input.
• The design reference does not do this. LUCENE-8694 (#9740) asked for "the intervals equivalent of SpanPayloadCheckQuery". SpanPayloadCheckQuery checks only positions (SpanPayloadCheckQuery.java:179-187) and handles a null payload per position (:220-230); TestPayloadTermQuery.testNoPayload (:273) expects a hit on a field without payloads.
• Tests and history. "no indexed payloads" occurs in no test source; the only upstream test indexes one document whose every token has a payload. The hasPositions guard, in contrast, is asserted (TestIntervals.java:260-269). The check arrived with the class in 2019 (5ca25249274b) and was never discussed on #9740.
• Precedent. #10794 / LUCENE-9755 (open): a search that fails on a segment without doc values for a field, the same failure class in different code.

The strongest case for "works as designed": the check is deliberate. It copies the hasPositions guard to fail fast when someone filters on payloads of a field that was never indexed with payloads, and IllegalArgumentException is Lucene's usual signal for a misconfigured field. The multiterm factories also fail per segment, so their failure also changes with merges. On #7333 / LUCENE-6271 (2015) a maintainer wrote that Terms.hasPayloads() and its siblings "should be used for these checks up-front". A user who filters on payloads should give every document a payload, as the upstream test does.

Why it fails: the LUCENE-6271 remark is about checking capabilities up front in general and says nothing about segments that differ; a per-segment check cannot tell a misconfigured field from a segment that by chance holds no payloads, so what ships rejects legitimate indexes and lets the outcome depend on flush timing. The multiterm limit is a documented resource bound; this rejection is undocumented and depends on the data. No javadoc or test requires every document to carry a payload, Lucene's own payload filters emit positions without one, and the class's design reference, SpanPayloadCheckQuery, accepts such segments. The misconfiguration point is real, which is why an index-wide error is listed as an alternative fix. Verdict: not intended, a defect.

Option A was applied in a scratch worktree of 645d4fe14a20 and run with gradle (the worktree was removed after the runs) and shadowed into the main and 10.6.0 jars. Options B and C were not built.

-12 lines in PayloadFilteredTermIntervalsSource.java (the hasPayloads() == false blocks at :57-62 in intervals() and :163-168 in matches()), +1 javadoc sentence on each Intervals.term(.., Predicate) overload. A segment without payloads then behaves like a segment whose positions have none: every position reaches the filter as null and the filter decides.

// PayloadFilteredTermIntervalsSource.java, intervals() :57-62 and matches() :163-168
-    if (terms.hasPayloads() == false) {
-      throw new IllegalArgumentException(
-          "Cannot create a payload-filtered iterator over field "
-              + field
-              + " because it has no indexed payloads");
-    }
// Intervals.java, javadoc of both term(.., Predicate<BytesRef>) overloads (:80, :88)
-   * term's payload at each position
+   * term's payload at each position. Positions without a payload reach the filter as {@code null}.

Trade-offs

• The hit set no longer depends on segment layout; nothing new reaches the filter.
• A field with no payloads in any segment (a real misconfiguration) no longer errors; it matches what the filter accepts for null, usually nothing.
• A filter that dereferences the payload without a null check throws NullPointerException on payload-less segments after the fix; it already does on payload-less positions in mixed segments today, and the javadoc sentence says so. The upstream test's own filter (b -> b.utf8ToString().endsWith("5") == false, TestPayloadFilteredInterval.java:68) is such a filter; it stays green only because every token there has a payload.
• Related registry row LFV-268 (under the default WEIGHT_MATCHES flag the highlighter's re-analysis drops payloads): its exception should turn into a missing highlight (read from the code; not run with this patch).

What the checks validated

• Gradle, stock 645d4fe14a20: the repro fails with assertions on and off.
• Gradle with the fix (seed DEADBEEF), assertions on: the whole :lucene:queries suite with the repro, 52 classes, 513 tests, 0 failures, 5 skipped (none of the 5 skipped tests is in the intervals package); the 8 intervals classes hold 118 of the tests, TestPayloadFilteredInterval and the repro included. Assertions off: the repro and the intervals and payloads packages, 152 tests, 0 failures. checkGoogleJavaFormat passes.
• Shadowed into the stock jars by the standalone runner: the repro passes on main (-ea) and on the 10.6.0 release jars (-da).
• Earlier, the finder's test and the security probe passed with the finder's patch (both checks removed, a two-line comment in place of each), the probe returning the correct hits; an independent verifier's test, run by the verifier and again by the adjudication, passed with the bare removal of the two checks.

Check once over the top-level reader, for example with ReaderUtil.getTopLevelContext(ctx) and FieldInfos.getMergedFieldInfos(reader), or where IntervalQuery.createWeight (IntervalQuery.java:122) sees the searcher.

Trade-offs

• Keeps a fail-fast for a field never indexed with payloads, and stops the layout dependence once any segment has payloads.
• IntervalsSource.intervals is also called directly with a leaf context, so a check in the source needs the top-level context on each call (merging field infos costs O(segments) per call); a check in IntervalQuery misses direct callers.
• An index that so far holds only payload-less documents still throws, including the monitor module's index for a multi-document batch without payloads. It also departs from how a query on missing data usually behaves (no hits).

Add @throws IllegalArgumentException to the factories and require a payload on every token.

Trade-offs

Keeps the layout-dependent failure; listed for completeness.

Method: code reading (not a formal-methods finding). A hunt unit (intervals builder and multi-term sources) was extending its reference semantics to payload-filtered leaves; none of its bounded-exhaustive harnesses produced the signal.

• The finder noticed that PayloadFilteredTermIntervalsSource checks hasPayloads() per segment, while the flag depends on which documents a segment holds.
• A probe built three one-document segments, the middle one with hasPayloads() == false: the payload-filtered search threw, while a plain term count gave 3.
• A public-API JUnit then checked that the hit set does not depend on segment layout (one segment per document against forceMerge(1)): it fails on main, 10.5.2, 9.12.3 and 9.1.0, and passes with the two checks removed.
• The repro was later minimised to the one-method, two-document test shown above.