Multi-term interval queries skip the positions check and silently return wrong hits (usually none) on a field without positions

On a field indexed without positions (IndexOptions.DOCS or DOCS_AND_FREQS), positional queries are documented to throw. Interval leaves made with Intervals.term do. The multi-term leaves (Intervals.prefix, wildcard, regexp, fuzzyTerm, range, multiterm, all built on MultiTermIntervalsSource) never call Terms#hasPositions(): they run the interval logic on the position -1 that the postings return for every occurrence. On one document alpha beta, ordered(prefix("alph"), prefix("bet")) returns 0 hits with no error where ordered(term("alpha"), term("beta")) throws. The check was lost when LUCENE-8646 added multi-term interval leaves in 8.0.0 (2019); every release since, through 10.6.0, and main are affected.

TermIntervalsSource.intervals() reads the field's Terms, returns null if the field is missing, then throws IllegalArgumentException("Cannot create an IntervalIterator over field ... because it has no indexed positions") when terms.hasPositions() is false (TermIntervalsSource.java:54-59), and only then calls its static helper intervals(term, te) (:64, :67). matches() does the same (:151-156, helper at :164). PayloadFilteredTermIntervalsSource has the same check (:51, :157).

Intervals.prefix, wildcard, regexp, range and multiterm return a MultiTermIntervalsSource (Intervals.java:176-180, :206-208, :237-241, :279-295, :369-370); fuzzyTerm goes through multiterm (:335-341). Its intervals() (MultiTermIntervalsSource.java:57-77) reads Terms, returns null for a missing field, and goes straight to automaton.getTermsEnum(terms) and TermIntervalsSource.intervals(term, te) for each expanded term (:64, :68). Its matches() (:79-103) calls TermIntervalsSource.matches(te, doc, field) (:90). Neither calls hasPositions().

The helper asks the postings for positions (TermIntervalsSource.java:68). The default postings reader grants them only when the field has them (Lucene104PostingsReader.java:420) and otherwise returns -1 from nextPosition() (:1293-1296), as PostingsEnum.nextPosition documents ("or -1 if positions were not indexed", PostingsEnum.java:84-88). -1 is also the value IntervalIterator uses for "not positioned yet" (IntervalIterator.java:31-33).

ordered needs each leaf to start after the previous one ends: it skips beta intervals while currentStart <= prevEnd (OrderedIntervalsSource.java:131, :145-151). With alpha ending at -1 and every beta starting at -1, no beta qualifies and the document does not match. Other operators turn the same coinciding -1 positions into other answers.

In a tree that mixes both kinds, leaf order matters: ordered and unordered build their sub-iterators in order and return null at the first leaf that returns null (MinimizingConjunctionIntervalsSource.java:42-52; ConjunctionIntervalsSource.java:47-57 likewise), and a multi-term leaf returns null when it expands to nothing in the segment (MultiTermIntervalsSource.java:73-75). So a term leaf throws only once it is reached: ordered(prefix("zzz"), term("beta")) returns 0 hits with no error, while ordered(term("beta"), prefix("zzz")) throws.

On 8 documents, containing, containedBy and overlapping over two multi-term leaves return the 5 documents that hold both expanded words, none of which the same query matches on a copy with positions, and a MUST_NOT(ordered(alph*, bet?)) filter excludes nothing (a MUST_NOT over containing(alph*, bet*) or a lone prefix(alph*) does exclude).

• Who is hit. Users of the opt-in, experimental intervals API (package-info.java:21) whose query has only multi-term leaves and whose field has no positions. The flexible StandardQueryParser reaches it through fn:wildcard and fn:fuzzyTerm (intervalfn/Wildcard.java:38-40, FuzzyTerm.java:44).
• What goes wrong. No error: positional shapes return no hits, containment shapes return documents their positional meaning excludes, a lone multi-term leaf returns the documents that hold an expanded word. The same query with a plain Intervals.term leaf would usually have thrown and pointed at the schema (it throws once it is evaluated).
• What it takes. One document, one query; default codec, IndexWriterConfig and searcher. Deterministic, with assertions on or off. The index is not damaged.
• How likely: rare. It needs both an all-multi-term interval query and a field without positions. Elasticsearch and OpenSearch guard against it in their own code: OpenSearch checks hasPositions(), and Elasticsearch text fields either throw or recompute positions from the field values (source read); Solr main's unreleased intervals query parser applies no such check (source read).

TestMultiTermIntervalsWithoutPositions (56 lines with the license header, one test method, public API only). It fails at the expectThrows on line 52. The result is the same with two seeds, that is two random codec, directory and IndexWriterConfig choices of the test framework; the mechanism rests on the PostingsEnum.nextPosition contract (-1 without positions), not on those choices.

As a regression test it goes into lucene/queries/src/test/org/apache/lucene/queries/intervals/ unchanged (or as a method of TestIntervals, next to testIntervalsOnFieldWithNoPositions); it passes with the fix. Formatting: google-java-format puts the two Intervals.prefix(...) arguments on one line.

The same test fails on the 10.6.0 release jars. The finder's 4-test class (prefix leaves, a DOCS_AND_FREQS field and a mixed MultiReader) fails on every release from 8.0.0; an independent test of all six factories on DOCS and DOCS_AND_FREQS fields fails on 9.12.0, 9.12.3, 10.5.2 and main. This test shows the missing exception for an ordered pair of prefixes that both expand; the other factories, Weight#matches, the parser path and the over-matching shapes rest on the separate probes.

cp $REPRO/TestMultiTermIntervalsWithoutPositions.java $LUCENE/lucene/queries/src/test/org/apache/lucene/queries/intervals/
cd $LUCENE && ./gradlew :lucene:queries:test --tests org.apache.lucene.queries.intervals.TestMultiTermIntervalsWithoutPositions -Ptests.seed=DEADBEEF

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF (verbatim)
TestMultiTermIntervalsWithoutPositions > testPrefixIntervalsThrowWithoutPositions FAILED
    junit.framework.AssertionFailedError: Expected exception IllegalArgumentException but no exception was thrown
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:7C4D00743B2EB14C]:0)
        at app/org.apache.lucene.test_framework@11.0.0-SNAPSHOT/org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1432)
        at app/org.apache.lucene.test_framework@11.0.0-SNAPSHOT/org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
        at app//org.apache.lucene.queries.intervals.TestMultiTermIntervalsWithoutPositions.testPrefixIntervalsThrowWithoutPositions(TestMultiTermIntervalsWithoutPositions.java:52)

# standalone runner, stock jars, seed DEADBEEF (summary of result.json)
main 645d4fe14a20  -ea / -da   tests_run=1 failures=1   same message
10.6.0             -ea         tests_run=1 failures=1   same message

# with the recommended fix, gradle (summary)
TestMultiTermIntervalsWithoutPositions  tests=1 failures=0

• Contract. IndexOptions.java:31-36 (DOCS): "Phrase and other positional queries on the field will throw an exception"; :37-41 (DOCS_AND_FREQS): "PhraseQuery and other positional queries will throw an exception". It does not name interval queries; they are positional queries, and TermIntervalsSource and PayloadFilteredTermIntervalsSource apply the rule to interval leaves.
• The symptom was removed once. CHANGES.txt:15444-15446, 4.0.0-ALPHA: "Positional queries (PhraseQuery, *SpanQuery) will now throw an exception if use them on a field that omits positions during indexing (previously they silently returned no results)".
• Siblings throw. PhraseQuery.java:504-510, MultiPhraseQuery.java:255-262, SpanTermQuery.java:149-155 (so SpanMultiTermQueryWrapper(PrefixQuery) throws even alone), TermIntervalsSource.java:54-59, PayloadFilteredTermIntervalsSource.java:51. Not every positional query checks: the sandbox TermAutomatonQuery has no check either (TermAutomatonQuery.java:423) and returned 0 hits with no error for alpha (beta | gamma delta) on such a field. Within one interval query, a plain term leaf throws once it is evaluated: ordered(prefix("alph"), term("beta")) throws, ordered(prefix("alph"), prefix("bet")) does not.
• Test. TestIntervals#testIntervalsOnFieldWithNoPositions (TestIntervals.java:260-270) asserts the exact message for Intervals.term("wibble"), a word not in the index. No test covers a multi-term leaf on such a field.
• History. LUCENE-8196 (7.4.0) added the check; its review asked that "TermIntervalsSource should check whether positions were indexed" (#9243). LUCENE-8646 (#9692, 8.0.0) split TermIntervalsSource.intervals()/matches() into a checked instance method and an unchecked static helper, and built MultiTermIntervalsSource on the helpers only; its discussion never mentions positions. No later commit added the check.

The strongest case for "not a bug": the intervals package is experimental; IntervalsSource.intervals may return null "if no intervals for this field exist in this segment" (IntervalsSource.java:39), so an empty result could be within contract; LUCENE-9462 deliberately made Weight#matches of TermQuery, SynonymQuery and MultiTermQuery work without positions (CHANGES.txt:4717); a lone multi-term leaf returns the right documents, so a check changes behaviour; and a positional query on a positions-free field is the user's misconfiguration.

Why it fails: the javadoc names the outcome, an exception, not an empty result; the null rule covers a segment without the field or the words, and Intervals.term throws even for an absent word; LUCENE-9462 changed no interval code and IntervalQuery(term).matches still throws, so it can at most argue for leaving the matches() half lenient; the span wrapper over the same PrefixQuery throws even alone; and the misconfiguration is exactly the input Lucene promises to reject. Verdict: not intended, a missing validation.

Option A was validated with gradle in a scratch worktree at 645d4fe14a20; option B is its first hunk alone; option C is a refactoring of A that was not run. Offer A and B: the search half is clear-cut, while MatchesIterator allows -1 "if positions are not available" (MatchesIterator.java:48, :55) and LUCENE-9462 kept Weight#matches of non-interval queries lenient, so the matches() half is the maintainers' decision.

Right after the null check in intervals() (MultiTermIntervalsSource.java:60-62) and in matches() (:83-85), the same check and message as TermIntervalsSource.java:54-59. The change is +12 lines in one file.

    if (terms == null) {
      return null;
    }
    if (terms.hasPositions() == false) {
      throw new IllegalArgumentException(
          "Cannot create an IntervalIterator over field "
              + field
              + " because it has no indexed positions");
    }

Trade-offs

• Multi-term leaves then behave exactly like Intervals.term, with the same message; one hasPositions() call per segment in intervals() and one per matches() call (per Weight#matches request for a document).
• Behaviour change, the main compatibility decision: a lone multi-term leaf on a positions-free field (for example a prefix on a StringField) throws where it matches today; worth a CHANGES entry.

What the checks validated

• Scratch worktree at 645d4fe14a20, JDK 25, gradle, -Ptests.seed=DEADBEEF: the minimal test passes; :lucene:queries:test 52 suites, 513 tests, 0 failures (all 8 intervals suites, including TestIntervals 43 and TestIntervalQuery 34); :lucene:highlighter:test 34 suites, 722 tests, 0 failures (TestMatchHighlighter, TestMatchRegionRetriever, TestUnifiedHighlighterTermIntervals); :lucene:queryparser:test 30 suites, 533 tests, 0 failures (TestStandardQPEnhancements, TestStandardQP). In that run checkGoogleJavaFormat failed on the test file's line wrap only (see the repro), not on the fix; with the test reflowed it passes.
• Without the fix, the same minimal test fails through gradle.
• Earlier, on main a9779ef3c193 with the class shadowed: the finder's 4 tests and an independent reproducer's 4 tests pass, and 81 upstream interval tests pass; on 8.0.0 the same two checks in the 8.0.0 source turn 2 of 4 failures into 4 of 4 passes.

The first hunk of option A's change alone (the check in intervals()), in the spirit of LUCENE-9462, which made Weight#matches of non-interval queries work without positions.

Trade-offs

• Search fails loudly as documented, but Weight#matches of a multi-term interval query still reports a match at position -1, while Intervals.term's matches() throws (TermIntervalsSource.java:151-156): the two leaf kinds stay inconsistent there.
• Code that calls Weight#matches directly for a document (highlighters, for example) still gets a match at position -1.

What the checks validated

• Scratch worktree at 645d4fe14a20, gradle, -Ptests.seed=DEADBEEF: the minimal test passes (it only searches); the 8 intervals suites of :lucene:queries pass, 118 tests, 0 failures; :lucene:highlighter:test passes, 34 suites, 722 tests, 0 failures. The queryparser suite was not re-run for this option.

Move the check into a package-private static method of TermIntervalsSource and call it from both methods of TermIntervalsSource, MultiTermIntervalsSource and PayloadFilteredTermIntervalsSource.

Trade-offs

Same behaviour as A with one copy of the message; a larger diff touching three classes.

Method: code reading, then differential and metamorphic testing. Neither step is formal verification.

• First finder (2026-10-26, main ea57360912ed): while writing a reference model of the multi-term interval source for a translation-validation harness, an AI tool read MultiTermIntervalsSource next to TermIntervalsSource, saw that only the latter checks hasPositions(), and confirmed it with a public-API probe and a JUnit on the stock jars ("got intervals [-1,-1]" where or(term, term) throws).
• Second, independent finder (2026-10-26, main a9779ef3c193): a harness indexes the same documents at 29 index-option levels and checks relations between them. Relation R3: a positional query on a level without positions must throw. Over seeds 41-440 it saw 4,665 silent mismatches and 462 more on a mixed MultiReader, all printed ones on multi-term interval leaves, while PhraseQuery, MultiPhraseQuery, span and Intervals.term queries whose terms were present threw on every positions-free level.
• The two findings were merged into one record; the hunt then ran the checks listed below.