Intervals.ordered(a, a) reports a minimum width of 1, so an Intervals.or under phrase or maxgaps silently drops the two-word match

Under a gap-sensitive parent (phrase, maxgaps), Intervals.or is documented to rewrite itself so that alternatives of different lengths are not minimized away. The rewrite decides "one position or longer" with minExtent() == 1. Since LUCENE-9099 (8.5.0), Intervals.ordered(a, a) and Intervals.unordered(a, a) are a RepeatingIntervalsSource whose minExtent() returns the inner source's value, 1 for a term, although on ordinary text each interval it returns spans at least 2 positions. So or(ordered(a, a), a) stays one OR; on x a a c that OR returns [1,1] and [2,2] and drops [1,2], and phrase(x, or(ordered(a, a), a), c) finds nothing, while phrase(x, ordered(a, a), c) alone and phrase(x, or(ordered(a, b), a), c) on x a b c match. The same happens with unordered(a, a), under maxgaps(0, ...) and on the big side of containedBy/containing, with any other alternative that also reports minExtent() 1 and falls inside the copies' span (a word between them, or a shorter repeat such as ordered(a, a) next to ordered(a, a, a)), through the flexible query parser's fn: functions and through Intervals.analyzedText("a a", ...) with gaps. Silent wrong results with the default writer, codec and searcher, in every release from 8.5.0 to 10.6.0 (10.6.0 by reading) and on main. Counting the copies in minExtent() (in.minExtent() + childCount - 1, saturated) makes the repro pass with the whole lucene/queries module green; applied alone it widens two separate bugs (LFV-291, LFV-810), so it should land with or after their fixes.

• Interval query: matches runs of positions. Intervals.term("a") gives one-position intervals such as [1,1]; ordered(a, b) gives [start, end] of an a followed by a b; phrase(x, s, c) needs x, then s, then c back to back.
• Minimal intervals: most operators, and or() in particular, keep only intervals that contain no other interval they found (the repeated source below is documented as "avoiding minimization" of its own copies). If one alternative matches [2,2] and another [1,2], the OR keeps [2,2] and drops [1,2].
• The disjunction rewrite (LUCENE-8477, 8.1.0): because of that, phrase and maxgaps rewrite an inner or() whose alternatives have different widths into an OR of separate queries, so phrase(x, or(ordered(a, b), a), c) becomes or(phrase(x, ordered(a, b), c), phrase(x, a, c)). or(false, ...) turns this off and is documented to "miss valid hits due to minimization".
• minExtent(): "the minimum possible width of an interval returned by this source". The rewrite keeps every alternative with minExtent() == 1 together in one OR, as one-position alternatives.
• Repeated source (LUCENE-9099, 8.5.0): ordered(a, a) and unordered(a, a) are built as one RepeatingIntervalsSource(a, 2), which joins two consecutive a intervals into one.

1. Intervals.ordered and unordered merge equal arguments into a RepeatingIntervalsSource and return it as the whole query when nothing else is left (OrderedIntervalsSource.java:29-57, UnorderedIntervalsSource.java:33-57).
2. Its minExtent() is return in.minExtent(); (RepeatingIntervalsSource.java:93-96): 1 for a term. On distinct positions its intervals are never narrower than 2 (all 510 documents of 1 to 8 positions over {a, b}, executed).
3. Disjunctions.splitDisjunctions (Disjunctions.java:75-95) groups every alternative with minExtent() == 1 into one OR, citing "we know that all combinatorial sources have a minExtent > 1". or(ordered(a, a), a) therefore stays one OR inside the phrase. phrase, maxgaps, containedBy, containing, notContainedBy and notContaining run this rewrite when they are built.
4. The OR's iterator orders intervals by end, then by start descending, and drops every interval that contains the one it just returned (DisjunctionIntervalsSource.java:171-175, :230-247). On x a a c it returns [1,1] and [2,2] and drops [1,2] of ordered(a, a). The phrase needs [1,2] between x (0) and c (3): 0 hits.
5. ordered(a, b) sums its parts (OrderedIntervalsSource.java:69-75), 2, so it is split out and matches. A clause that merely contains a repeat, ordered(a, a, b), reports 2 (true minimum 3), so it is split out too; there only the score is affected.
6. Side effect: IntervalScorer computes the sloppy frequency as 1 / max(length - minExtent + 1, 1) (IntervalScorer.java:66), so an exact a a match of ordered(a, a) scores 0.333 where ordered(a, b) on a b scores 0.5.

With a repeated source, the default or() behaves like the opt-out or(false, ...) (the last-but-one row). Also executed with the default IndexWriterConfig and StandardAnalyzer: ordered(a, a, a) on x a a a c, the operands swapped (or(a, ordered(a, a))), Intervals.analyzedText("a a", analyzer, field, 1, true) as the alternative, and the StandardQueryParser string fn:phrase(x fn:or(fn:ordered(a a) a) c): 0 hits each, controls 1. Added after the fact checks (main and 10.5.2), 0 hits on stock and 1 with the fix each: phrase(x, or(ordered(a, a, a), ordered(a, a)), c) on x a a a c (two repeats, neither one position wide), and containedBy(b, or(ordered(a, a), a)) and containing(or(ordered(a, a), a), b) on a b a (controls with ordered(a, c) on a b c: 1).

• Who. Applications that build interval queries with the Intervals factories, the flexible StandardQueryParser's fn: functions, or Intervals.analyzedText over query text that repeats a word, where a clause made only of copies of one source (ordered(a, a), unordered(a, a), ordered(a, a, a), or a filter over one such as maxgaps(1, ordered(a, a))) is an alternative of the default or() under phrase or maxgaps (or on the big side of containedBy/containing), next to another alternative that reports minExtent() 1 (a word, or a shorter repeat). A natural shape is "the word doubled, or the word once".
• Effect. Documents in which that other alternative falls inside the copies' span are missed, silently and every time, with assertions on or off. Exact repeated matches also score lower than comparable two-word matches (0.333 against 0.5). Nothing is written to the index.
• Default vs opt-in. Default writer, codec, analyzer and searcher; the rewrite that should protect the match is the default (or() means or(true, ...)). The intervals package is documented as experimental.
• Scale. None: one tiny document triggers it, deterministically, at any index size.
• How likely. Uncommon: it needs a repeated-only clause as an OR alternative under a parent that rewrites. A clause that merely contains a repeat (ordered(a, a, b)) matches correctly. No report was found in the six and a half years since 8.5.0.
• Related, separate reports in the same class (each with its own fix line): the width() sign of the same iterator (LFV-306), a repeated disjunction that is never pulled up (LFV-809), overlapping copies in ordered(X, X) (LFV-807), nested repeats leaking inner gaps (LFV-810), and notContainedBy over the same rewrite (LFV-291).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: phrase(x, or(unordered(a, a), a), c) on 'x a a c' expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.queries.intervals.TestRepeatAlternativeUnderPhrase.testPhraseOverOrWithUnorderedRepeat(TestRepeatAlternativeUnderPhrase.java:42)

• The contract is a minimum width. IntervalsSource#minExtent: "Return the minimum possible width of an interval returned by this source" (IntervalsSource.java:63).
• The rewrite promises this match. The Intervals class javadoc: gap-sensitive sources "will rewrite their sub-sources so that disjunctions of different lengths are pulled up to the top of the interval tree ... to ensure that documents containing "b c" are matched" (Intervals.java:47-54); or(): "Automatically rewrites if wrapped by an interval source that is sensitive to internal gaps" (:118-125). Only or(false, ...) "can miss valid hits due to minimization" (:127-136). LUCENE-8477, a bug fix in 8.1.0: "Interval disjunctions could miss valid hits if some of the clauses of the disjunction are minimized away. We now rewrite ..." (lucene/CHANGES.txt:6502-6506).
• The code states the invariant the repeated source breaks: "we know that all combinatorial sources have a minExtent > 1" (Disjunctions.java:76-78).
• Two changes that never met. LUCENE-8477 (2019, 8.1.0) added the grouping when it was sound: ordered(a, a) summed to 2, as executed on 8.2.0 to 8.4.1. LUCENE-9099 (2020, 8.5.0) added the repeated source with return in.minExtent();; its issue, commit message, CHANGES entry (lucene/CHANGES.txt:5996-6001) and its one approving review never mention minExtent() or the rewrite.
• No upstream test asserts the current value. TestIntervals asserts minExtent() 20 times (for example 2 for unordered(pease, hot), TestIntervals.java:380-402); none of its repeat tests checks it, and TestDisjunctionRewrites has no repeated shape.
• Legal input. Repeated arguments are supported since LUCENE-9099, analyzedText builds them from plain query text, and or() under phrase is the canonical LUCENE-8477 shape.

The strongest case for "works as designed" (the cross-model review, reading only, rated it moderate; the intended-behaviour check built it and rejected it): minExtent() is only a lower bound and 1 is a valid one; for identical tokens stacked at one position (position increment 0) ordered(a, a) really returns [p, p], so 1 is then exact and the fix over-reports; and minimal-interval semantics drop containing intervals by design, in an experimental package.

Two more points a maintainer may raise: in the LUCENE-8477 issue discussion (a JIRA comment, now on #9523; not in the PR review) jimczi warned that minimization would keep producing pathological cases; and the flexible query parser's docs say, for fn:ordered/fn:unordered over an fn:or, that "the algorithm is not required to return or highlight all matching interval alternatives" (intervalfn/package-info.java:347-349, :381-383).

Why it fails: the only consumer that affects matching, splitDisjunctions, reads minExtent() == 1 as "every interval is one position", and the documented default promises the rewrite. On the stacked document every query keeps its hit with and without the fix, and the fix leaves its score unchanged (0.5), because the scorer clamps at 1 (executed). The sibling UnorderedIntervalsSource already over-reports the same way (unordered(a, b) returns [0,0] on stacked a/b, minExtent() 2, the value upstream asserts). LUCENE-8477 chose to rewrite rather than document such misses. The query parser note is about which intervals of a matching document are returned or highlighted (in its examples fn:ordered/fn:unordered do not rewrite and the document still matches); here the whole document is missed under a parent whose rewrite the Intervals note promises "to ensure that documents ... are matched". File the claim for distinct positions.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle; the alternative was not run.

Two couplings, to state in the PR. Land the fix with or after the fixes of LFV-291 and LFV-810, or say so.

• LFV-291 (notContainedBy over the same rewrite): applied alone, the fix lets more queries reach it. notContainedBy(x, or(ordered(a, a), b)) on a x a b goes from 0 hits (correct) to 1 (wrong), because the OR is now split the way or(ordered(a, b), b) already is (that control gives 1 on stock). Executed by the devil's advocate and re-run with this fix.
• LFV-810 (a nested repeat passes its inner gap up to its parent): applied alone, the fix pulls the repeat out of the OR, so it then sits directly inside an ordered/unordered under maxgaps(n >= 1), where LFV-810 loses correct hits. maxgaps(1, unordered(x, or(unordered(a, a), b))) on x a c c a x and maxgaps(1, ordered(x, or(ordered(a, a), b), c)) on x a x x a c give 1 hit on stock and 0 with the fix; stock is right, because IntervalIterator#gaps counts only "the gaps between the immediate sub-intervals" (IntervalIterator.java:62-71). Over all 5,460 documents of 1 to 6 positions over {x, a, b, c} and 26 query shapes, these two queries lose 87 and 10 hits and 30 documents score lower; with the LFV-810 candidate fix applied as well, 0 hits are lost, no score falls and every gain is kept (the fact check's probe; the two queries re-run for this doc).

Workarounds today (executed, also on 10.5.2): build the rewritten form by hand, or(phrase(x, ordered(a, a), c), phrase(x, a, c)) (1 hit on x a a c and on x a c); or, when the copies must be adjacent anyway, write the repeat as phrase(a, a), whose minExtent() is 2, so phrase(x, or(phrase(a, a), a), c) is rewritten and finds x a a c.

In RepeatingIntervalsSource.minExtent() (RepeatingIntervalsSource.java:93-96) return in.minExtent() + childCount - 1, computed in long and saturated at Integer.MAX_VALUE (+4/-1 in one file). The iterator joins childCount consecutive intervals of in; on distinct positions each copy ends after the previous one, so the joined interval spans at least in.minExtent() + childCount - 1 positions when the first copy is at least in.minExtent() wide. Tight for unordered, whose copies may overlap. Not a universal bound (executed): stacked identical tokens give [p, p], and extend() clamps starts at 0, so ordered(extend(a, 2, 0), extend(a, 2, 0)) returns [0,1] on a a while reporting 3 on stock and 4 with the fix. The saturation is local: ExtendedIntervalsSource.minExtent() returns Integer.MAX_VALUE on overflow (ExtendedIntervalsSource.java:126-132) and ordered(extend(a, 0, MAX), extend(a, 0, MAX)).minExtent() stays Integer.MAX_VALUE, but the enclosing sums in OrderedIntervalsSource, UnorderedIntervalsSource and BlockIntervalsSource are unchecked: ordered(extend(a, 0, MAX), b).minExtent() already wraps to Integer.MIN_VALUE on stock, and with the fix so does ordered(ordered(h, h), b) for h = extend(a, 0, MAX - 2) (stock: MAX_VALUE). Add the structural test to TestDisjunctionRewrites (no upstream test covers a repeated shape).

  @Override
  public int minExtent() {
-    return in.minExtent();
+    // On distinct positions each copy ends after the previous one, so the copies together span at
+    // least in.minExtent() + childCount - 1 positions (stacked tokens, or an `in` whose own
+    // minExtent() over-reports, can be narrower). Saturate: extend() can report MAX_VALUE.
+    return (int) Math.min((long) in.minExtent() + childCount - 1, Integer.MAX_VALUE);
  }

Trade-offs

• Query size and the clause limit: the rewrite now splits alternatives it grouped before, and Disjunctions.pullUp expands them as a Cartesian product, as it already does for different words (the cost the Intervals class note describes). A phrase of 10 clauses or(ordered(ti, ti), ti) grows from 1 BLOCK to 1,024; 11 such clauses build on stock but throw IllegalArgumentException: Too many disjunctions to expand with the fix (Disjunctions.java:43-45, default max clause count 1024), as 11 clauses or(ordered(ti, ui), ti) already do on stock (executed). The rewritten query's toString(), equals/hashCode (so query-cache keys) and the number of iterators change too. minExtent() itself is one addition, called when the query is built and once per segment when the scorer is created (IntervalQuery.java:189-196). No benchmark was run.
• Hits and scores: the queries shown here start matching. Where the rewrite does not change the query's shape, scores can only rise, because the sloppy frequency uses minExtent() (ordered(a, a) on a a: 0.333 to 0.5). Queries that the rewrite now splits can lose hits and score lower through LFV-810 until that fix lands, and LFV-291 gains wrong hits (above). Broader monotonicity was not established.
• For identical tokens stacked at one position the new value over-reports, as UnorderedIntervalsSource already does; in every executed stacked case the hits and scores stayed the same.
• Not covered by any validation: custom IntervalsSource implementations, payload-filtered terms at one position, and extend() near position 0 or Integer.MAX_VALUE beyond the executed cases.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: stock + the new test, 5 of 5 fail with assertions on and off; stock + the TestDisjunctionRewrites test, it fails and the other 12 pass.
• + the fix: the whole lucene/queries module, 52 suites, 518 tests (123 in the intervals package, 6 of them new), 0 failures with assertions on and off (5 and 6 skipped); TestStandardQPEnhancements (23), TestUnifiedHighlighterTermIntervals (116), TestMatchHighlighter (6) and TestMatchRegionRetriever (22), 0 failures; :lucene:queries:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis) passes.
• Standalone runner: 5 of 5 pass with assertions on and off; the probe shows the rewrite now splits the OR.
• After the fact checks the two code comments were reworded (comments only) and re-run in a new scratch worktree: checkGoogleJavaFormat passes; the intervals package passes with the fix (8 suites, 123 tests, assertions on and off); on stock the structural test and the 5 repro tests fail as before; the standalone runner passes 5 of 5.
• Earlier, the same change: the finder (3 of 3; 404 of 412 upstream tests, the same 8 testCambridgeMA highlighter failures under every fix configuration it ran, with no stock row recorded; this doc's gradle run of TestUnifiedHighlighterTermIntervals with the fix passed 116 of 116), the independent reproducer (8 of 8; 117 of 117 upstream interval tests) and the audit (16 of 16 probe checks).

Preferred by the intended-lens cross-model reviewer, who places the defect here: group only alternatives known to return one-position intervals, rather than every alternative with minExtent() == 1 (Disjunctions.java:83).

Trade-offs

Keeps minExtent() exact for stacked tokens. A conservative "always a single position" property that defaults to false is correct without any propagation: every source not known to be single-position is split, as different words are today (with the same expansion cost). Propagating it through wrappers that keep it (maxgaps, maxwidth, fixField, ...) only keeps more of today's grouping. It leaves the low score. A comment in splitDisjunctions that its grouping relies on minExtent() being a true minimum would help with either option.

or(phrase(x, ordered(a, a), c), phrase(x, a, c)) instead of phrase(x, or(ordered(a, a), a), c); 1 hit on x a a c and on x a c on stock main and 10.5.2.

Trade-offs

Needs the application to know about the bug and to expand every such OR itself; does not fix the score.

Method: bounded-exhaustive translation validation on the real jar (semi-formal tier).

• First signal (2026-10-26): a hunt unit on spans and intervals checked whether Lucene's default disjunction rewrites keep their documented intent. It was clean except for repeated-source leads, and a probe on the stock jar showed phrase(x, or(ordered(a, a), a), c) missing x a a c while the ordered(a, b) control matched.
• A follow-up unit (2026-10-27, main a9779ef3c193) wrote a reference semantics from the javadoc and compared it with the real classes on every document of 1 to 4 positions over three letters and 6,856 expressions of depth up to 3 (1,100,295 checks; 23,911 failures on stock, spread over several separate bugs). Switchable shadow fixes attributed them: adding only this minExtent() fix to the two known fixes removed the missing-interval failures under phrase and maxgaps over or(.. ordered(a, a) ..); with all seven fixes there were 0 failures at up to 6 positions (11,287,592 checks).
• Then a public-API JUnit on the stock jar, checked by an independent reproducer, reading-only reviews by a model from another family (OpenAI GPT-5.6), an intended-behaviour devil's advocate, an adjudication and a credibility audit.