Intervals.analyzedText drops stacked synonyms when another token of the text spans several positions (e.g. a multi-word synonym)

Intervals.analyzedText turns analyzed text into an interval source. When no token spans more than one position, tokens stacked at one position (a single-word synonym, or a stem next to the original word) are ORed. As soon as one token spans two or more positions (a multi-word synonym, a WordDelimiterGraphFilter catenation), the whole text goes through IntervalBuilder.analyzeGraph, which, at every position without a multi-word alternative, keeps only the first finite string and asserts there is no second one. Stacked tokens of equal length are exactly such a position with several strings, so all but the first emitted token are dropped with assertions off, and AssertionError is thrown with assertions on. With wifi => wi fi and dog => canine, analyzedText("wifi dog") finds 2 of 4 documents and misses "wifi dog"; core QueryBuilder finds all 4. It is reached through both analyzedText overloads and the plain-term and quoted-text operands of the flexible StandardQueryParser fn: functions (not the patterns of fn:wildcard and fn:fuzzyTerm), in every release from 9.1.0 to 10.5.2 and on main. The index is not affected.

• Stacked token: a token with position increment 0, at the same position as the token before it, for example canine on top of dog. Interval queries should treat stacked tokens as alternatives, or(canine, dog).
• Token graph: a token stream in which some token spans more than one position (position length > 1), for example wifi spanning the two positions of wi fi. The paths through the graph are its finite strings: wi fi dog, wi fi canine, wifi dog, wifi canine.
• Side path: GraphTokenStreamFiniteStrings.hasSidePath(state) is true when alternatives of different length start at a state (wifi versus wi fi). Equal-length stacked tokens (dog, canine) are not a side path: they leave the state together and arrive together.

1. IntervalBuilder.analyzeText notes whether any token has position length > 1 (IntervalBuilder.java:95-98). If so, the whole text goes to analyzeGraph (:109-111) instead of analyzeSynonyms, which would OR the stacked tokens (:172-199).
2. analyzeGraph (:201-237) walks the graph segment by segment. A segment with a side path becomes an OR of phrases, one per finite string (:215-228). Any other segment runs TokenStream ts = it.next(); once and then assert it.hasNext() == false; (:230-233).
3. For wifi dog the last segment holds canine and dog: no side path, two finite strings (the diagram probe prints hasSidePath=false finiteStrings(in order)=["canine", "dog"]). Line 231 keeps canine, line 233 throws under -ea, and under -da dog is dropped. The built source is or(BLOCK(wi,fi,canine),BLOCK(wifi,canine)).
4. Which token survives is the one the analyzer emits first: SynonymGraphFilter emits the synonym canine before the original dog, so the user's own word is the one lost.

With assertions on, every row except the controls throws java.lang.AssertionError while the query is built.

• Entry points. Both Intervals.analyzedText overloads, and the plain-term and quoted-text operands of the flexible StandardQueryParser fn: functions (the grammar's IntervalText rule, StandardSyntaxParser.jj:681-686, then AnalyzedText.java:38); the patterns of fn:wildcard and fn:fuzzyTerm are not analyzed. Core QueryBuilder and the parsers built on it are not affected.
• What it takes. In one query text, a token that spans several positions (a multi-word SynonymGraphFilter rule, or WordDelimiterGraphFilter catenation) and a token stacked at another position (a single-word synonym, or KeywordRepeatFilter with a stemmer). Mixed multi-word and single-word synonym lists are an ordinary query-time setup.
• Opt-in. The intervals package is marked experimental, and both executed triggers need query-time filters configured on purpose (Lucene's ready-made analyzers were not checked one by one); the index side needs nothing special (default writer, codec and searcher).
• Deterministic. Every such query text loses all but the first stacked token at each position that no multi-position token spans (stacked tokens inside such a span are kept), in every (maxGaps, ordered) mode and at any index size; it does not depend on the documents.
• Silent in production. With assertions off nothing signals the loss; the lost token can be the word the user typed. With assertions on, building the query throws java.lang.AssertionError (through StandardQueryParser it reaches the caller as that Error, not as a QueryNodeException).
• Negation flips it. A MUST_NOT clause or the subtrahend of fn:notWithin built this way returns documents it should exclude, including the one with exactly the excluded words (executed). Rated not a security issue: the dropped token is fixed by the query text and the analyzer, so index contents play no part (documents matter only if an application builds its query text from them).
• Downstream (code reading only, not executed). Solr's {!intervals} parser, merged in 2026 and not yet released, calls Intervals.analyzedText. Elasticsearch and OpenSearch carry their own copies of the same branch; a Lucene fix does not change those copies.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError
       org.apache.lucene.queries.intervals.IntervalBuilder.analyzeGraph(IntervalBuilder.java:233)
       org.apache.lucene.queries.intervals.IntervalBuilder.analyzeText(IntervalBuilder.java:111)
       org.apache.lucene.queries.intervals.Intervals.analyzedText(Intervals.java:645)
       org.apache.lucene.queries.intervals.TestAnalyzedTextGraphStackedToken.testStackedTokenInGraph(TestAnalyzedTextGraphStackedToken.java:50)

• Javadoc. analyzedText "Returns intervals that correspond to tokens from a TokenStream returned for text by applying the provided Analyzer" (Intervals.java:601-603). A stacked token is a token of that stream.
• The class itself. analyzeSynonyms ORs stacked tokens (IntervalBuilder.java:172-199), and the side-path branch ORs every finite string (:215-228). Whether dog is honoured should not depend on an unrelated word having a multi-word synonym.
• Core parity. Core QueryBuilder keeps stacked tokens on the same graph. The executed control, createPhraseQuery, goes through analyzeGraphPhrase, one phrase per finite string (QueryBuilder.java:340-343, :566-583); by code reading, analyzeGraphBoolean walks the same segments and keeps every stacked term of a segment without a side path through graph.getTerms(start) (:540-556).
• Core test. TestGraphTokenStreamFiniteStrings#testGraphWithRegularSynonym (fast|speedy wi|wifi fi network) asserts hasSidePath(0) == false and then the two finite strings fast and speedy (:467-473), since LUCENE-7638 (2017): the input the interval builder asserts away.
• Upstream interval tests stack tokens only inside side paths (TestIntervalBuilder#testGraphSynonyms*, testGraphTerminatesOnGap), which is why it stayed hidden; testSimpleSynonyms* assert the OR on the non-graph path.
• History. The branch came with the method (LUCENE-10220, #427, 2021) and has not changed since; the introducing issue and PR never discuss graphs or synonyms, and no CHANGES, MIGRATE or javadoc text says stacked tokens are dropped.

Strongest case for "intended": the intervals package is experimental; the javadoc never spells out synonym semantics; SynonymGraphFilter's javadoc says that fully correct positional queries over multi-token synonyms need a TermAutomatonQuery; the code was ported as-is from another engine that still has it; maybe the assert was valid when written.

Why it fails: the TermAutomatonQuery note is about multi-token replacements, while the token dropped here is a single-position one, often the user's own word; the class and core QueryBuilder keep stacked tokens on every other path; hasSidePath has meant "different length" since 2017, so the assert was never valid; and an AssertionError on stock filters through public API is not a design decision. Every checker concluded "not intended". One honest caveat: the miss with assertions off rests on this implicit contract, while the AssertionError with assertions on is unambiguous.

Option A was validated through gradle in a scratch worktree; option B through the standalone harness only.

the same hunk as the finder's sketch. In such a segment every transition from the start state reaches the same state, so each finite string is one token; analyzeTerms keeps the existing hole handling. The result is an or(...) of the stacked terms, as on the non-graph path; with a gap before the position, option A extends each term where analyzeSynonyms extends the OR (or(extend(a,h,0), extend(b,h,0)) against extend(or(a,b),h,0)), with the same matches in every tested case (the bounded-exhaustive harness, holes included, finds no graph-plus-stacked mismatch with the fix). A structural test for TestIntervalBuilder (stacked positions before and after a side path) fails on stock and passes with the fix.

      } else {
        // no side path: every finite string is a single token at this position, and there is
        // more than one if synonyms are stacked here
        List<IntervalsSource> synonyms = new ArrayList<>();
        Iterator<TokenStream> it = graph.getFiniteStrings(start, end);
        while (it.hasNext()) {
          synonyms.addAll(analyzeTerms(it.next()));
        }
        if (synonyms.size() == 1) {
          clauses.add(synonyms.get(0));
        } else {
          clauses.add(Intervals.or(synonyms.toArray(IntervalsSource[]::new)));
        }
      }

Trade-offs

• No maxClauseCount check, unlike the side-path branch (IntervalBuilder.java:221); the non-graph path has none either, and the number of alternatives is the number of tokens the analyzer emitted at that position.
• synonyms.size() == 1 is only a shortcut: Intervals.or of one source returns that source. The OR order does not matter, since disjunction equality is set-based.
• It relies on each finite string of a no-side-path segment being a single token (the code comment says so and every tested graph agrees, but getFiniteStrings does not document it); if that changed, addAll would turn a phrase into an OR of its words. The PR can assert one term per finite string, or use option B.
• Not reviewed by a committer.

What the checks validated

• Stock: the new test fails 2 of 2 with assertions on and off; the structural test fails (got ORDERED(term1,or(BLOCK(term3,term4),term2),term5)).
• Fix: every interval suite of lucene/queries, 8 classes, 120 tests (117 upstream plus the 3 new), 0 failures; the whole :lucene:queries:test, 52 classes, 515 tests, 0 failures, with assertions on and off.
• Fix: the match and interval highlighter suites (4 classes, 164 tests; TestMatchHighlighter uses analyzedText and fn: functions) and TestStandardQPEnhancements (23 tests), 0 failures; tidy passes (the fix hunk saved after it equals the finder's) and :lucene:queries:check -x test passes.
• Harness with the fix shadowed in: the SynonymGraphFilter/fn:or test 3/3, an independent reproducer's 6-test JUnit 6/6, and the bounded-exhaustive harness drops the graph-plus-stacked failures to 0 over 208,009,964 checks; the mismatch categories left, with and without the fix, all involve a term repeated across segments (known RepeatingIntervalsSource behaviours and OR minimal-interval semantics the reference does not reproduce), not this bug.

one extend(Intervals.term(term), posInc - 1) per token returned by graph.getTerms(start), then Intervals.or(...) of them. Mirrors QueryBuilder.analyzeGraphBoolean (QueryBuilder.java:541).

Trade-offs

Skips building one token stream per stacked token, but repeats the hole handling of analyzeTerms by hand and needs one more import.

What the checks validated

Harness only: the new test 2/2, TestIntervalBuilder with the structural test 13/13 and the SynonymGraphFilter test 3/3, with assertions on and off. Not run through gradle or the wider suites.

Add the side-path branch's TooManyClauses check (IntervalBuilder.java:221-223) to either option.

Trade-offs

Consistency only; the non-graph path has no such check today. Untested.

State in the javadoc that stacked tokens are dropped when the text is a token graph.

Trade-offs

Leaves the AssertionError and contradicts the non-graph path and QueryBuilder.

Method: bounded-exhaustive translation validation (a hunt unit, 2026-10-26, main ea57360912ed). No model checker, SMT solver or prover was involved.

• The finder (an AI tool) wrote an independent reference semantics for query token graphs: segments with alternatives, holes and multi-position side paths, and when a document matches in each (maxGaps, ordered) mode. While writing it, it suspected the defect by reading IntervalBuilder.analyzeGraph.
• The harness (AtTV.java, public API only) compared every 1- and 2-segment query from a pool of 68 segments against every document of 1 to 4 positions over {a, b, c, d, z} in 6 modes: 24,882,731 checks on stock main with assertions off, plus 315,197,137 in larger runs on separate build machines. The failure category "graph plus stacked token" is this bug; with the fix shadowed it is 0 (208,009,964 checks).
• Vacuity check: three deliberately broken copies of IntervalBuilder (ignore holes, keep only the first synonym on the non-graph path, unordered side paths) were each caught in their own category.
• Then a public-API JUnit, a parser reachability probe, independent verifiers, an adjudication, a release sweep, a prior-report search and a security screen (see Independent checks).