FastVectorHighlighter loses a match at a position with stacked index-time synonyms when the first stacked token starts a phrase that does not continue

With phrase highlighting (the default), FastVectorHighlighter groups all tokens of one position into one entry of its FieldTermStack, as a circular list in term order. FieldPhraseList starts a candidate with the first of those tokens that has any entry in the query's phrase map, even when that entry is only the first word of a phrase. If the phrase does not continue there, nothing is added and the other tokens of the position are never tried. Index my dogs with an index-time synonym dog stacked on dogs and search dogs OR "dog house": the document matches through dogs (Weight#matches reports offsets 3-7), but getBestFragment returns null instead of my <b>dogs</b>, which dogs alone gives. Search hits and scores are unaffected and nothing fails loudly. Lucene 4.7.0 highlights the input; the loop came with LUCENE-5538 in 4.7.1 (2014), and every release since is affected, up to 10.6.0 and main. A one-file fix that tries each token of the start position passes the repro and every highlighter-module test.

1. One entry per position. FieldTermStack collects the query's terms from the term vector in term order, sorts them by position (FieldTermStack.java:105-106; TermInfo.compareTo compares positions only, :225-227) and links the tokens of one position into a circular next list, removing all but the first from the stack (:108-130). For my dogs the stack holds one entry, position 1: dog -> dogs -> dog (my is not a query term).
2. The phrase map. QueryPhraseMap.add gives a PhraseQuery a non-terminal map for each leading term and marks only the last one terminal; a TermQuery gets a terminal map (FieldQuery.java:398-418).
3. The start loop. The FieldPhraseList constructor pops the entry and walks its alternatives only while getFieldTermMap(...) returns null (FieldPhraseList.java:73-78). dog sorts first and has a (non-terminal) map, so the walk stops at dog. The entry is skipped only when no token has a map (:80-81).
4. The dead end. The next pop finds nothing (:86), so the phrase cannot continue (:95). The one-token candidate [dog] is not a valid term or phrase, because its map is not terminal (FieldQuery.java:460-465), and backtracking runs only while the candidate has more than one token (FieldPhraseList.java:102). Nothing is added (:97-113), and dogs, which shares the popped entry, is never tried. The phrase list is empty and getBestFragment returns null.
5. Order matters. With dog indexed and dogs stacked on it, dog OR "dogs house" is highlighted. The analysis docs leave that order undefined (lucene/core/.../analysis/package-info.java:341-344). A token pushed back after a failed phrase can also become the one tried first: the exhaustive check shows a OR "b b" losing the a of b z a/b (a and b stacked at position 2).
6. Why 4.7.0 was right. Before LUCENE-5538 every stacked token was its own stack entry, so after the dead end at dog the next pop was dogs, which was added. LUCENE-5538 collapsed them into one entry and added the start loop, which stops at the first token with any map.

• Who is exposed. Applications that highlight with FastVectorHighlighter and phrase highlighting on (DEFAULT_PHRASE_HIGHLIGHT = true, FastVectorHighlighter.java:28, default constructor :37-38), on a field with term vectors, positions and offsets (an opt-in field setting FVH requires), analyzed at index time with a chain that stacks tokens.
• Production chains (executed on 5adff78bb168). SynonymGraphFilter + FlattenGraphFilter, as the SynonymGraphFilter javadoc prescribes for indexing (tv -> television; the classic QueryParser query tv OR "television stand" on my tv returns null); KeywordRepeatFilter + PorterStemFilter (running OR "run fast" on she was running returns null). By reading, any other filter that stacks tokens (for example a word-delimiter filter that keeps the original) can do it too.
• The trigger. At a position with a query match, the token tried first (the one that sorts first among the position's tokens the query knows, or one pushed back after a failed phrase) is only the first word of a query phrase, and that phrase does not continue there. Phrase OR phrase is hit too: "dogs bark" OR "dog house" on dogs bark returns null.
• Silent. No exception with assertions on or off. getBestFragment returns null when it was the field's only match; otherwise that occurrence is just not marked. Search hits, counts and scores are correct.
• Not affected in the tested cases, and workarounds. The UnifiedHighlighter and FastVectorHighlighter with phraseHighlight=false both give my <b>dogs</b> for dogs OR "dog house" on my dogs, and both highlight tv OR "television stand" on my tv (SynonymGraphFilter + FlattenGraphFilter). This is shown for the tested inputs only.
• How likely. Rare: each ingredient is ordinary for FVH users with index-time synonyms, but the combination is narrow, and the searches found no report of it in the 12 years since 4.7.1. No scale is needed: one two-word document.

The test sits in org.apache.lucene.search.vectorhighlight only for locality; it uses only public API. ByteBuffersDirectory, a plain IndexWriterConfig with MockSynonymAnalyzer, a stored TextField with term vectors, positions and offsets, one document, single thread. The test body draws nothing at random; as a LuceneTestCase it still gets the framework's per-seed defaults (codec, locale, time zone), and a fixed seed makes a run reproducible. The miss did not depend on the seed in any run (the independent reproducer's three claim tests failed on all 20 random seeds). The count assertion shows the document is a hit; the last assertion fails. The same test is the suggested regression test.

MockSynonymAnalyzer is used because it is the smallest input; the same miss happens with SynonymGraphFilter + FlattenGraphFilter at index time and a classic QueryParser query, and with KeywordRepeatFilter + PorterStemFilter (see "Who is affected").

It needs Lucene 9.1 or later as written (the org.apache.lucene.tests.* packages); the earlier history sweep used era-adapted copies (an inline stacking filter before 6.2, older imports) for 4.x to 9.0.

cp $REPRO/TestFastVectorHighlighterStackedSynonym.java $LUCENE/lucene/highlighter/src/test/org/apache/lucene/search/vectorhighlight/
cd $LUCENE
./gradlew :lucene:highlighter:test --tests org.apache.lucene.search.vectorhighlight.TestFastVectorHighlighterStackedSynonym -Ptests.seed=DEADBEEF
# production setting (assertions off): add -Ptests.asserts=false

# gradle, main 5adff78bb168, seed DEADBEEF, assertions on (the same with -Ptests.asserts=false)
TestFastVectorHighlighterStackedSynonym > testTermAtStackedPositionIsHighlighted FAILED
    java.lang.AssertionError: expected:<my <b>dogs</b>> but was:<null>

# with the recommended fix: BUILD SUCCESSFUL, assertions on and off

# JUnit runner, seed DEADBEEF, assertions on: the same failure on 10.6.0, 10.5.2, 10.0.0, 9.12.3 and 9.1.0

# earlier tests re-run on 5adff78bb168 (JUnit runner, seed 5ADFF78B, assertions on)
finder's test (MockSynonymAnalyzer): count 1, Weight#matches [3,7), dogs alone 'my <b>dogs</b>'; dogs OR "dog house" -> null
independent reproducer: 3 of 5 fail
  testTermOrDeadEndPhraseStackedFilter  FieldPhraseList must hold the TermQuery occurrence expected:<1> but was:<0>
  testTermOrDeadEndPhraseSynonymGraph   expected:<my <b>dogs</b>> but was:<null>
  testPhraseOrDeadEndPhrase             expected:<<b>dogs bark</b>> but was:<null>
  controls (dogs alone, mirrored order) pass
devil's-advocate verifier: 2 of 5 fail
  testSynonymGraphTermOrPhraseParsedQuery  expected:<my <b>tv</b>> but was:<null>
  testKeywordRepeatStemTermOrPhrase        expected:<she was <b>running</b>> but was:<null>
  controls (UnifiedHighlighter, phraseHighlight=false, mirrored order) pass

• A regression of a bug fix. CHANGES for 4.7.1: "LUCENE-5538: Fix FastVectorHighlighter bug with index-time synonyms when the query is more complex than a single phrase." (lucene/CHANGES.txt:12742-12743). The issue text (#6601): "in some situations a query of (P1 OR PQ) returns no results, even though individually, both P1 or P2 by themselves will highlight correctly". The shape is similar but the input is not the same: in #6601 both phrases highlight on their own; here the term alone highlights and the phrase alone matches nothing, yet the disjunction loses the term's highlight.
• The fix author's intent. On #6601, Robert Muir described their first patch: "when we reach a dead-end (nextMap == null), we keep looking as long as the incoming position is the same until we find a match", and the final one: "This should be correct for all crazy combinations." Adrien Grand gave a +1. The committed code keeps looking for the continuation token (FieldPhraseList.java:90-93), but at the start only while no map is found (:75).
• 4.7.0 highlights the input; 4.7.1 returns null, and swapping only the two source files LUCENE-5538 changed (FieldPhraseList.java, FieldTermStack.java) flips the result in both directions. 4.7.0 gets this input right because each stacked token was its own stack entry there; it is not more correct in general (it lacks LUCENE-5538's own fix).
• Documented feature. The package javadoc lists "support phrase-unit highlighting with slops" (package-info.java:26); its worked example is a TermQuery OR a PhraseQuery (Lucene^2 OR "search library"~1, :47-53), whose phrase list holds one entry for the term and one for the phrase (:124-137). Stacked tokens are a documented analysis pattern with undefined order (analysis/package-info.java:341-344), and SynonymGraphFilter tells index-time users to stack with FlattenGraphFilter (SynonymGraphFilter.java:67-68).
• No test pins the current output. TestIndexTimeSynonym's term-or-phrase cases (pc OR "personal computer", TestIndexTimeSynonym.java:167-175, :218) put the term first in sort order, so the dead end is never reached; LUCENE-5538's testBooleanPhraseWithSynonym (TestFastVectorHighlighter.java:714) uses two phrases that both match. All 723 highlighter-module tests pass with the fix.

The strongest case for "intended": FastVectorHighlighter is a greedy heuristic that promises fragments, not every match. On LUCENE-4734 (#5799, open since 2013) a contributor wrote "FVH works great for simple phrase and single term queries but it has so many corner cases", and Adrien Grand ("This is my feeling too") and Simon Willnauer agreed; a broader FieldPhraseList rewrite that also handled "several terms at the same position" was committed and reverted before any release because "it makes the runtime of FastVectorHighlighter very bad". #16820 (open) proposes deprecating FVH. getBestFragments documents a null result "when no fragments created" (FastVectorHighlighter.java:131; getBestFragment itself only says "the best fragment (snippet) string", :107). Users who need exact highlights should use the UnifiedHighlighter.

Why it fails: the revert was in 2013; the defective lines came later, in 2014, from LUCENE-5538, a bug fix whose stated goal this input violates and which turned a correct 4.7.0 result into null. No document states the limitation. This is not a missed corner of a complex query but the loss of a plain term match caused by adding a SHOULD clause that matches nothing. The documented null of getBestFragments ("when no fragments created") does not say that a matching term query may go unhighlighted. The fix is bounded to the tokens of the start position (not the reverted search), and FVH is not deprecated on main. The cross-model intended-behaviour review (OpenAI GPT-5.6, with executed tests) rated the argument weak and the bug real (high); the devil's-advocate verifier found that no intended-behaviour argument survives; the independent reproducer rated the argument medium-strong before rebutting it, and rated the bug real. Expect the answer "won't fix, use the UnifiedHighlighter" anyway, especially without a worst-case cost bound for the fix: a later cross-model review rated the "intended" case moderate for that reason.

Two options. A was built and validated on main 5adff78bb168; B was not built. Until a fix lands, new FastVectorHighlighter(false, true) (no phrase highlighting) or the UnifiedHighlighter highlight the match in the tested cases.

Pop the entry and walk its alternatives; for each one with a map, run the existing longest-phrase search from it, and stop at the first that reaches addIfNoOverlap. The search moves unchanged into a private helper searchFrom(...) that returns whether it did; a dead end restores the stack, as today's backtracking already does. One file, +51/-46.

final TermInfo start = fieldTermStack.pop();
TermInfo alt = start;
do {
  QueryPhraseMap startMap = fieldQuery.getFieldTermMap(field, alt.getText());
  if (startMap != null && searchFrom(fieldTermStack, fieldQuery, field, alt, startMap)) {
    break;
  }
  alt = alt.getNext();
} while (alt != start);

Trade-offs

• Cost: at most one more run of the existing phrase search per extra token at a start position, and only after a dead end: at a position with k stacked tokens, at most k runs instead of one, each walking at most as far as today's search and restoring the stack when it fails. Bounded, not the combinatorial search reverted in LUCENE-4734, but no worst-case benchmark was run, and the passing tests are not a proof that every dead-end path restores the stack. It keeps LUCENE-5538's data structure and tests.
• It keeps term order as the preference among a position's tokens: when several of them start a valid term or phrase (with different boosts, or overlapping other hits), the first in term order still wins, and no test settles which one should.
• It does not cover a related trigger where the first-tried token's candidate is valid but rejected by addIfNoOverlap for overlap (the helper returns true once it calls addIfNoOverlap); that trigger did not lose highlights on main in the earlier probe. The helper's javadoc ("returns true if a phrase was added") should say "offered to addIfNoOverlap".

What the checks validated

• Gradle in a scratch worktree of 5adff78bb168, OpenJDK 25.0.3, seed DEADBEEF: stock, the repro fails with assertions on and off; with the fix it passes, on and off.
• All tests of the highlighter module pass with the fix and the repro in place: 35 classes, 723 tests, 0 failures, 0 skipped (the 13 vectorhighlight classes have 123 tests).
• checkGoogleJavaFormat, ecjLint, forbiddenApisMain and renderJavadoc of lucene/highlighter pass (the build disables the stock javadoc task, which the log shows as SKIPPED, in favour of renderJavadoc).
• The independent reproducer's and the devil's-advocate verifier's suites (production analysis chains included) pass 5 of 5 each with the fix applied on 5adff78bb168.
• In the bounded-exhaustive run, with the study's fix sketches for several highlighter findings on the classpath together (among them a start-token retry like fix A; the run does not isolate fix A), the class "term OR phrase, a clause not highlighted" dropped from 401 cases to 0 and "phrase OR phrase" from 118 to 0. An independent verifier's own equivalent fix passed the same tests.

Make the helper return whether the candidate was actually added (a private boolean variant of addIfNoOverlap, which is public void), so the next token of the start position is tried after an overlap rejection too.

Trade-offs

Covers the related trigger as well, but changes which highlight wins when stacked tokens overlap an earlier hit, which #8644 (LUCENE-7593) shows is already delicate; it needs its own tests.

Method: semi-formal (bounded-exhaustive translation validation). A test program enumerated every single-valued document of 1 to 3 words over {a, b, c, a position hole, and a/b, two tokens stacked at one position with the same offsets} (152 documents) and 2,139 queries (terms, SynonymQuery, phrases with gaps and slop, MultiPhraseQuery, and term OR phrase / phrase OR phrase booleans). It compared the offsets in FastVectorHighlighter's FieldPhraseList (and the classic Highlighter's marks) with an exact match semantics written independently from the code, cross-checked against Weight#matches on every hit: 153,082 document-query checks on the stock main jars.

• The mismatch class "term OR phrase, a matching clause not highlighted" had 401 cases (for example a OR "b b" on b z a/b highlights nothing).
• Code reading traced it to the start-position loop; the case was reduced to a public-API JUnit test on the stock jar, which failed with assertions on and off.
• With the study's fix sketches for several highlighter findings on the classpath together (among them a start-token retry like fix A) the class dropped to 0 (and "phrase OR phrase" from 118 to 0); the remaining highlighter mismatches belong to other, already reported defects.