The classic Highlighter counts the empty positions before a MultiPhraseQuery as gaps and marks pairs that do not match

The classic Highlighter with QueryScorer converts a MultiPhraseQuery into a SpanNearQuery. WeightedSpanTermExtractor walks the position slots from 0 to the last position and counts every empty slot as a gap, including the slots before the phrase's first position, and adds that count to the slop. A MultiPhraseQuery that starts at position k > 0 therefore gets k extra units of slop. QueryBuilder, and so the classic QueryParser, build such a query for a quoted phrase that starts with a removed stop word and contains a same-position synonym: "the dogs bark" becomes [dogs|dog]@1 bark@2, and on dogs often bark but my dogs bark the highlighter also marks dogs often bark, although Weight#matches reports only positions 5-6. The PhraseQuery branch of the same method measures gaps from the first position and is correct. Search hits, counts and scores are not affected. The UnifiedHighlighter shows the same marks with weight matches off (its default in 6.3.0 to 8.11.x); its default since 9.0 is exact. Every release from 2.4.0 to 10.6.0 and main is affected (17 releases and main run, the rest by code history; 10.6.0 by reading). A one-condition fix passes all 732 tests of the highlighter module, the repro tests included.

• Classic Highlighter: org.apache.lucene.search.highlight.Highlighter with a QueryScorer. The scorer asks WeightedSpanTermExtractor which words of the text belong to a match of the query; for phrases it converts the query to a SpanQuery and runs it on a one-document index of the text.
• Positions: each word of a phrase query has a position. A removed stop word keeps its position, so "the dogs bark" gives dogs position 1 and bark position 2. MultiPhraseQuery documents its positions as relative: shifting them all by the same amount does not change what matches.
• MultiPhraseQuery: a phrase with several alternatives at one position, here [dogs|dog]. QueryBuilder builds one when the analyzer stacks a token at the same position, for example a single-word synonym. Without a stacked token the same input becomes a PhraseQuery.
• Slop of a SpanNearQuery: "the maximum number of intervening unmatched positions" between its clauses (SpanNearQuery.java:38-40).

// MultiPhraseQuery branch
final List<SpanQuery>[] disjunctLists = new List[maxPosition + 1];   // :200  slots 0..maxPosition
...
for (List<SpanQuery> disjuncts : disjunctLists) {                    // :218
  if (disjuncts != null) {
    clauses[position++] = new SpanOrQuery(disjuncts.toArray(SpanQuery[]::new));
  } else {
    ++positionGaps;          // :222  also for the empty slots BEFORE the first used slot
  }
}
SpanNearQuery sp = new SpanNearQuery(clauses, slop + positionGaps, inorder);   // :232, inorder = (slop == 0)

// PhraseQuery branch, for comparison: relative to the first position
positionGaps = Math.max(0, positions[positions.length - 1] - positions[0] - positions.length + 1);  // :154-155

1. For [dogs|dog]@1 bark@2 the array has 3 slots. Slot 0 is empty (the dropped the); slots 1 and 2 hold the clauses.
2. The loop counts slot 0 as a gap: positionGaps = 1, although nothing lies between the two clauses.
3. The span query is spanNear([spanOr([dogs, dog]), spanOr([bark])], 1, true): ordered, slop 1. A trace prints slop 0, 1 and 2 for the same query starting at position 0, 1 and 2.
4. On dogs often bark but my dogs bark that span query also matches dogs(0) ... bark(2), so the extractor records bark at positions [0,2] and [5,6] instead of [5,6] only, and QueryScorer marks the first pair too.

Where the input comes from: QueryBuilder.analyzeMultiPhrase starts at position = -1 and adds each token's increment (QueryBuilder.java:464-497), and position increments are on by default (QueryBuilder.java:198-206), so a removed leading stop word makes the first position 1. The classic QueryParser builds the same query from "the dogs bark".

With two leading stop words ("the a dogs bark", positions 2 and 3) a pair with two words between them is marked; slop 1 from position 1 marks a pair that needs slop 2. A text with no match can get marks too: [a|x]@1 b@2, slop 0, on a z b gives <B>a</B> z <B>b</B>, where the query at positions 0 and 1 returns no fragment.

• Who. Users of the classic Highlighter with QueryScorer (position-aware by default since 2.9.0), on re-analysed text and on term-vector token streams; and users of the UnifiedHighlighter with withWeightMatches(false) (a builder option since 9.1.0), whose PhraseHelper reuses the same extractor. From 6.3.0 to 8.11.x that was the UnifiedHighlighter's default path. By reading, any configuration that keeps strict phrase highlighting but loses weight matches gets there too: weight matches are set only when multi-term handling, strict phrases and passage relevancy are all on (UnifiedHighlighter.java:518-527), so withHandleMultiTermQuery(false) or withPassageRelevancyOverSpeed(false) also use PhraseHelper (:1219-1235). The default UnifiedHighlighter of 9.0 and later uses Weight#matches and is exact.
• Trigger. A MultiPhraseQuery whose smallest position is above 0. In practice: a query analyzer that removes stop words and stacks same-position tokens (single-word synonyms, or any filter that emits a token with position increment 0 and position length 1; a longer token sends QueryBuilder to its graph path, which builds phrase queries per path instead), and a quoted phrase that starts with a stop word and contains such a token. With Lucene's SynonymGraphFilter the filter order matters: synonyms before stop words trigger it for this example, stop words first gave positions 0 and 1 and an exact highlight (traced, this example only). Hand-built queries that start above 0 do the same. Plain stop-word phrases become a PhraseQuery and are not affected.
• Effect. Extra marks: pairs of the phrase's words that are up to k positions further apart than the query allows, where k is the first position (one per leading removed word); sloppy phrases too. A text without a match can get marks. Hits, counts and scores do not change, and there is no error with assertions on or off. By reading, not run: QueryScorer scores fragments by the query terms they contain, so the extra marks can also change which fragment is picked.
• Scale. A bounded-exhaustive harness (152 documents of 1-3 words, 2,139 queries, 153,082 checks, one per query hit, against an exact phrase reference cross-checked with Weight#matches) found this class in 92 cases in each of the two token-stream modes, and none for a PhraseQuery with the same positions.
• How likely. Rare: each ingredient is ordinary, but the combination plus a near-miss pair in the text is uncommon, and nobody has reported it in 18 years. The classic Highlighter is the older highlighter; #16820 (open) proposes deprecating it, nothing is deprecated yet.
• Workaround. Use the UnifiedHighlighter with weight matches (its default since 9.0), or shift the MultiPhraseQuery so its positions start at 0 before highlighting; the controls show both highlight exactly.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[dogs often bark] but my <B>dogs</B> ...> but was:<[<B>dogs</B> often <B>bark</B>] but my <B>dogs</B> ...>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.highlight.TestMultiPhraseLeadingPositionHighlight.testFirstPositionAboveZeroAddsNoSlop(TestMultiPhraseLeadingPositionHighlight.java:36)

• Positions are relative by contract. MultiPhraseQuery.Builder.add(Term[], int): "Allows to specify the relative position of terms within the phrase" (MultiPhraseQuery.java:122); getPositions(): "Returns the relative positions of terms in this phrase" (:176). Weight#matches gives the same positions for the query at 0,1 and at 1,2, and PhraseQuery.rewrite subtracts positions[0] (PhraseQuery.java:309-314).
• The same method already does it right for PhraseQuery. Since LUCENE-5503 (5.4.0) that branch uses positions[last] - positions[0] - n + 1 (:154-155). Its CHANGES entry says the extractor "converts a PhraseQuery to an equivalent SpanQuery" (CHANGES.txt:10042-10045), and PhraseHelper.java:107 says "WSTE will convert to an equivalent SpanQuery" (a code comment; PhraseHelper is @lucene.internal).
• The scorer's stated goal. QueryScorer "attempts to score only those terms that participated in generating the 'hit' on the document" (QueryScorer.java:33-37).
• No test covers it. The upstream highlighter tests that use a MultiPhraseQuery all start at position 0: TestHighlighter.java:820-849 (two tests) and :1732-1740 (a one-term query), and TestUnifiedHighlighterStrictPhrases.testSynonyms (:261-279, positions 0 and 1). The contributed patch that introduced the loop (LUCENE-794, 2008) tested only positions starting at 0. No CHANGES, MIGRATE or won't-fix entry mentions it.
• History. LUCENE-7417 (2016) edited this loop for a one-clause crash and kept the leading-slot count.

Strongest case (an intended-behaviour checker and a reviewer from another model family, OpenAI GPT-5.6): the classic phrase-to-span conversion is a known approximation. On LUCENE-2003 markrmiller called adding holes to the slop "one - uh - sloppy fix ... I think that keeps a fairly good approximation"; QueryScorer only "attempts" to score participating terms; the sandbox PayloadSpanUtil copies the same loop, so it looks like a convention; and #16820 proposes deprecating the classic Highlighter.

Why it fails: the accepted approximation is about holes between positions (the sibling bug LFV-827). A slot before the first position is not a hole, and every formula committers discussed gives 0 for it: "maxPos-minPos+1-numTokens" (yonik, LUCENE-2003) and LUCENE-5503's positions[last] - positions[0] - n + 1. Shifting the query by a constant, which the API defines as meaningless, changes the highlight. A copied loop (in a sandbox class marked @lucene.experimental) is not a decision, a proposed deprecation is not a decision, and the UnifiedHighlighter's weight-matches-off path is not part of that proposal. The reviewer rated the argument moderate and outweighed; the checker and the adjudication: not intended.

Fix A and fix B were built and run in a scratch worktree at a9779ef3c193 with the whole :lucene:highlighter:test task (the three repro test classes added). Option C was not run. The sandbox PayloadSpanUtil.java:137-152 has the same loop; a fix may want to cover it (not run). Related: LFV-827 (interior holes of an exact phrase become slop, same method) and LFV-397 (int overflow of the same slop + positionGaps sum).

position counts the clauses filled so far, so empty slots before the first clause are skipped; the array ends at the last position, so there are no trailing empty slots.

         for (List<SpanQuery> disjuncts : disjunctLists) {
           if (disjuncts != null) {
             clauses[position++] = new SpanOrQuery(disjuncts.toArray(SpanQuery[]::new));
-          } else {
+          } else if (position > 0) {
+            // only unused positions between the first and the last query position are gaps
             ++positionGaps;
           }
         }

Trade-offs

• Smallest change; interior holes keep today's handling (that part is LFV-827's topic).
• One comparison per empty slot; not benchmarked.
• The array still has slots below the first position (option C removes them).

What the checks validated

• Gradle on a9779ef3c193, seed DEADBEEF: :lucene:highlighter:test with the 3 repro classes passes with assertions on and off: 37 classes, 732 tests, 0 failures (722 upstream tests, e.g. TestHighlighter 59, TestHighlighterPhrase 7, TestUnifiedHighlighter 156, plus the 10 repro tests).
• checkGoogleJavaFormat, forbiddenApisMain, forbiddenApisTest, ecjLintMain and ecjLintTest pass with the fix and the repro test.
• With the standalone runner (fix as a shadow class): the independent reproducer's 8 of 8 and a reviewer's 2 of 2 pass, and 79 classic highlight-package tests; the intended-behaviour checker's probe now highlights exactly for the parsed query, for starts 0-3, and in the UnifiedHighlighter with weight matches off.
• In the bounded-exhaustive harness, with this and two sibling fix sketches (which change other code) on the classpath, the class drops from 92 to 0 cases per token-stream mode.

positionGaps = (maxPosition - minPosition + 1) - distinctPositions, with minPosition the smallest position, and the else branch removed (+6/-3). It is the formula yonik proposed on LUCENE-2003 for the PhraseQuery branch and the form that branch uses today, so both branches then read alike.

-        int positionGaps = 0;
+        int minPosition = positions[0];
+        for (int p : positions) {
+          minPosition = Math.min(minPosition, p);
+        }
+        // gaps are the unused positions between the first and the last
+        int positionGaps = (maxPosition - minPosition + 1) - distinctPositions;
         ...
           if (disjuncts != null) {
             clauses[position++] = new SpanOrQuery(disjuncts.toArray(SpanQuery[]::new));
-          } else {
-            ++positionGaps;
           }

Trade-offs

Same result as A, a few more lines; one extra pass over the positions.

What the checks validated

• Gradle on a9779ef3c193, assertions on: :lucene:highlighter:test with the 3 repro classes passes, 37 classes, 732 tests, 0 failures; the same lint tasks pass. With the standalone runner the independent reproducer's 8 of 8 and 70 upstream highlight tests pass. Not run with assertions off.

Subtract the smallest position from every position before building disjunctLists, as PhraseQuery.rewrite does.

Trade-offs

Same result as A and B for non-negative positions, and the array no longer has slots below the first position; more lines. Not run. MultiPhraseQuery.Builder.add does not reject a negative position: A and B would still index disjunctLists[-1] there, as today, and C would not.

Until a fix ships: highlight with the UnifiedHighlighter and weight matches (its default since 9.0), or rebuild the MultiPhraseQuery with positions starting at 0 before passing it to QueryScorer.

Trade-offs

Application code only; it does not fix Lucene.

What the checks validated

• Main a9779ef3c193: the same query at positions 0,1 highlights exactly (the repro's control 2), and the default UnifiedHighlighter marks only dogs bark for the query at 1,2.

• Harness (bounded-exhaustive translation validation of the stock jars). All single-valued documents of 1-3 words over a small alphabet (152 documents, including position holes and stacked tokens) against 2,139 queries: term, synonym, phrase and multi-phrase queries with holes, slop 0-2 and start position 0 or 1, and Boolean combinations; 153,082 checks, one per query hit. Each highlight was compared with an independent exact phrase reference, itself cross-checked against Weight#matches on every hit (0 calibration mismatches), for re-analysed text and for term-vector token streams.
• Signal. The class classic EXTRA mpq+start: 92 cases per mode, for example [a]@1 [b]@2 on a a b marks the first a. The PhraseQuery with the same positions had no such class, which pointed at the MultiPhraseQuery branch.
• Reduction. Code reading found the slot loop; the finder wrote the public-API JUnit with a QueryBuilder-built query and ran it on main and on 10.5.2 and 9.12.3.