A reused classic Highlighter with SimpleSpanFragmenter cuts the next text differently, because start() does not reset waitForPos

The classic Highlighter keeps one Fragmenter and calls its start(text, tokenStream), documented as "Initializes the Fragmenter", before every text. SimpleSpanFragmenter, the span-aware fragmenter that keeps phrase matches whole, stores in waitForPos the position just after a match that has started and refuses to start a new fragment until its position counter gets there. Its start() resets the counter, the fragment count, the text size and the attributes, but not waitForPos.

So when a phrase, multi-phrase or span match ends a text, the next text highlighted by the same Highlighter inherits the pending position: no fragment boundary until the counter reaches it, then one-word fragments while the fragment count catches up, a different best fragment, and sometimes a phrase split in two. With the phrase "search library" at size 20 the next text's first fragment is 70 characters; at the default size 100 the best fragment is 491 characters instead of 114. Search hits and scores are unaffected. The omission dates from the class's creation in 2008 (2.4.0) and reproduces on every release to 10.6.0 and on main; the fix is one line.

• Fragmenter: the policy that cuts a text into fragments. Highlighter holds one (Highlighter.java:44, default SimpleFragmenter; setTextFragmenter, :423-425) and calls start(text, tokenStream) at the start of every getBestTextFragments call (:194), which every getBestFragment(s) method reaches. Fragmenter#start: "Initializes the Fragmenter. You can grab references to the Attributes you are interested in from tokenStream and then access the values in isNewFragment()" (Fragmenter.java:28-35).
• SimpleSpanFragmenter: "breaks text up into same-size fragments but does not split up {@link Spans}. This is a simple sample class." (SimpleSpanFragmenter.java:26-29). It gets the query's matches in the current text from QueryScorer, which re-extracts them for every text in init (QueryScorer.java:170-181, called at Highlighter.java:175) and stores each as PositionSpan(start, end) with an inclusive end (WeightedSpanTermExtractor.java:332).

1. isNewFragment() adds the token's position increment to position (SimpleSpanFragmenter.java:63). While waitForPos > position it returns false at once, before the span lookup and the size test (:65-69).
2. When a span of the current token starts at position, it sets waitForPos = positionSpan.end + 1 (:76-81).
3. Otherwise a new fragment starts when the token's end offset reaches fragmentSize * currentNumFrags and at least half a fragment of text is left (:84-90).
4. The Highlighter calls isNewFragment() only when a new token arrives (Highlighter.java:209-224) and flushes the last token group without another call (:241-254). A span that ends the text therefore leaves waitForPos set.
5. start() resets position = -1, currentNumFrags = 1, textSize and the three attributes (:98-106), not waitForPos (:36). The next text gets no boundary until its counter reaches the old value; then the size test, still at the first fragment's limit, fires at every token until currentNumFrags catches up; and while the stale wait holds, a match that starts is never registered, so it can be cut in two.

In hit 1, "search library" is at token positions 9-10. The fragmenter's counter runs one token behind (the sibling LFV-859), so it sees the match start at library and sets waitForPos = 10 + 1 = 11. In hit 2 the counter reaches 11 at the 13th word, reader: the first 12 words are one fragment.

• Who is exposed. Code that reuses one classic Highlighter with SimpleSpanFragmenter (opt-in; the default is SimpleFragmenter, Highlighter.java:44) for several texts: hits, fields, or the values of a multi-valued field. Reuse is the Highlighter's design (one fragmenter field, start() per text) and the package example's pattern (package-info.java:36-59, one Highlighter for 10 hits and two fields, with the default fragmenter).
• What it takes. A positional query (phrase, MultiPhraseQuery or span query, also inside a BooleanQuery) whose match ends the previous text (its last token, or with the stock counter also the second-to-last), and a previous text long enough that the stale position lies beyond the next text's first fragment boundary (at size 100 roughly more than one fragment of previous text, by reading). A plain TermQuery or SynonymQuery has no position spans and never sets waitForPos.
• What the damage looks like. One oversized fragment (70 characters at size 20; 491 instead of 114 at the default size 100), then one-word fragments, a different best fragment, and sometimes a phrase split across fragments (1,632 such splits in the finder's bounded check of tiny texts). Repeating getBestFragments on the same text gives a different answer the second time. Silent: no error.
• What is not affected. Search hits, counts and scores; SimpleFragmenter and NullFragmenter reused (same output as new ones).
• How likely. Rare: opt-in fragmenter, reuse, a positional query and a previous text that ends with a match. No user report has been attributed to it in 18 years.
• Scale. Nothing depends on index size or concurrency; two short strings reproduce it.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[Highlighting picks[,  short fragments of,  text around each,  match so that a reader,  of the] <B>search</B> <B>li...> but was:<[Highlighting picks[ short fragments of text around each match so that a,  reader,  of,  the, ] <B>search</B> <B>li...>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.highlight.TestSimpleSpanFragmenterReuse.testReusedHighlighterFragmentsLikeAFreshOne(TestSimpleSpanFragmenterReuse.java:51)

• Contract. Fragmenter#start: "Initializes the Fragmenter" (Fragmenter.java:28-35). Highlighter keeps one fragmenter (:44, :423-425) and calls start for every text (:194). The contract is implicit: no javadoc says in so many words that a reused fragmenter must cut a text as a new one does; the case rests on start() being the per-text initializer, the one-fragmenter design and the points below.
• The class itself. start() resets position, currentNumFrags, textSize and the attributes (SimpleSpanFragmenter.java:100-105). waitForPos holds a position of the previous text, which means nothing in the next one: QueryScorer.init re-extracts the spans for every text (QueryScorer.java:170-181). SimpleFragmenter.start resets its own state (SimpleFragmenter.java:47-50).
• Tests. TestHighlighter#testHighlightingSynonymQuery reuses one Highlighter and one SimpleSpanFragmenter for documents 2 and 3 (TestHighlighter.java:238-258, reuse at :248-256); a SynonymQuery has no position spans, so it never sees the bug. No test or javadoc says a fragmenter is single-use.
• Intent. Michael McCandless asked on LUCENE-1685 (#2759, 2009-06-12) for a way "to not have to make a SpanScorer (and, by extension, fragmenter) per query, but instead make it up-front", and later "Should we also default the fragmenter to SimpleSpanFragmenter?" LUCENE-1685 made the span-aware QueryScorer reusable in 2.9.0; the fragmenter's own state was not discussed.
• Precedent. LUCENE-2229 (#3305): "Fix Highlighter's SimpleSpanFragmenter when multiple adjacent stop words following a span can unduly make the fragment way too long" (lucene/CHANGES.txt:9891-9893, 5.4.1), with the test assertion "Fragment must be less than 60 characters long" (TestHighlighter.java:435). It changed the same waitForPos check, within one text.

The intended-behaviour checker's strongest case: the javadoc calls the class "a simple sample class", and most upstream tests build a new SimpleSpanFragmenter per document, which could suggest single use.

Why it fails: the per-document line in those tests is a leftover of the 2008 design, when the SpanScorer took one text's token stream in its constructor and had to be built per document, together with its fragmenter; in the LUCENE-1685 commit 10b41d2dce2 the HighlighterTest diff moves the scorer and the Highlighter out of the loop and keeps the setTextFragmenter(new SimpleSpanFragmenter(...)) line inside it, changing only its argument. The class is maintained public API (LUCENE-1389 changed it in 2.9.0, LUCENE-2229 fixed it in 5.4.1), "sample" does not license output that depends on an unrelated earlier text, and the cross-model reviewer rated the intended-behaviour argument weak.

One fix, validated with gradle on 645d4fe14a20; a workaround for released versions; and one alternative that is not recommended.

one added line after position = -1; in SimpleSpanFragmenter.start(). It makes start() initialize the last per-text field it missed.

   public void start(String originalText, TokenStream tokenStream) {
     position = -1;
+    waitForPos = -1;
     currentNumFrags = 1;
     textSize = originalText.length();

Trade-offs

None found. The first text of every fragmenter is unchanged; later texts get what a new fragmenter gives; no cost. Independent of the sibling fixes (LFV-858, LFV-859): the finder's test and 2 of the independent reproduction's tests still fail with both sibling fixes applied, and pass with this one.

What the checks validated

• Gradle, scratch worktree at 645d4fe14a20: the fix plus the paste-ready test plus the finder's and the independent reproduction's tests, the whole :lucene:highlighter:test task with assertions on and off: 37 classes, 733 tests, 0 failures (34 upstream classes with 722 tests, TestHighlighter 59; the 3 reproducing classes 5 + 1 + 5 tests).
• ecjLintMain, ecjLintTest and checkGoogleJavaFormat of the highlighter module pass (eclint and error-prone were not configured in that run and did not apply).
• Before the fix the paste-ready test fails 4 of 5 (the control passes). The same reset shadowed into 2.4.0, 2.9.0 and main makes the history repro pass.

Call highlighter.setTextFragmenter(new SimpleSpanFragmenter(scorer, size)) before each text.

Trade-offs

One allocation per text; every caller has to know about it.

What the checks validated

The history repro's control does exactly this and matched a new Highlighter in every run.

A javadoc note instead of a code change.

Trade-offs

Contradicts Fragmenter#start ("Initializes the Fragmenter"), the one-fragmenter design of Highlighter and the upstream test that reuses one; a one-line reset is simpler than a new usage rule.

Method: model-based, bounded-exhaustive differential check on the stock jar (the study's "semi-formal" tier). A hunt unit, a transducer model of Highlighter.getBestTextFragments composed with SimpleSpanFragmenter, 2026-10-28, main a9779ef3c193.

• While writing the fix-sketch shadow class for the model, the finder read that start() resets position and currentNumFrags but not waitForPos.
• ReuseProbe then ran every pair (first text of at most 4 tokens, second text of at most 6 tokens, over {a, c, x}) with PhraseQuery(a c) and fragment sizes 1 to 8 through a reused and a new Highlighter: 160,425 of 1,048,320 second-text results differ, 1,632 of them split the phrase only when reused. Smallest example: size 4, first text a c, second text a a a a: reused |a a a a|, new |a a| a a|.
• An AI finder then wrote the public-API JUnit; independent AI tools reproduced, reviewed, judged and audited it (see the checks).