matchhighlight's passage window can cut a two-unit character in half, so the snippet ends in half of it

matchhighlight's PassageSelector measures its passage window in UTF-16 units. The passage of a value without matches (what FieldValueHighlighters.maxLeadingCharacters shows) ends at from + window, and a match longer than the window (in highlighted) is cut at from + window. The default BreakIteratorShrinkingAdjuster then moves the end back to a word boundary, but when the first word is longer than the window there is none, so it keeps the raw end, and it never moves the end of a cut match. Nothing checks for surrogate pairs. When the raw cut falls between the two halves of a supplementary character, the snippet ends in a lone high surrogate: 𝐁𝐎𝐋𝐃 text (mathematical bold capitals, two units each) with window 5 gives 𝐁𝐎\uD835... and >𝐁𝐎\uD835<..., while the ASCII value boldtext text gives boldt... and >boldt<....

getBytes(UTF_8) turns the half into ?, and a strict UTF-8 encoder throws. Hits, scores and the other highlight positions are right. The reach through the default adjuster is narrow: a first word of supplementary letters longer than the window, or one match longer than the window, with the cut mid-pair; a single emoji between BMP text gets its own word boundaries, so the default passage does not cut it (inside a match longer than the window it can be cut). Present since 9.0.0 and never fixed. Moving such an edge to the character boundary fixes it, and the whole highlighter test task passes with that fix.

• maxLeadingCharacters(n, ellipsis, fields): "Displays up to maxLeadingCharacters of the field's value, regardless of whether it contained highlights or not" (FieldValueHighlighters.java:52-55). It calls defaultPassageSelector().pickBest(value, [], n, 1, valueRanges) (:68-70).
• highlighted(maxPassageWindow, maxPassages, formatter, fields): passages around matches (:88-115), the same selector, with the query's match offsets (:108-110).
• defaultPassageSelector(): new PassageSelector(PassageSelector.DEFAULT_SCORER, new BreakIteratorShrinkingAdjuster()) (:82-86); the adjuster uses BreakIterator.getWordInstance(Locale.ROOT) (BreakIteratorShrinkingAdjuster.java:30-32).
• For a single-valued field MatchHighlighter passes the value and one range [0, length) (MatchHighlighter.java:260-266, with contiguousFieldValue and computeValueRanges at :276-299), which is what the 4-argument pickBest does (PassageSelector.java:67-74).

1. Path 1, no matches. No marker enters the queue, so pickDefaultPassage runs (PassageSelector.java:170-175) and returns new Passage(o.from, o.from + Math.min(maxCharacterWindow, o.length()), ...) (:293-299): [0, 5).
2. Path 2, a match longer than the window. splitOrTruncateToWindows cuts the marker: if (to - from > maxPassageWindow) { to = from + maxPassageWindow; ... } (:251-254), so [0, 8) becomes [0, 5); the centred window around it (:128-141) is [0, 5) too.
3. The adjuster keeps the raw end. For the end it loops to = bi.preceding(to) while to is not a word boundary (BreakIteratorShrinkingAdjuster.java:63-69); when that reaches 0 it restores the raw end, if (to == 0) { to = passage.to; } (:70-72). Inside a word of supplementary letters isBoundary is false and preceding() is 0 at every offset (an independent probe on a Deseret word). It then widens the passage to its markers' ends (:75-78), so a cut marker's end stays as well.
4. The formatter copies the range. PassageFormatter.format appends value[from, to) (PassageFormatter.java:110-125) and the ellipsis when the passage ends before the value (:87-89): 𝐁𝐎\uD835..., or >𝐁𝐎\uD835<... with markers.
5. No check. A case-insensitive grep of the package sources for surrogate, codepoint and code point finds nothing, and the package history never had such a check.

Which windows split. In a run of two-unit characters starting at unit 0 every character starts at an even unit, so an odd window ends inside one and an even window does not (sweep over Deseret, mathematical bold and CJK Extension B words: every odd window shorter than the word splits).

Centred windows. The centred-window arithmetic (PassageSelector.java:128-141) can also put an edge inside a pair, and the adjuster restores a raw start edge too (BreakIteratorShrinkingAdjuster.java:56-58). With the default adjuster the sweep found no split from those lines; a PassageSelector without an adjuster (new PassageSelector(), as TestPassageSelector uses) keeps any raw edge.

• Who is hit. MatchHighlighter with FieldValueHighlighters.maxLeadingCharacters or highlighted, which use the default PassageSelector and BreakIteratorShrinkingAdjuster, or PassageSelector used directly. matchhighlight is an opt-in alternative to the UnifiedHighlighter; within it, the trigger is the ready-made default configuration, with the default codec, writer config and searcher. A code search found matchhighlight in none of Solr, Elasticsearch and OpenSearch; maintainers want to keep the package (#16820).
• Which text. Path 1: the value starts with a word, as the JDK word iterator sees it, longer than the window and made of supplementary letters: mathematical "styled" letters as pasted into posts, Deseret, Adlam, Osage, or consecutive CJK Extension B ideographs, possibly mixed with BMP letters. Path 2: one match longer than the window, which also covers CJK text and emoji (a long match over Han text with U+20000 splits; a phrase match over Hello 😀 world with window 7 gives >Hello \uD83D<...). The cut must land mid-pair: for a run of such characters that starts at an even unit, as a leading word does, any odd window; for a match it depends on where the match starts.
• Not hit by the default passage (path 1). A single emoji or a single supplementary ideograph between BMP text: the word iterator puts boundaries around each one, so the adjuster moves the end off it. Inside a match longer than the window (path 2) it can still be cut.
• Window sizes. Small windows hit it easily: the upstream test uses maxLeadingCharacters(10, ...), and a value that starts with the one word Hello + 𝐁𝐎𝐋𝐃 + world splits at window 10. Windows of 100 or more units need very long such words or matches.
• Effect. One malformed character at the end of the snippet or of the cut match. getBytes(UTF_8) gives ? and a strict UTF-8 encoder throws MalformedInputException (executed); other encoders may write U+FFFD or something else (not run). Hits, scores and the other highlight positions are right.
• Scale and likelihood. One document, deterministic, assertions on and off. An accidental hit needs the little-used package, such a word or match and an unlucky window; no matching public report was found, and it is graded rare.

The test sits in org.apache.lucene.search.matchhighlight by upstream convention. It calls exactly what maxLeadingCharacters(5, ...) calls for a single-valued field and asserts only the claimed property, so a fix that moves the end back, or widens it to the end of the character, passes (a fix that drops the passage would make get(0) throw). tidy leaves it unchanged. It is also the suggested regression test.

Not asserted here (shown by the checks and the probe): the end-to-end snippets through MatchHighlighter (𝐁𝐎\uD835... and, for a long match with highlighted(5, ...), >𝐁𝐎\uD835<...), the ASCII control (boldt...), and what UTF-8 encoders do with the half character.

# stock main 645d4fe14a20, gradle, seed DEADBEEF (identical with -Ptests.asserts=false); abridged
TestMatchHighlightWindowSurrogatePair > testLeadingPassageDoesNotSplitSurrogatePair FAILED
    java.lang.AssertionError: expected a passage ending on a code point boundary, actual: [[from=0, to=5], markers=[]]
        at org.apache.lucene.search.matchhighlight.TestMatchHighlightWindowSurrogatePair.testLeadingPassageDoesNotSplitSurrogatePair(TestMatchHighlightWindowSurrogatePair.java:29)
BUILD FAILED

# 10.6.0 release jars, standalone, -ea and -da: the same AssertionError

# end to end (the hunt's test, MatchHighlighter + StandardAnalyzer; all 33 releases 9.0.0-10.5.2 and main)
testMaxLeadingCharacters: snippet has an unpaired surrogate at index 4: '\uD835\uDC01\uD835\uDC0E\uD835...' expected:<-1> but was:<4>
testHighlightedMatchLongerThanWindow: snippet has an unpaired surrogate at index 5: '>\uD835\uDC01\uD835\uDC0E\uD835<...' expected:<-1> but was:<5>

# with the fix (asserts true and false): passage [0, 4); 1 test, 0 failures

• The budget is "up to". maxLeadingCharacters "Displays up to maxLeadingCharacters of the field's value" (FieldValueHighlighters.java:52-55). 𝐁𝐎... (4 units) meets a 5-unit budget, which is what the fix returns; for any window of 2 or more units such an answer exists. No javadoc defines "characters" or promises whole code points, so the expectation is implied.
• The adjuster's job. "A PassageAdjuster that adjusts the Passage range to word boundaries hinted by the given BreakIterator" (BreakIteratorShrinkingAdjuster.java:22-25, {@link} markup removed). Its raw-end fallback (:70-72) handles a first word longer than the window; on ASCII it gives a cut between characters, which is all the tests pin.
• Tests. TestPassageSelector.noHighlights expects 01234... for window 5 (:82-84) and highlightLargerThanWindow expects >0123<... for an 11-char marker and window 4 (:103-105); none has a supplementary character at the cut. randomizedSanityCheck uses random Unicode but checks only that nothing throws, that no passage is empty and that no empty marker >< appears (:338-359); nothing about surrogate pairs. With the fix every matchhighlight test and the whole highlighter test task pass.
• Ready to use. MatchHighlighter is "An example highlighter that combines several lower-level utility classes in this package into a fully featured, ready-to-use component" (MatchHighlighter.java:38-39).
• Precedents, fixed as bugs. CHANGES.txt:1215-1217 (10.5.0 Bug Fixes): "GITHUB#15899, GITHUB#15900: Improve TruncateTokenFilter to truncate on codepoints not chars and no longer produce half surrogates." CHANGES.txt:13362-13365 (4.5.0 Bug Fixes; the older Highlighter in the same highlighter module, not matchhighlight): "LUCENE-5191: SimpleHTMLEncoder in Highlighter module broke Unicode outside BMP because it encoded UTF-16 chars instead of codepoints." CHANGES.txt:7496-7497 (7.4.0 Bug Fixes): "LUCENE-8325: Fixed the smartcn tokenizer to not split UTF-16 surrogate pairs."
• History. Neither the introducing PR (apache/lucene-solr#1750) nor LUCENE-10135 (#11172 / #334) discusses code points.

The strongest case for "intended" (verifiers and a cross-model review): the whole API works in UTF-16 offsets (OffsetRange, CharSequence), and "characters" naturally means Java chars; the upstream tests pin raw cuts; the raw-end fallback is deliberate; String#substring splits pairs too; the pipeline is pluggable (a custom PassageAdjuster, the protected pickDefaultPassage), and the no-argument PassageSelector() has no adjuster at all. On #15900 @uschindler first took exactly this position: "If it truncates tokens after n utf16 characters, so you may get half surrogates. That's expected. This is exactly the same like if you use String#substring(n,m) in your Java code" (2026-04-02).

Why it fails: the report keeps the UTF-16 budget and the mid-word cut and objects only to a cut inside one character; no javadoc, test or CHANGES entry accepts that; the trigger is the default configuration on well-formed text; and a whole-character answer within "up to N" exists for any window of 2 or more units. #15900 was merged under Bug Fixes, and the same day @uschindler wrote "The old variant produces incorrect tokens. That's right."; the concern Uwe Schindler kept was existing indexes ("you won't find the broken tokens anymore in an existing index"). Lucene did keep the char mode there, as an explicit legacy option ("This may split surrogate pairs.", TruncateTokenFilter.java:52-55; kept "for backwards compatibility", :64-66), so the precedent is a code-point-safe mode added as a bug fix, not a ban on char cuts. A snippet is computed at query time and never indexed, so the backwards-compatibility concern does not apply. An overridable hook does not make the default output intended.

One fix was built and validated through gradle at 645d4fe14a20 (the hunt's fix, re-based and formatted). The other options are listed for comparison.

Two small helpers check whether an offset falls between a high and a low surrogate. The default passage end moves back one unit (unless that empties the passage); a too-long match is cut one unit earlier (unless that empties it); the centred window's start moves forward and its end back, never past the marker it was built for. Excerpt:

// pickDefaultPassage; the long-match cut and the centred window snap the same way
int end = o.from + Math.min(maxCharacterWindow, o.length());
int snapped = codePointStartAtOrBefore(value, end);
defaultPassages.add(new Passage(o.from, snapped > o.from ? snapped : end, Collections.emptyList()));

private static int codePointStartAtOrBefore(CharSequence value, int i) {
  if (i > 0 && i < value.length()
      && Character.isHighSurrogate(value.charAt(i - 1))
      && Character.isLowSurrogate(value.charAt(i))) {
    return i - 1;
  }
  return i;
}

Trade-offs

• Known gap: a 1-unit window. When the window is smaller than one supplementary character, snapping back would empty the passage or the match, so the fix keeps the raw edge and still splits (a cross-model reviewer's 1-unit-window test fails with it). A complete fix must choose: widen to the whole character (exceeds the cap by one unit), drop the passage, or keep the split.
• The budget stays in UTF-16 units; output changes only where an edge fell inside a pair, where the passage or the cut match loses one unit at that edge. ASCII and the pinned mid-word cuts are unchanged.
• By construction (the selector snaps before any adjuster runs) it also covers a selector without an adjuster, or with a custom adjuster that keeps raw edges; not run separately. It does not stop a custom adjuster from moving an edge into a pair itself, and it does not snap caller-supplied permittedPassageRanges or match offsets that already fall inside a pair (the 5-argument pickBest).
• splitOrTruncateToWindows is private, so giving it the value is no API change; pickDefaultPassage is protected, so a subclass that overrides it does not get the fix.
• The paste-ready test covers path 1 only; path 2 is covered by the finder's end-to-end test (2 of 2 fail on stock, 0 of 2 with the patch).
• No benchmark was run; the extra work is two charAt checks per edge.

What the checks validated

• Scratch worktree at 645d4fe14a20, gradle --no-daemon, seed DEADBEEF unless noted:
  • stock, paste-ready test only: 1 test, 1 failure, assertions on and off;
  • with the patch, the matchhighlight package (TestPassageSelector, TestMatchHighlighter, TestMatchRegionRetriever and the repro): 4 classes, 49 tests, 0 failures, assertions on and off; with -Ptests.iters=5 (random seeds) 245 tests, 0 failures;
  • with the patch, the whole :lucene:highlighter:test task: 35 classes, 723 tests, 0 failures (assertions on);
  • the hunt's end-to-end test (both paths): 2 of 2 fail on stock, 0 of 2 with the patch;
  • ecjLintMain, ecjLintTest, checkGoogleJavaFormat, forbiddenApisMain, forbiddenApisTest: pass.
• Probe with the patch shadowed: 𝐁𝐎... and >𝐁𝐎<... for the bold value, both encode cleanly; the ASCII control stays boldt... and >boldt<....
• Earlier, on a9779ef3c193: the hunt's and an independent reproducer's tests pass with it, and the 48 upstream matchhighlight tests pass (TestPassageSelector with three more seeds).

Make BreakIteratorShrinkingAdjuster move a restored raw edge (:56-58, :70-72) to a character boundary.

Trade-offs

Smaller, but misses the long-match path (the adjuster keeps the passage around its markers, :75-78, and the cut marker itself ends mid-pair) and a selector without an adjuster. Not run.

Measure maxPassageWindow / maxLeadingCharacters in code points instead of UTF-16 units.

Trade-offs

Changes the length of every snippet over supplementary text, against the UTF-16 offsets the whole API uses (OffsetRange, CharSequence); no upstream test covers non-BMP lengths either way, and none needs code points. Not run.

Wrap the FieldValueHighlighter and drop a lone high surrogate before the ellipsis or the end marker of each returned snippet.

Trade-offs

Hides the symptom in one application; shown for completeness. Not run.

• Signal. The hunt unit's table of documented length units lists the matchhighlight window (maxPassageWindow, maxLeadingCharacters: named in characters, counted in UTF-16 units in the code).
• Metamorphic probes. On the stock jars, a substitution that maps ASCII letters to supplementary letters (Deseret, two UTF-16 units each) should only rename text. For maxLeadingCharacters on a value whose first word is longer than the window, the ASCII value was cut inside the word (the accepted behaviour) and its image inside a character; a second probe on Chinese text with a CJK Extension B ideograph showed the same for highlighted() with a match longer than the window.
• Bridge. Reading PassageSelector and BreakIteratorShrinkingAdjuster located the raw cut and the fallback; the finder wrote a public-API JUnit (mathematical bold, both paths) that fails on main, 10.5.2 and 9.12.3 and passes with the fix. It is not a product of the exhaustive harness the unit ran on the UnifiedHighlighter.