UnifiedHighlighter's maxLength cut can split an emoji, so the snippet ends in half a character

The UnifiedHighlighter highlights only the first maxLength characters of a stored value (10,000 by default). LimitedStoredFieldVisitor makes that cut in UTF-16 units and never checks whether it falls between the two halves of a supplementary character, such as an emoji, a CJK Extension B ideograph or a historic-script letter. When it does, the text to highlight ends in a lone high surrogate, and each executed offset source (offsets in postings, term vectors, re-analysis) returns it at the end of the snippet once the chosen passage reaches the cut: Hello 😀 world at maxLength 7 gives <b>Hello</b> \uD83D, while 6 and 8 give whole characters. getBytes(UTF_8) turns the half into ?, and a strict UTF-8 encoder throws. Hits and scores are right; on the re-analysis path with StandardAnalyzer a short value also loses its highlight, through a separate tokenizer bug. Present since 6.3.0 (2016); moving the cut back by one unit fixes it, and the whole highlighter test task (728 tests) passes with the fix.

• maxLength: "The maximum content size to process. Content will be truncated to this size before highlighting" (UnifiedHighlighter.java:596-603); DEFAULT_MAX_LENGTH = 10000 (:100); Builder.withMaxLength: "The text to be highlight is effectively truncated by this length" (:354). No unit is named; the cut counts Java chars.
• LimitedStoredFieldVisitor: the stored-field visitor that loads the text to highlight. loadFieldValues creates it through newLimitedStoredFieldsVisitor with getMaxLength() (:1387, :1409-1411) and calls init() per document (:1395). Values of a multi-valued field are joined with the separator U+0000.
• Offset source: POSTINGS when offsets are indexed (POSTINGS_WITH_TERM_VECTORS when term vectors are stored too), TERM_VECTORS when only term vectors are stored, else ANALYSIS, which re-analyzes the stored text (getOffsetSource, :696-709). For a multi-term or rewriting query a postings field is re-analyzed too (getOptimizedOffsetSource, :1268-1273). POSTINGS, TERM_VECTORS and ANALYSIS were executed; all of them highlight the same loaded content (:958).

1. First value: values[currentField] = value.substring(0, Math.min(maxLength, value.length())) (:1446-1447).
2. Later value: lengthBudget = maxLength - curValue.length() (:1450), then the separator and append(value, 0, Math.min(lengthBudget - 1, value.length())) (:1463-1464); needsField stops once curVal.length() >= maxLength (:1475).
3. Neither cut looks at Character.isHighSurrogate; the word "surrogate" does not occur in the highlighter's main sources. When unit end - 1 is a high surrogate and unit end its low half, the content ends in the lone high half.
4. Everything after the cut works on whole tokens and break-iterator boundaries: FieldHighlighter drops a token that straddles the cut (FieldHighlighter.java:164-166) and clamps passage ends to the content length (:186). A passage that ends at the content end copies the half character out (DefaultPassageFormatter.append, "Appends original text to the response", DefaultPassageFormatter.java:101). This is about a well-formed stored value and the stock passage machinery: append copies chars as they are (:108-138), so it neither makes nor repairs a split.

Main 645d4fe14a20, assertions on and off; the same on the 10.5.2 and 9.12.3 jars. With StandardAnalyzer and offsets in postings the snippet is <b>Hello</b> \uD83D; getBytes(UTF_8) gives <b>Hello</b> ?, and a strict UTF-8 encoder throws MalformedInputException: Input length = 1.

• Who is hit. Any UnifiedHighlighter use, defaults included: default maxLength 10,000, offsets in postings, term vectors or re-analysis (all three executed), any analyzer (the cut is made on the stored text, not on tokens), default writer and codec. One document is enough; no scale is needed.
• Which text. A stored value longer than the budget that contains supplementary characters: emoji in user text, CJK Extension B and later ideographs, historic scripts, mathematical alphanumerics. The first half of such a character must be exactly the last kept unit (index maxLength - 1 of the first value, or the budget edge of a later value), and the chosen passage must reach the cut. Per truncated value the chance is roughly the share of units that start a supplementary character, so it is rare.
• Effect. The snippet ends in one unpaired high surrogate. In the executed probe getBytes(UTF_8) replaced it with ? and a strict UTF-8 encoder threw MalformedInputException; other encoders may replace it differently (not run). The truncation itself does not change hits or scores. With offsets in postings or term vectors (and a term query) the matched word stays highlighted; on re-analysis with a JFlex analyzer a short truncated value can also lose its highlight, through a separate bug (next point).
• Secondary symptom, through another bug. On the re-analysis path, StandardAnalyzer and the other JFlex tokenizers drop the tokens of a text that ends in a lone high surrogate (a separately registered bug), so a short truncated value comes back with no highlight: Hello \uD83D instead of <b>Hello</b> . In the hunt's runs the long re-analyzed value at the default maxLength kept its highlight. Either fix removes the symptom here (both executed). With the test framework's MockTokenizer the same path throws IllegalStateException (test-only).
• Downstream (read, not executed). Solr's unified highlighting passes hl.maxAnalyzedChars (default 51,200) to setMaxLength, so it reaches the cut; Elasticsearch's unified highlighter loads values itself and calls FieldHighlighter directly, so it is likely not exposed; OpenSearch was not checked.
• Why nobody noticed. One invisible or replaced character at the very end of a snippet, only when the cut lands inside a pair; the predecessor PostingsHighlighter had the same unchecked cut from 4.1.0 (2012) with no report.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected a well-formed snippet, but was: <b>Hello</b> \uD83D
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.search.uhighlight.TestUnifiedHighlighterMaxLengthSurrogatePair.testFirstValueCutInsideSurrogatePair(TestUnifiedHighlighterMaxLengthSurrogatePair.java:59)

• The budget is an upper bound, and the fix keeps it. "The maximum content size to process" (UnifiedHighlighter.java:597) is met by a cut one unit earlier, which is what maxLength 6 already produces. The fix keeps the UTF-16 unit of maxLength, and every test of the executed highlighter test task still passes (728 tests, assertions on and off). No javadoc promises well-formed snippets; the argument rests on the bound, the formatter's wording and the precedents.
• The formatter's contract is original text. DefaultPassageFormatter.append: "Appends original text to the response" (DefaultPassageFormatter.java:101). The stored value was well-formed; half a code point is not text in any encoding.
• Precedents, fixed as bugs. CHANGES.txt:1215-1217, 10.5.0 Bug Fixes: "GITHUB#15899, GITHUB#15900: Improve TruncateTokenFilter to truncate on codepoints not chars and no longer produce half surrogates." In this module, CHANGES.txt:13362-13364, 4.5.0 Bug Fixes: "LUCENE-5191: SimpleHTMLEncoder in Highlighter module broke Unicode outside BMP because it encoded UTF-16 chars instead of codepoints." Also LUCENE-8325 (smartcn) and LUCENE-3905 (test-framework truncation).
• No highlighter test accepts a split pair. A source scan finds no highlighter test that combines supplementary characters with maxLength. The pinned truncation tests (TestUnifiedHighlighter.java:194-223, LUCENE-5166) cut mid-word, which is still well-formed text, and they pass with the fix. Of the threads read, none accepts a cut inside a character for the highlighter.

The strongest case for "intended": maxLength is documented as a plain size cut in chars, with no unit and no promise of code-point-safe output; upstream tests pin a cut inside a word; in LUCENE-5166 @mikemccand accepted truncated passages ("I'm not even sure it needs fixing. It ought to be rare-ish that a truncated passage is selected."); TruncateTokenFilter still ships a char mode documented with "This may split surrogate pairs" (TruncateTokenFilter.java:53-54, :61-62); the visitor is @lucene.internal and its factory method can be overridden. In #15900 @uschindler first called a half surrogate after a UTF-16 cut "expected", like String#substring. Later in that thread @uschindler agreed that "The old variant produces incorrect tokens", and added: "But when the search side also produces the same incorrect tokens it's not a big issue. Now we produce different tokens, so you won't find the broken tokens anymore in an existing index."

Why it fails: the accepted LUCENE-5166 behaviour is a cut mid-word, not mid-character, and nobody there discussed code points. TruncateTokenFilter's char mode remains as documented legacy behaviour, kept for backwards compatibility (TruncateTokenFilter.java:34-37, :61-67); the reason given in #15900 was existing indexes (Uwe Schindler's comment above), and the factory javadoc warns that moving to code points may require reindexing (TruncateTokenFilterFactory.java:32-37). The factory's default switched to code points from luceneMatchVersion 10.5.0 (:93-97), under Bug Fixes. The highlighter's snippet is computed at query time and not indexed, so that reason does not apply, and the highlighter documents no such caveat. An overridable internal hook does not make the default behaviour intended. The fix changes only the one case where the output is not text. The independent reproducer, the intended-lens cross-model review (it rated the intended argument "moderate"), the devil's advocate, the adjudication and the audit all concluded it is not intended.

Two fixes were built and validated through gradle at 645d4fe14a20: the recommended one (the hunt's fix) and a smaller variant that a probe showed to be weaker. The other options are listed for the discussion with maintainers.

A helper used at both cut sites returns end - 1 when unit end - 1 is a high and unit end a low surrogate, and records in a per-field boolean[] (allocated in init()) that the field's budget is spent, also for an ordinary cut; needsField then stops loading further values of that field. The budget stays in UTF-16 units.

// both cut sites in stringField: end = endBeforeSplitPair(value, end); needsField stops once truncated
private int endBeforeSplitPair(String value, int end) {
  if (end > 0
      && end < value.length()
      && Character.isHighSurrogate(value.charAt(end - 1))
      && Character.isLowSurrogate(value.charAt(end))) {
    truncated[currentField] = true; // the budget is spent even though one unit is left
    return end - 1;
  }
  if (end < value.length()) {
    truncated[currentField] = true;
  }
  return end;
}

Trade-offs

• One small array per document. The helper both computes the cut and records state, which a reviewer may want split.
• A moved-back cut leaves the content one unit short of maxLength; the flag is what keeps the visitor from appending a separator and an empty later value (see the smaller variant).
• A PR could add the paste-ready test (or a shorter one) to TestUnifiedHighlighter.

What the checks validated

• Scratch worktree at 645d4fe14a20, gradle: the paste-ready test fails 5 of 6 on stock main, assertions on and off, and 50 of 60 with -Ptests.iters=10; with the patch the whole :lucene:highlighter:test task passes, 35 classes, 728 tests, 0 failures, assertions on and off (counting the 6 paste-ready tests and the upstream truncation tests), and 60 of 60 with -Ptests.iters=10.
• ecjLintMain, ecjLintTest, checkGoogleJavaFormat, forbiddenApisMain and forbiddenApisTest for the module pass.
• Probes with the patch: a two-valued field cut inside the emoji gives <b>hello</b> and the no-match summary hello for 1 and 2 passages; with StandardAnalyzer on re-analysis the highlight comes back (<b>Hello</b> ).
• Earlier, on a9779ef3c193: the finder's, the independent reproducer's and the devil's advocate's tests pass with it, the upstream mid-word cut control included, and the adjudication's re-run with a fresh seed (0 of 6 fail).

A static cutPoint(value, end) with the same pair check at both sites, and no new state.

Trade-offs

• The whole highlighter test task passes with it (728 tests, 0 failures) and all 6 paste-ready tests pass.
• After a moved-back cut the content is maxLength - 1 long, so the next value is still accepted and the U+0000 separator plus an empty value is appended (hello \u0000). Snippets stay right, but a two-passage no-match summary of that field returns hello \u0000, the separator NUL of a separately registered bug, instead of hello . Stock does the same when a value ends exactly one unit before the budget, so this adds one more way into that bug.

What the checks validated

• Gradle at 645d4fe14a20: the whole highlighter test task with the paste-ready test, assertions on: 728 tests, 0 failures; the two probes above (runner); the encode probe, run by the fact check, gives the same clean <b>Hello</b> as the recommended fix for offsets in postings and for re-analysis with StandardAnalyzer.

Measure the budget in code points instead of chars.

Trade-offs

Would change the current UTF-16-unit behaviour, and with it the content size and output length for every text with supplementary characters; the javadoc does not define the unit, but existing snippet lengths would change. Nothing needs it. Not run.

Override newLimitedStoredFieldsVisitor (protected, @lucene.internal, UnifiedHighlighter.java:1409-1411) with a visitor that backs the cut off, or strip a trailing lone high surrogate from returned snippets.

Trade-offs

Every application has to know about it. Not run.

• Harness. A translation-validation harness over the stock jars: 2,238 single-valued texts (all 1-3-token sequences over a, b, ab, Ab, B1, 1 with space or . separators, and all 4-token sequences) and 1,764 two-valued documents; StandardAnalyzer and KeywordAnalyzer; fields with offsets in postings, term vectors and none (re-analysis); 49 and 35 queries of 11 kinds (term, boolean, phrase, slop, prefix, wildcard, regexp, fuzzy, range, span, intervals); maxLength 10000 and 1-30; three mappings of letters and digits (Deseret and Adlam, Armenian and Adlam, and an all-BMP control, Georgian and Saurashtra).
• Relations. "No snippet of a mapped text contains an unpaired surrogate" failed 2,202,831 times in 57,460,950 checks, always with the cut between the two halves of a character; the all-BMP control failed 0 times. Hits and scores never differed. With the cut on a code-point boundary every mapped snippet was the image of the original one, apart from 1,622 differences explained by two UTF-16-length heuristics that are not part of this report (the passage centre for phrase matches across a boundary, and PassageScorer's length weights for fuzzy matches).
• Bridge. Reading LimitedStoredFieldVisitor while modelling the cut located the two lines; a public-API JUnit on the stock jars failed 5 of 5 on main, 10.5.2 and 9.12.3 and passed with the fix.