The UnifiedHighlighter's summary of a multi-valued field returns its hidden U+0000 separator, which can push out a sentence

When a hit has no match in a highlighted field, the UnifiedHighlighter returns a default summary, documented as "the first maxPassages sentences from the field". For a multi-valued stored field it joins the values with its internal separator U+0000 and reports a boundary on both sides of each separator. The summary loop turns every pair of neighbouring boundaries into a passage, so the 1-character separator becomes a passage of its own: it takes one of the requested slots and is copied into the snippet as a raw NUL. Two values First value. and Second value. with 2 passages give First value.\u0000; with an empty first value even the default 1-passage call gives \u0000. Hits, scores and the index are unaffected. The code dates from the UnifiedHighlighter's introduction (6.3.0, 2016): the test fails on the 9 releases tried from 6.3.0 to 10.5.2 and on main, and the code is the same at every other tagged release. Skipping that 1-character interval in the summary loop fixes it (7 added lines, one of them an import; validated with the whole highlighter module).

1. Values are joined with U+0000. UnifiedHighlighter.MULTIVAL_SEP_CHAR = (char) 0 (UnifiedHighlighter.java:98), appended between the stored values (:1463). The example content is First value. + U+0000 + Second value.: 26 characters, the separator at index 12.
2. The field's iterator is wrapped. Every field's BreakIterator is wrapped in new SplittingBreakIterator(getBreakIterator(field), MULTIVAL_SEP_CHAR) (UnifiedHighlighter.java:1135), which "virtually slices the text on both sides of every occurrence" of the separator, so that "the enclosed BreakIterator will never 'see' the splitting character" (SplittingBreakIterator.java:24-28). Its next() returns the end of value 1 and then, one character later, the start of value 2 (:104-124, :114). Boundaries: 0, 12, 13, 26. This is intended and tested (TestSplittingBreakIterator: aa|aa gives ^ ^^ ^), and the highlight path needs the boundary at 13 so that a passage in value 2 starts at its first character.
3. No match, so a summary. When no passage has a match, FieldHighlighter calls getSummaryPassagesNoHighlight(maxNoHighlightPassages == -1 ? maxPassages : maxNoHighlightPassages) (FieldHighlighter.java:89-94); -1 is the default.
4. The loop counts the gap. "By default this just returns the first maxPassages sentences" (FieldHighlighter.java:104-108), but the loop (:115-125) makes a passage of every pair of consecutive boundaries: [0, 12) First value., then [12, 13), which holds only U+0000, then [13, 26) Second value..
5. The formatter copies it. DefaultPassageFormatter adds its ellipsis "... " only between passages that do not touch (DefaultPassageFormatter.java:66-68); these touch, so the NUL is copied verbatim and no ellipsis appears.

The same with each of the three offset sources (re-analysis, offsets in postings, term vectors), each executed as a test of its own, and with the whole-text iterator; the hunt's exhaustive harness saw it with every iterator family it covered (sentence, word, custom separator, whole, length goal). When every value is empty the summary is \u0000 alone instead of no passage. All of this assumes stored values without a U+0000 of their own (a stored U+0000 is a separate bug, LFV-599). Every upstream no-match summary test is single-valued, which is why the tests miss it.

• Who is hit. Direct users of Lucene's UnifiedHighlighter on multi-valued stored fields, for hits with no match in the highlighted field (common when several fields are highlighted and the query matches only some), with the default no-match summary, and either 2 or more passages (maxPassages, or withMaxNoHighlightPassages(n) with n >= 2) while the first value has fewer sentences than asked, or an empty first value. The default 1-passage call with a non-empty first value is not affected (for stored values without a U+0000 of their own; that case is LFV-599).
• Reach limit (read). The separator is only added while the field's maxLength budget lasts (default 10,000 characters): the stored-field visitor truncates the first value to maxLength and adds no further value once the budget is used up (UnifiedHighlighter.java:1446-1452). A custom maxLength that ends inside the first value prevents this summary shape.
• Defaults. Default builder, default sentence iterator, default maxNoHighlightPassages (-1), all three offset sources (each executed separately). UnifiedHighlighter is @lucene.experimental; FieldHighlighter is @lucene.internal, but users never pick it: the UnifiedHighlighter builds it.
• What happens. Display text only: a raw U+0000 in the snippet, which takes one of the requested passage slots and so pushes out a real sentence or value whenever the field has one left (2 values and 2 passages: the second value is lost; 3 values and 3 passages: the third). With nothing left to push out, the NUL is still there (2 one-sentence values and 3 passages: First value.\u0000Second value.). Hits, scores and the index are unaffected; nothing throws. The NUL is invisible on screen. An application that writes the snippet into XML without escaping control characters produces a document an XML parser rejects (run only with the JDK's own XML writer and parser as a stand-in; control characters from document text do the same; no real product was run).
• Scale. Deterministic per document and field, independent of index size.
• Downstream (by reading their source as examined during the hunt, not run; not a guarantee for their current code). Elasticsearch and OpenSearch override getSummaryPassagesNoHighlight and skip the separator, so in the examined source they are not exposed. Solr's default summary is off unless hl.defaultSummary=true; then Solr's own snippet separator is also U+0000 and it splits on it, which hides the NUL but returns one snippet fewer than hl.snippets (emulated).
• Workarounds (not run). withMaxNoHighlightPassages(0) turns the summary off; or override newFieldHighlighter to return a FieldHighlighter whose summary skips the separator. Stripping U+0000 from the snippet afterwards still leaves a sentence missing.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: 2-passage summary contains the U+0000 value separator: First value.\u0000
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertFalse(Assert.java:65)
       org.apache.lucene.search.uhighlight.TestUHSummaryMultiValueSeparator.testNoMatchSummaryOfMultiValuedFieldHasNoSeparator(TestUHSummaryMultiValueSeparator.java:53)

• The public contract. highlight(field, query, topDocs, maxPassages): "If no highlights were found for a document, the first maxPassages sentences from the field will be returned" (UnifiedHighlighter.java:759-761); the 3-argument form: "the first sentence for the field" (:742). The separator is inserted by the highlighter (:1463), and SplittingBreakIterator promises that the user's iterator "will never 'see' the splitting character" (SplittingBreakIterator.java:28).
• The highlight path. For the same document with a matching query, values come out as separate passages joined by the ellipsis: TestUnifiedHighlighter.java:339 asserts highlightedValue + "... " + highlightedValue. LUCENE-7438 (#8490), which introduced the highlighter: "a passage will never span across values".
• Lucene's own tests treat the separator as non-content. TestUnifiedHighlighterMTQ.java:1139 skips random values that contain it ("will throw things off"). Every no-match summary test is single-valued. No javadoc, CHANGES or MIGRATE entry sanctions a separator in the summary or a dropped value.
• Downstream fixed it for themselves. Elasticsearch's override of the summary method (2017) skips leading separators and ends the passage at the next one (code reading).
• Origin, not intent. The loop is a near-verbatim copy of PostingsHighlighter.getEmptyHighlight, whose multi-value separator was a visible, configurable space (getMultiValuedSeparator returned ' ') that its iterator saw. LUCENE-7438 replaced it with a hidden control character and did not adapt the loop.

Strongest case for "intended" (built by the cross-model reviews, the independent reproducer, the devil's advocate and the adjudication): getMaxNoHighlightPassages says "the number of leading passages (as delineated by the BreakIterator)", and the iterator FieldHighlighter holds is the SplittingBreakIterator, which deliberately reports both sides of each separator (javadoc and TestSplittingBreakIterator), so [12, 13) is literally a delineated passage; the loop came from the PostingsHighlighter, where summaries contained the visible separator; FieldHighlighter is @lucene.internal, the UnifiedHighlighter and SplittingBreakIterator are @lucene.experimental, and the summary method is a protected override hook; Solr's U+0000 snippet separator hides the NUL.

Why it fails: the iterator's boundaries are fine, the summary loop's use of them is the defect, and the more specific public promise is sentences "from the field"; the literal reading also does not explain why a real sentence is pushed out. The PostingsHighlighter lineage explains where the bug came from, not that anyone wanted it. The labels allow API change; they do not make the output correct. Solr still loses a snippet (emulated), and Elasticsearch replaced the loop. The cross-model reviews rated the intended case moderate; the devil's advocate and the adjudication rejected it. Verdict: a defect; expect the literal reading to come up, and answer it with the highlight() javadoc and the dropped value. In the end it is a contract call for the maintainers: the boundaries are documented and tested, but nothing documents the separator as a summary passage.

Two options were applied and tested with gradle on main 645d4fe14a20 (option B recommended, option A the hunt's original); two more were considered and not run. A and B skip exactly the same interval; they differ only in how they read the character.

In FieldHighlighter.getSummaryPassagesNoHighlight (FieldHighlighter.java:109-128), take a copy of the iterator's text once before the loop and skip a boundary pair of length 1 whose character is MULTIVAL_SEP_CHAR. +7 lines, one of them the import of java.text.CharacterIterator.

What it changes. Only the no-match summary. The separator interval no longer counts toward maxPassages, so the next value's first sentence takes the slot; the two values' passages no longer touch (12 != 13), so the formatter puts its ellipsis between them, as on the highlight path. An empty first value gives the second value's first sentence; when every value is empty there is no passage and the snippet is null, as with maxNoHighlightPassages 0 (FieldHighlighter.java:96-100, read, not run). Single-valued fields (whose value holds no U+0000) and matched highlighting are unchanged.

     List<Passage> passages = new ArrayList<>(Math.min(maxPassages, 10));
+    // a copy, so that reading characters cannot move the break iterator
+    CharacterIterator text = (CharacterIterator) breakIterator.getText().clone();
     int pos = breakIterator.current();
     ...
       if (next == BreakIterator.DONE) {
         break;
       }
+      if (next - pos == 1 && text.setIndex(pos) == UnifiedHighlighter.MULTIVAL_SEP_CHAR) {
+        pos = next; // the separator between two values is not a passage
+        continue;
+      }
       Passage passage = new Passage();

Trade-offs

• One getText() and one clone() per summary call (two small StringCharacterIterator objects with the default iterator).
• It recognises the separator by its value, so a stored value that itself contains U+0000 is still mis-split; that is a separate known bug (LFV-599).
• No CHANGES entry is drafted. Engines that override the summary method are unaffected either way.

What the checks validated

• Gradle in a scratch worktree of main 645d4fe14a20: the 5-test repro passes with assertions on and off and with 3 seeds (stock: 3 of 5 fail); every uhighlight class, 15 classes and 476 tests (470 upstream), 0 failures; all of :lucene:highlighter:test, 36 classes and 728 tests (722 upstream), 0 failures.
• :lucene:highlighter:check -x test passes with the repro added (google-java-format, ecjLint, forbiddenApis).
• With the fixed class shadowed in the repro harness on 645d4fe14a20: the hunt's test passes 2 of 2 and the independent reproducer's test 4 of 4 (3 offset sources); stock they fail 2 of 2 and 3 of 4.

Five added lines in the same loop: if (next - pos == 1 && breakIterator.getText().setIndex(pos) == UnifiedHighlighter.MULTIVAL_SEP_CHAR) { pos = next; continue; }. With the highlighter's own wrapping it is correct, because SplittingBreakIterator.getText() returns a new StringCharacterIterator each time (SplittingBreakIterator.java:59-64).

+      if (next - pos == 1
+          && breakIterator.getText().setIndex(pos) == UnifiedHighlighter.MULTIVAL_SEP_CHAR) {
+        pos = next; // the separator between two values is not a passage
+        continue;
+      }

Trade-offs

• Breaks custom iterators (executed). A FieldHighlighter can get any BreakIterator through the protected newFieldHighlighter hook. A JDK RuleBasedBreakIterator (the sentence and word instances) returns its live CharacterIterator from getText(), so setIndex(pos) moves the iterator back and the next next() repeats a boundary. With a raw JDK word iterator over the same two values and 8 passages, A gives [0,5)[5,6)[6,6)[6,11)[11,12)[12,12)[13,13)[13,19), three of them empty; stock gives [0,5)[5,6)[6,11)[11,12)[12,13)[13,19)[19,20)[20,25) and B [0,5)[5,6)[6,11)[11,12)[13,19)[19,20)[20,25)[25,26).
• That is a custom, @lucene.internal configuration, but option B avoids it at no cost.

What the checks validated

• The same results as option B for the repro (assertions on and off, 3 seeds), the 476 uhighlight tests and the 728 module tests (722 upstream), and check -x test.

Remove the second boundary, so the separator is never an interval of its own.

Trade-offs

Not recommended: the two-sided boundaries are documented and tested (TestSplittingBreakIterator), and the highlight path relies on the boundary at a value's start (breakIterator.preceding(...), FieldHighlighter.java:178-182) so that a passage in a later value does not begin with the separator.

Compute the separator positions from the content, or test breakIterator instanceof SplittingBreakIterator.

Trade-offs

The protected getSummaryPassagesNoHighlight(int) has no content parameter, and changing its signature would break the overrides downstream engines ship; B gets the same result without that.

Method (semi-formal): bounded-exhaustive translation validation of the real highlighter jar against a reference semantics written from the javadoc, not from the code.

• The reference says: when no passage has a match, the summary is the first N passages of the values in order, as delineated by the user's BreakIterator, and never the separator (N = maxPassages when maxNoHighlightPassages < 0).
• Multi-valued tiers: every content of 2-3 values over a, space and . up to 5 characters (8,657 contents, 7 plain and 54 length-goal iterator configurations, 3,168,462 checks), and with a supplementary character up to 4 characters (2,096 contents, 31 configurations, 584,784 checks).
• The failure class P2-summary-separator-passage appeared for every iterator family (sentence, word, custom separator, whole, length goal), on multi-valued content only; the same class also caught the all-values-empty case. The hypothesis came first from reading getSummaryPassagesNoHighlight while the reference was written; the harness confirmed it on its first multi-valued scope. The finder then wrote the public-API JUnit.