UnifiedHighlighter.highlightWithoutSearcher ignores maxLength and highlights past it

UnifiedHighlighter has a maxLength option, 10,000 chars by default, documented as "The maximum content size to process. Content will be truncated to this size before highlighting." With the default stored-field loader, the searcher entry points (highlight, highlightFields) cut the text while they load it. highlightWithoutSearcher, the entry point for a highlighter built with builderWithoutSearcher, takes the text from the caller and never cuts it: the whole text is analyzed, and a match or passage after maxLength can be returned. With withMaxLength(20), the 48-char text Alpha beta gamma. Delta epsilon. Lucene is here. gives <b>Lucene</b> is here. where the searcher path gives Alpha beta gamma. ; with the default, a match at char 10,512 of a 10,527-char text is highlighted. The work grows with the whole text (58 ms at 1 million chars and 217 ms at 4 million in one probe, against about 1 ms when the caller cuts it). Hits, counts and scores are correct, and no recorded run threw an exception. Present since the class was added in 6.3.0 (2016); a 4-line cut, with a javadoc sentence, passes the whole highlighter test suite.

• maxLength: DEFAULT_MAX_LENGTH = 10000 (UnifiedHighlighter.java:100). getMaxLength(): "The maximum content size to process. Content will be truncated to this size before highlighting. Typically snippets closer to the beginning of the document better summarize its content." (:596-603). Builder.withMaxLength: "The text to be highlight is effectively truncated by this length." (:354). builderWithoutSearcher returns that same Builder (:453-455).
• Searcher entry points: the public highlight (:747, :766) and highlightFields (:796, :828, :854) overloads, which all reach the protected highlightFieldsAsObjects (:890). It loads each hit's text with loadFieldValues (:958, :1382-1404); a subclass may override that loader.
• highlightWithoutSearcher(field, query, content, maxPassages) (:1096-1108): "Highlights text passed as a parameter. ... This use-case is more rare. Naturally, the mode of operation will be OffsetSource#ANALYSIS" (:1079-1081), that is, the text is re-analyzed. Its javadoc says nothing about maxLength.

1. On the searcher path, the default loadFieldValues reads stored fields through new LimitedStoredFieldVisitor(fields, MULTIVAL_SEP_CHAR, getMaxLength()) (:1409-1410), which keeps value.substring(0, Math.min(maxLength, value.length())) (:1446-1447) and caps later values of a multi-valued field by the remaining budget (:1450-1465). The cut text goes to fieldHighlighter.highlightFieldForDoc(leafReader, docId, content.toString()) (:987-988).
2. highlightWithoutSearcher checks that the searcher is null and that content is not null, then calls highlightFieldForDoc(null, -1, content) with the caller's full String (:1104-1107).
3. Nothing downstream cuts: FieldHighlighter and the offset strategies never read maxLength; the only getMaxLength() use in the class is the visitor above.
4. With the default flags, the re-analysis is done by MemoryIndexOffsetStrategy (chosen at :1294-1302), whose getOffsetsEnum says "note: don't need LimitTokenOffsetFilter since content is already truncated to maxLength" (MemoryIndexOffsetStrategy.java:104). With phrases not position sensitive and both passage relevancy and weight matches off, TokenStreamOffsetStrategy re-analyzes the full text the same way (the fact check's probe ran both strategies: the same snippet).
5. The break iterator, the scorer and the formatter then see the whole text, so the best passage can lie after maxLength, and the analysis work grows with the whole length.

Main 645d4fe14a20, assertions on and off; the same on the 10.5.2 and 10.0.0 jars.

• Who is hit. Every caller of highlightWithoutSearcher (a highlighter built with builderWithoutSearcher, or the deprecated constructor with a null searcher) whose text is longer than maxLength. With the default that is any text over 10,000 chars, such as a full article; with a small withMaxLength almost any text. Every such call skips the cut; the snippet differs when the chosen passage depends on text past maxLength.
• Default vs opt-in. The entry point is opt-in and rare: its javadoc says "This use-case is more rare", the class is @lucene.experimental, and Lucene itself has no non-test caller. Once used, the default bound is skipped with no option set.
• Effect on the snippet. A match or passage after maxLength can be returned where the searcher path returns the first passage. The snippet is valid text from the caller's own string.
• Effect on the work. One probe (StandardAnalyzer, one TermQuery, default maxLength; time is the median of 5 calls, allocation the bytes the thread allocated in the last of them) measured 21 ms and 63.5 MiB for 100,000 chars, 58 ms and 635 MiB for 1,000,000 chars, and 217 ms and 2,538 MiB for 4,000,000 chars; about 1 ms and 6.4 MiB when the caller cuts the text to 10,000 first. Only these sizes, one analyzer and one query were measured: a sanity check on one machine, not a benchmark; the allocation is not retained memory.
• Not affected. Hits, counts and scores; the searcher entry points with the default loader. No recorded run threw an exception or crashed, with assertions on or off.
• Why nobody noticed. The output is a plausible snippet, and the upstream tests call highlightWithoutSearcher only on short text.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected no match past maxLength 20, got: <b>Lucene</b> is here.
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.junit.Assert.assertFalse(Assert.java:65)
       org.apache.lucene.search.uhighlight.TestHighlightWithoutSearcherMaxLength.testMaxLength(TestHighlightWithoutSearcherMaxLength.java:37)

• The contract is unconditional. getMaxLength (UnifiedHighlighter.java:596-603, wording from the original 2016 commit) and Builder.withMaxLength (:354, added in #412 / LUCENE-10197, 2021) describe the text that is highlighted, with no exception for any entry point. The highlightWithoutSearcher javadoc (:1078-1093) is silent on maxLength.
• The code relies on it on this exact path. MemoryIndexOffsetStrategy.java:104 ("content is already truncated to maxLength") came in with highlightWithoutSearcher in 722e8271243 (then in AnalysisOffsetStrategy; moved by LUCENE-7526).
• The 2021 builder review used this combination. In #412 the withMaxLength wording was suggested in review by @dsmiley, and the PR author's example in the same review was builderWithoutSearcher(indexAnalyzer).withMaxLength(100), with no remark that it has no effect.
• Tests pin maxLength as visible on the searcher path. TestUnifiedHighlighterMTQ.testWithMaxLen (withMaxLength(25): "ask for 2 but we'll only get 1") and TestUnifiedHighlighter.testFormatWithMatchExceedingContentLength (withMaxLength(17)) assert snippets cut by it. The only highlightWithoutSearcher tests (TestUnifiedHighlighterReanalysis, TestLengthGoalBreakIterator, TestUnifiedHighlighterStrictPhrases.testMatchNoDocsQuery) use short text and the default.
• Precedent. maxLength edge cases of the predecessor PostingsHighlighter were fixed as bugs: LUCENE-5166 (a term spanning the boundary) and LUCENE-5032 (multi-valued fields exceeding it). No CHANGES, MIGRATE or package entry exempts this path.

The strongest case for "works as designed" (the intended-behaviour checker, and this hunt's own earlier triage, which had left it as a documentation gap): maxLength is a stored-value loading bound. It is enforced only in LimitedStoredFieldVisitor, whose javadoc says it "honors a max length to retrieve"; a subclass that overrides loadFieldValues also gets uncut text (executed: with a searcher and withMaxLength(20), an override returning the full text gives <b>Lucene</b> is here.); highlightWithoutSearcher loads nothing, its caller owns the String, its javadoc is silent, the class is experimental, and the behaviour has not changed in ten years, so a code change alters what existing callers get.

Why we still read it as a bug: the option's own javadoc describes the highlighted text, not loading; the module's code on this exact path assumes the cut (this shows that the documentation and the implementation disagree, not that the authors considered caller-supplied text and chose to cut it); the default 10,000 is skipped with stock settings, while the override case needs a user to replace the loader on purpose; and we found no doc, test or discussion that records the loading-only reading. Our reading: not intended. The reviewers rated the intended-behaviour case weak to moderate, so a maintainer may fairly prefer to document the gap; the fix section offers both.

Two code fixes were applied in a scratch worktree of 645d4fe14a20 and tested with gradle; the documentation-only option was not run.

Four lines and one javadoc sentence in UnifiedHighlighter.java (the full patch):

--- a/lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/UnifiedHighlighter.java
+++ b/lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/UnifiedHighlighter.java
@@ -1078,7 +1078,8 @@ public class UnifiedHighlighter {
   /**
    * Highlights text passed as a parameter. This requires the {@link IndexSearcher} provided to this
    * highlighter is null. This use-case is more rare. Naturally, the mode of operation will be
-   * {@link OffsetSource#ANALYSIS}. The result of this method is whatever the {@link
+   * {@link OffsetSource#ANALYSIS}. As with stored values, {@code content} is truncated to {@link
+   * #getMaxLength()} before highlighting. The result of this method is whatever the {@link
    * PassageFormatter} returns. For the {@link DefaultPassageFormatter} and assuming {@code content}
    * has non-zero length, the result will be a non-null string -- so it's safe to call {@link
    * Object#toString()} on it in that case.
@@ -1102,6 +1103,10 @@ public class UnifiedHighlighter {
               + " without an IndexSearcher.");
     }
     Objects.requireNonNull(content, "content is required");
+    if (content.length() > getMaxLength()) {
+      // like LimitedStoredFieldVisitor on the searcher path; the offset strategies rely on it
+      content = content.substring(0, getMaxLength());
+    }
     Set<Term> queryTerms = extractTerms(query);
     return getFieldHighlighter(field, query, queryTerms, maxPassages)
         .highlightFieldForDoc(null, -1, content);

Objects.requireNonNull(content, "content is required");
if (content.length() > getMaxLength()) {
  // like LimitedStoredFieldVisitor on the searcher path; the offset strategies rely on it
  content = content.substring(0, getMaxLength());
}
Set<Term> queryTerms = extractTerms(query);

Trade-offs

• Behaviour change for callers who pass more than maxLength chars today (10,000 by default) and rely on highlights beyond it; they would need withMaxLength(...).
• The cut is in UTF-16 units, like the searcher path's, so it can split a supplementary character; that is the separate bug LFV-775 (the same cut on the searcher path). One surrogate-safe helper can serve both sites.
• getMaxLength() is overridable and is read twice; a local variable would read it once.
• Not tested at the edges: withMaxLength(0) (allowed), a maxLength near Integer.MAX_VALUE, an overridden getMaxLength(), or a supplementary character at the cut.

What the checks validated

• The paste-ready test: 3 of 3 pass with gradle, assertions on and off, and 30 of 30 with 10 random seeds per method; also through the JUnit runner with the patched class.
• The whole :lucene:highlighter:test task with the test added: 35 classes, 725 tests, 0 failures, assertions on and off (seed DEADBEEF), including TestUnifiedHighlighterReanalysis, TestLengthGoalBreakIterator and TestUnifiedHighlighterStrictPhrases.
• :lucene:highlighter:check -x test (google-java-format, ecj lint, forbidden APIs) passes.
• The hunt's version of the same cut (no javadoc) made the finder's, the reproducer's, the devil's advocate's and the history probe's tests pass.

A private truncateToMaxLength(String) applied in highlightWithoutSearcher and at the searcher path's highlightFieldForDoc call (UnifiedHighlighter.java:987-988), +8/-2. It also bounds text that a loadFieldValues override returns.

Trade-offs

A second behaviour change, for subclasses that override loadFieldValues (whose javadoc says "a subclass can change the source" and says nothing about maxLength); arguably a separate question. The upstream override test testCustomFieldValueSource uses short text and passes. Better as a follow-up to the recommended fix.

What the checks validated

• The whole highlighter test task with the paste-ready test: 35 classes, 725 tests, 0 failures (assertions on); format and ecj lint pass.
• The loader-override observation changes as intended: an override returning the full 48-char text with withMaxLength(20) gives <b>Lucene</b> is here. on main and Alpha beta gamma. with this fix.

Say in the highlightWithoutSearcher, getMaxLength and withMaxLength javadocs that this entry point does not apply maxLength and the caller must cut content, and correct the comment in MemoryIndexOffsetStrategy.

Trade-offs

No behaviour change, but the default bound stays a no-op on this path, and getMaxLength() keeps reporting a value that has no effect there.

• Unit. An inventory of override hooks and accessors that can bypass documented behaviour. One leg crossed each documented option with the entry points (the public highlight and highlightFields overloads, the protected highlightFieldsAsObjects they reach, and highlightWithoutSearcher).
• Signal. Code reading: the only getMaxLength() use is newLimitedStoredFieldsVisitor, reached only from loadFieldValues, and highlightWithoutSearcher never calls it.
• Witness. A differential public-API JUnit (the same text, query and maxLength through a searcher and without one) failed 1 of 1 on main with assertions on and off, and on 10.5.2 and 9.12.3; with the hunt's cut shadowed it passed (2026-10-27).
• Independent checks followed (see "Independent checks").