UnifiedHighlighter: a LengthGoalBreakIterator length goal of 0 throws "offset out of bounds" for hits with no match

LengthGoalBreakIterator.createMinLength and createClosestToLength accept any int length goal, and with a goal of 0 the UnifiedHighlighter highlights matches normally in every case tested. When a hit has non-empty text in the highlighted field and no match there, the highlighter instead builds the default summary, documented as "the first maxPassages sentences", by calling the iterator's next() from offset 0. With goal 0, next() aims at offset 0 + 0 = 0 and asks the wrapped iterator for following(0 - 1), that is following(-1), which java.text.BreakIterator rejects. So IllegalArgumentException: offset out of bounds escapes and the whole highlight call fails. On "First sentence. Second sentence." with maxPassages 1 the expected summary is "First sentence. ", which 8.4.1 returns. It broke in 8.5.0 with LUCENE-9093 (branch_8x commit 5874b9c7933; 4c9cc2cefd7 on master), which rewrote next(), and still fails on 10.6.0 and main. It takes an application that builds the highlighter itself with this iterator and a goal of 0; the failure is loud and nothing is corrupted. Negative goals also throw on this path (seen on main and 10.5.2); that wider behaviour is outside this report.

1. FieldHighlighter.java:77-79 returns null for an empty field, so only non-empty text gets further. :81 sets the text; with no match, highlightOffsetsEnums returns no passages and :89-93 asks for getSummaryPassagesNoHighlight(maxNoHighlightPassages == -1 ? maxPassages : maxNoHighlightPassages). -1 is the default (UnifiedHighlighter.java:116, :153).
2. FieldHighlighter.java:113-116: pos = breakIterator.current() (asserted to be 0 at :114), then breakIterator.next(). The highlighter wraps the user's iterator in a SplittingBreakIterator (UnifiedHighlighter.java:1135), which forwards next() (SplittingBreakIterator.java:104-106).
3. LengthGoalBreakIterator.java:148-152: "Called by getSummaryPassagesNoHighlight to generate default summary." return following(currentCache, currentCache + lengthGoal); With currentCache 0 and goal 0 the target is 0.
4. LengthGoalBreakIterator.java:166-173: the target is below the end of the text (32), so the helper looks for the first break at or after it as baseIter.following(targetIdx - 1), here following(-1).
5. A BreakIterator offset must lie between the start and the end of the text (both included); -1 is before the start. The JDK sentence iterator checks it in RuleBasedBreakIterator.checkOffset; Lucene's CustomSeparatorBreakIterator.java:74-76 does the same. Both throw IllegalArgumentException("offset out of bounds"), and nothing catches it.

With goal 1 or more the target is at least 1, so targetIdx - 1 is at least 0. In 8.4.1, next() was following(current()), which delegated straight to baseIter.following(0) for goal 0 and goal 1 alike, so both returned the first sentence.

By code reading only (not reachable today, because the first step already throws): with goal 0 and the JDK sentence iterator, a next() from a later break p would ask for following(p - 1) and get p back. A fix therefore has to advance by at least 1, not only guard offset 0. (With CustomSeparatorBreakIterator as base, following(p - 1) skips to the break after p instead, CustomSeparatorBreakIterator.java:62-88.)

• Who is hit. Users of withBreakIterator or an overridden getBreakIterator that return LengthGoalBreakIterator.createMinLength or createClosestToLength with goal 0, on the default summary path (maxNoHighlightPassages -1) for hits with non-empty field text and no match, with highlight, highlightFields or highlightWithoutSearcher. Both factories and both the JDK sentence iterator and CustomSeparatorBreakIterator as base fail.
• Default vs opt-in. Opt-in: the default break iterator is the JDK sentence iterator (UnifiedHighlighter.java:112-113), which is not affected; LengthGoalBreakIterator is @lucene.experimental. The summary path itself is on by default (UnifiedHighlighter.java:634-641).
• How it shows. One unmatched hit throws, and the whole call fails, so matched hits in the same call get no highlight either. In the probe the next call was unaffected and the same highlighter, searcher and index stayed usable.
• Scale. No size needed: a 32-character text and one non-matching query.
• Downstream. Solr maps hl.fragsize 0 to WholeBreakIterator (the whole field), uses the plain base iterator for any other fragsize <= 1 (negatives included) and wraps with this class only from 2 up, so it never wraps with goal 0. No use of the class was found in Elasticsearch or OpenSearch (code search). So the exposure found is in applications that use the Lucene API directly.
• Likelihood: rare. Goal 0 is an unusual value, the failure appears on the first unmatched hit (likely in testing), and no prior report was found in the recorded searches, more than six years after 8.5.0.
• Not only 0. Every negative goal the security probe tried (-1, -7, -1000, Integer.MIN_VALUE) also throws on the summary path, on main and 10.5.2. Those were not swept across releases, so this doc claims only goal 0 as the 8.5.0 regression (see Fixes).

Copy the test into lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/ of a Lucene checkout and run it with gradle (command below). highlightWithoutSearcher takes the same FieldHighlighter.highlightFieldForDoc -> getSummaryPassagesNoHighlight path as a searched hit without a match; the hunt's and the independent reproducer's tests also showed it through an indexed document and with createClosestToLength and CustomSeparatorBreakIterator.

Executed for this doc (2026-10-30): gradle on a clean 645d4fe14a20 (fails); the standalone runner on the stock main jars with -ea and -da (fails); the 10.6.0, 10.5.2 and 9.12.3 release jars (fail, same frames). The test does not compile on 8.x or 9.0.0 (the builder API and the test-framework package first appear in 9.1.0); the 8.x and 9.0.0 results come from version-neutral probes. The test catches the exception only to name the expected value in the failure message; the cause is kept.

cp $REPRO/TestLengthGoalZeroGoalSummary.java $LUCENE/lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/
cd $LUCENE && ./gradlew :lucene:highlighter:test --tests org.apache.lucene.search.uhighlight.TestLengthGoalZeroGoalSummary -Ptests.seed=DEADBEEF

# gradle, main 645d4fe14a20, clean scratch worktree, -Ptests.seed=DEADBEEF
TestLengthGoalZeroGoalSummary > testZeroLengthGoalDefaultSummary FAILED
    java.lang.AssertionError: expected the summary "First sentence. " but got java.lang.IllegalArgumentException: offset out of bounds
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:51624CE1C572F627]:0)
        at org.apache.lucene.search.uhighlight.TestLengthGoalZeroGoalSummary.testZeroLengthGoalDefaultSummary(TestLengthGoalZeroGoalSummary.java:44)

        Caused by:
        java.lang.IllegalArgumentException: offset out of bounds
            at java.base/sun.text.RuleBasedBreakIterator.checkOffset(RuleBasedBreakIterator.java:716)
            at java.base/sun.text.RuleBasedBreakIterator.following(RuleBasedBreakIterator.java:730)
            at org.apache.lucene.search.uhighlight.LengthGoalBreakIterator.following(LengthGoalBreakIterator.java:173)
            at org.apache.lucene.search.uhighlight.LengthGoalBreakIterator.next(LengthGoalBreakIterator.java:151)
            at org.apache.lucene.search.uhighlight.SplittingBreakIterator.next(SplittingBreakIterator.java:106)
            at org.apache.lucene.search.uhighlight.FieldHighlighter.getSummaryPassagesNoHighlight(FieldHighlighter.java:116)
            at org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:92)
            at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightWithoutSearcher(UnifiedHighlighter.java:1107)
            at org.apache.lucene.search.uhighlight.TestLengthGoalZeroGoalSummary.testZeroLengthGoalDefaultSummary(TestLengthGoalZeroGoalSummary.java:40)

1 test completed, 1 failed

# standalone runner, stock main jars: the same exception and frames with -ea and with -da;
# the 10.6.0, 10.5.2 and 9.12.3 release jars: the same exception at LengthGoalBreakIterator.java:173 <- :151

• No documented or enforced bound. createMinLength: "Breaks will be at least {@code minLength} apart (to the extent possible)" (LengthGoalBreakIterator.java:43-46); createClosestToLength: "Breaks will be on average {@code targetLength} apart" (:53-57). The constructor validates only fragmentAlignment (:73-75).
• The failing path is a named caller. "the only methods that will get called are ... getSummaryPassagesNoHighlight: current(), first(), next()" (:80-82); next() is commented "Called by getSummaryPassagesNoHighlight to generate default summary" (:148). FieldHighlighter.java:104-107: "By default this just returns the first {@link #maxPassages} sentences".
• Goal 0 is legal upstream. TestLengthGoalBreakIterator.java:100 uses random().nextInt(4) as the goal (0 included) and :143 asserts highlightMinLen(CONTENT, query, 0, 0.f) is " Cc <b>dd</b>.". No test expects an exception for any goal.
• A regression, with the test narrowed in the same commit. All 34 releases 6.4.0-8.4.1 return the summary for goal 0. The class's original summary test (LUCENE-7620) drew the goal from random().nextInt(10); LUCENE-9093 changed it to 6 + random().nextInt(4) in the commit that introduced the failure, with no reason given in #10135 or the lucene-solr #1123 review.
• The class prefers not to throw. Its javadoc: "Some assumptions are checked with Java assertions" (:28-29). When adding the class, its author, David Smiley, wrote on #8671 (LUCENE-7620): "I added assertions not exceptions because if per chance this circumstance happens in production, it's really okay to return possibly the wrong break and have a passage that isn't quite the ideal size rather than throw some exception."

The strongest case for "intended or misuse" (the cross-model lenses, the independent reproducer, the devil's advocate and the adjudication): the class is @lucene.experimental and "not a general purpose BreakIterator" (LengthGoalBreakIterator.java:27-31); a goal of 0 is degenerate, since "at least 0 apart" adds nothing over the base iterator; Solr never wraps the iterator for fragsize <= 1; and no current summary test uses goal 0.

Why it fails: the failing path is the UnifiedHighlighter's own default summary, which the class names as its caller; the value is accepted, and upstream's own tests use it on the highlight path; 8.4.1 returned the right summary; and the error is an internal JDK offset check on targetIdx - 1, raised on a later, unrelated hit, not an argument check. At most the case supports rejecting goals below 1 up front (fix option B). That is a plausible maintainer choice and a fix direction, not evidence that the current exception is intended; either A or B removes the late exception.

Two options plus a workaround. Option A was validated with gradle in a clean scratch worktree at 645d4fe14a20; option B was run too and breaks two upstream tests. Not part of this report (seen by the security probe on main and 10.5.2, not swept across releases, not analysed): on the summary path every goal it tried other than 1 throws "offset out of bounds" (0, -1, -7, -1000, Integer.MIN_VALUE, Integer.MAX_VALUE); on the highlight path -1000 throws, and so do Integer.MIN_VALUE and Integer.MAX_VALUE with some alignments (apparently the target arithmetic going negative or overflowing). Option B would also cover the negative values.

In LengthGoalBreakIterator.next() (:151) aim at currentCache + Math.max(1, lengthGoal).

   // Called by getSummaryPassagesNoHighlight to generate default summary.
   @Override
   public int next() {
-    return following(currentCache, currentCache + lengthGoal);
+    return following(currentCache, currentCache + Math.max(1, lengthGoal));
   }

Trade-offs

• Goal 0 then behaves like goal 1 on the summary path (the next base break), which is what 8.4.1 returned. No change for goals of 1 or more.
• The highlight path (following(int), preceding(int)) is untouched, so goal-0 highlights stay as today.
• It also covers the later-break case (a next() from break p can no longer return p).
• By code reading, it also turns every negative goal into goal 1 on the summary path (not run); it does not touch the Integer.MAX_VALUE overflow cases.
• Silent: a goal of 0 stays accepted. Maintainers who want degenerate goals rejected would choose B instead.

What the checks validated

Executed 2026-10-30 in a clean scratch worktree at 645d4fe14a20 with gradle: the minimal test passes (it fails there without the fix), and the saved logs of ./gradlew :lucene:highlighter:test report 723 passing tests and 0 failures with seed DEADBEEF and with seed 1E9600C4 (run with -Ptests.asserts=false). An independent fact check repeated it in a fresh worktree (seed FC493DA2, -Ptests.asserts=false): 723 tests in 35 suites, 0 failures, the new test among them. The hunt had already seen its own and the independent reproducer's tests pass with the same line, and 470 upstream highlighter tests under 3 seeds on ea57360912ed.

In the private constructor, before the alignment check: if (lengthGoal < 1) throw new IllegalArgumentException("lengthGoal must be >= 1, got " + lengthGoal);

Trade-offs

• A clear error when the iterator is built, instead of a JDK offset error on a later hit; also rejects negative goals, but not the Integer.MAX_VALUE overflow cases.
• An API behaviour change: goal 0 works today on the highlight path, and upstream tests rely on it. It needs those tests changed and a CHANGES note, and an application passing 0 would then fail on every call instead of only on unmatched hits. Its regression test would be an expectThrows on the factory, not the minimal test.

What the checks validated

Run 2026-10-30, same worktree, seed DEADBEEF: TestLengthGoalBreakIterator.testMinLen fails at :143 (always) and testTargetLen at :100 (when random().nextInt(4) draws 0, as with this seed), both with "lengthGoal must be >= 1, got 0". The minimal test still fails, now with that message, which is B's intended behaviour.

Use the base iterator when the goal is below 1 (Solr does this for 1 and below, except that 0 gives the whole field), or call withMaxNoHighlightPassages(0) to switch the summary off; an unmatched hit then gets null instead of a summary.

Trade-offs

The second loses the default summary for unmatched hits.

What the checks validated

Executed: createMinLength(sentences, 0, 0.5f) with withMaxNoHighlightPassages(0) returns null for the unmatched text instead of throwing (main, -ea and -da).

Method: bounded-exhaustive translation validation (a semi-formal method, no proof), on the stock main jars of ea57360912ed.

• A reference semantics for the UnifiedHighlighter's passage layer was written from the javadoc (break iterators, the length-goal rule, the default summary, the formatter), independent of the code.
• The harness ran the real highlighter on every single-valued text of up to 7 code points over {a, space, period, U+1D400} and every 2-3-valued content up to 5 code points, with 90 LengthGoalBreakIterator configurations (both factories x goals 0, 1, 2, 3, 5 x alignment 0, 0.5, 1 x three bases), about 18 million checks in these three tiers (the 2-3-valued tier with 54 of the configurations), with assertions on and off.
• One oracle was "no exception on legal input": IllegalArgumentException: offset out of bounds appeared only for goal 0, only on the summary path. The other failures of the run belong to four separate bugs with different fix sites.
• The finder then wrote a public-API JUnit test; it was later reduced to the single test shown here.