OpenNLPSentenceBreakIterator crashes on text that starts with whitespace

OpenNLPSentenceBreakIterator, the public java.text.BreakIterator of the OpenNLP analysis module, stores the sentence starts that OpenNLP's detector reports, and the detector trims whitespace. For a text that starts with whitespace the first stored start (2 for two leading spaces) therefore lies after the beginning of the text, which first() and next() treat as the first boundary. Every method that uses the stored first start then goes wrong: a backward walk returns the non-boundary 2 and then throws ArrayIndexOutOfBoundsException, following() (and the JDK's default isBoundary()) at an offset inside the whitespace recurses until StackOverflowError on a one-sentence text or, with two or more sentences, fails an internal assertion when assertions are enabled (with them off the result is right), and preceding() and next(-n) return 2. One leading space, newline, tab or no-break space is enough. OpenNLPTokenizer, the class's only user in Lucene, only walks forward and is not affected. Every release from 7.3.0 to 10.6.0 (10.6.0 by reading) and main is affected; a one-statement fix in setText was validated with the module's tests.

• OpenNLPSentenceBreakIterator: a public final java.text.BreakIterator in the exported package org.apache.lucene.analysis.opennlp (OpenNLPSentenceBreakIterator.java:26-27). It keeps an array sentenceStarts with the start offset of each detected sentence and an index currentSentence.
• The detector: NLPSentenceDetectorOp.splitSentences calls OpenNLP's SentenceDetectorME.sentPosDetect (NLPSentenceDetectorOp.java:39-47), which trims whitespace from each sentence span. For " Sentence number 1 has 6 words. Sentence number 2, 5 words." it returns the spans [2..32) and [33..60). Every opennlp-tools version Lucene has bundled (1.8.3 to 2.5.12) trims. A single leading space, newline, tab, no-break space (U+00A0) or em space (U+2003) is trimmed the same way.

1. setText stores the spans' starts unchanged (:195-205, the store at :203): sentenceStarts = [2, 33].
2. first() returns the begin index, 0 (:44-48), and next() moves to sentenceStarts[1] = 33 and then to the end, 60 (:63-72). The forward walk counts the two spaces as part of sentence 0, so 2 is never returned forward.
3. previous() returns DONE only when the current index equals the begin index (:119). From last() (60) it moves to sentenceStarts[1] = 33 (:126-127), then to sentenceStarts[0] = 2 (:129). 2 is not the begin index, so the next call reads sentenceStarts[-1]: ArrayIndexOutOfBoundsException: Index -1. A one-sentence text fails the same way after 32, 2.
4. following(pos) accepts any offset from begin to end (:76-77), returns DONE in the last sentence (:81-87), and otherwise runs a binary search commented "there are at least two sentences" (:88-90). An offset inside the leading whitespace is below every stored start. With one sentence (sentenceStarts = [2]) the search is moveToSentenceAt(0, 0, -1), which calls itself with the same arguments forever (:97-102): StackOverflowError. With two sentences the assertion pos >= sentenceStarts[currentSentence] fails (:110) when assertions are on; with them off it returns 33, the right value, by accident.
5. For a non-zero offset the JDK's default isBoundary(offset) calls following(offset - 1) (offset 0 returns true directly), so isBoundary(1) on a one-sentence text with one leading newline overflows the stack too; on a two-sentence text it fails the same assertion with -ea and returns false with -da. preceding(38) and next(-2) after last() return sentenceStarts[0] = 2 (:157, :175): no crash, but not a boundary.
6. Trailing whitespace is harmless: last() and the end checks use the end index (:54, :64, :85), never a span end. And preceding() already handles an offset below the first stored start (:143-149, present since the class was added); previous(), following() and next(int) do not.

On main 645d4fe14a20 with the module's test sentence model: forward walk [0, 33, 60]; backward walk 60, 33, 2, then ArrayIndexOutOfBoundsException; following(0) and following(1) on the one-sentence text: StackOverflowError; following(0) on the two-sentence text: AssertionError with -ea, 33 with -da; preceding(38) = 2 and next(-2) = 2. The same text without the leading spaces walks 58, 31, 0, DONE; with three trailing spaces 61, 31, 0, DONE.

• Who is exposed. Users of the opt-in lucene-analysis-opennlp module who navigate OpenNLPSentenceBreakIterator themselves, as a general java.text.BreakIterator. Not default; Lucene itself never does this.
• Lucene's own paths are safe. OpenNLPTokenizer (OpenNLPTokenizer.java:52), the only in-tree user, drives the iterator through SegmentingTokenizerBase, which calls only setText, current() and next() (SegmentingTokenizerBase.java:114, :167, :194, :199); the failing texts tokenize and index normally (executed). UnifiedHighlighter and FastVectorHighlighter configured with this iterator made 32 and 9 calls on ordinarily tokenized fields without an exception: FieldHighlighter asks for boundaries only at offsets inside a match (FieldHighlighter.java:182, :186), which never lie in the leading whitespace, and BreakIteratorBoundaryScanner calls last() then one previous() (BreakIteratorBoundaryScanner.java:40-41), which stops before the failing step. Only a contrived field whose tokens cover the leading whitespace (EdgeNGramTokenizer(1, 3), query gram " S") made UnifiedHighlighter.highlight throw StackOverflowError.
• What it takes. One whitespace character (space, newline, tab, no-break space, em space) before the first sentence, with OpenNLP's model-backed detector: the trim is in its code, not in the model, though every run used the module's test models. The no-model NLPSentenceDetectorOp() returns one untrimmed span (NLPSentenceDetectorOp.java:43-45) and is not affected, and whitespace-only text has no sentence and walks correctly. Deterministic: the backward walk and the one-sentence following() fail with assertions on and off, the two-sentence following() only with them on.
• Effect. One failed call. ArrayIndexOutOfBoundsException is a RuntimeException; StackOverflowError is an Error, so catch (Exception e) does not stop it. It comes within milliseconds (at most 9 ms over 20 runs), nothing is written to an index, and the iterator works again after the next setText.
• Versions. Every release from 7.3.0 (2018) to 10.6.0 (10.6.0 by reading), and main.
• How likely. Rare: a public code search (2026-10-26) found no use of the class outside its tokenizer in Solr, Elasticsearch or OpenSearch, and the recorded prior-art search found no public report.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.StackOverflowError
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       org.apache.lucene.analysis.opennlp.OpenNLPSentenceBreakIterator.moveToSentenceAt(OpenNLPSentenceBreakIterator.java:102)
       [...]

• Contract (JDK 25 javadoc). previous(): "Returns the boundary preceding the current boundary. If the current boundary is the first text boundary, it returns BreakIterator.DONE". following(int offset): "Returns the first boundary following the specified character offset"; the only listed exception is IllegalArgumentException for an offset "less than the first text boundary or greater than the last text boundary", and the class's own check accepts 0 (OpenNLPSentenceBreakIterator.java:76-77). The BreakIterator class javadoc's example "print each sentence in reverse order" is the last()/previous() loop that crashes.
• The class's own boundaries. first() returns the begin index (:44-48), previous() stops there (:119), and the upstream tests assert first() equals the begin index (TestOpenNLPSentenceBreakIterator.java:127-128, :219). The stored first start is the only place that disagrees.
• Public, tested as a general iterator. public final class in an exported package (module-info.java:25), no @lucene.internal or @lucene.experimental tag, javadoc "A BreakIterator that splits sentences using an OpenNLP sentence chunking model". The upstream test asserts previous(), following(), preceding() and next(n) (TestOpenNLPSentenceBreakIterator.java:84-115, :126-149).
• Precedent. #12210, another contributor's report of an ArrayIndexOutOfBoundsException in this class's preceding(), was fixed as a bug by #13170 (first released in 10.0.0).
• A coverage gap, not a decision. The upstream tests cover whitespace-only text and padding before a slice's begin index (testWhitespaceOnly, testSliceStart), never an iterated range that starts with whitespace followed by a sentence. With the fix all 33 upstream tests of the module still pass.
• Anticipated in one method. preceding() has handled an offset below the first stored start since the class was added (:143-149); previous() and following() never did.
• Reference behaviour. The JDK's sentence iterator walks the same text with a leading newline correctly (0, 32, 59 forward and backward), and this class handles trailing whitespace correctly.

The strongest case for "intended or misuse": the class was added as a "shim class" so that OpenNLPTokenizer can extend SegmentingTokenizerBase (LUCENE-2899 patch comment), and the tokenizer only walks forward; the class already documents a deviation from the BreakIterator javadoc (the JDK-8015110 comments: DONE in the last sentence for following(), before the first start for preceding()); and callers could trim their text.

Why it fails: nothing marks the class internal or forward-only, its tests assert backward and random-access navigation, and #12210 came from a direct preceding() user. JDK-8015110 ("following(int) changes the current position when it returns DONE") justifies returning DONE, not crashes or a wrong first start, and the fix leaves that convention unchanged. No javadoc asks for trimmed text, setText accepts any text, and one newline triggers it. Verdict: not intended, a defect. The cross-model reviews (refute and works-as-designed) rated the "intended" case weak; the devil's advocate found that no argument survives, and the adjudication confirmed the bug.

Option A was applied in a scratch worktree of main 645d4fe14a20 and run through gradle with assertions on and off; options B and C are described only. The workaround needs no Lucene change and was run on unfixed main.

In setText (OpenNLPSentenceBreakIterator.java:195-205), after the loop that stores spans[i].getStart() + text.getBeginIndex(), set sentenceStarts[0] = text.getBeginIndex() when there is at least one sentence. This is the finder's fix.

What it changes. Only texts that start with whitespace (otherwise sentenceStarts[0] already equals the begin index). Backward walks end ..., 0, DONE; following(0) on two sentences returns 33 with assertions on too; preceding(38) and next(-2) return 0 instead of 2; following() inside the whitespace of a one-sentence text returns DONE, the class's existing last-sentence convention. The detector's spans are untouched.

       sentenceStarts[i] = spans[i].getStart() + text.getBeginIndex();
     }
+    if (sentenceStarts.length > 0) {
+      // the first sentence begins at the start of the text: first() and next() already treat any
+      // whitespace before the detected (trimmed) sentence start as part of the first sentence
+      sentenceStarts[0] = text.getBeginIndex();
+    }
   }

Trade-offs

• One statement at the root cause; every navigation method inherits the corrected start. No performance effect (one store per setText).
• With the fix, the JDK's default isBoundary(1) on a one-sentence text with one leading character throws IllegalArgumentException instead of StackOverflowError: following(0) now returns the class's usual DONE, and isBoundary turns DONE into that exception. A one-sentence text without leading whitespace gets the same exception today. That last-sentence convention is upstream-asserted and separate from this bug.
• preceding()'s branch for an offset below the first start (:143-149) becomes unreachable; it can stay or go.
• A caller that used preceding() or next(-n) as a snippet start in the second sentence now gets offset 0, the leading whitespace included, instead of 2 (read, not run through a highlighter).

What the checks validated

• The 3 new tests pass with assertions on and off (stock: 3 of 3 and 2 of 3 fail).
• The upstream TestOpenNLPSentenceBreakIterator (9 tests) and all of :lucene:analysis:opennlp:test (36 tests: 33 upstream + 3 new) pass with assertions on and off.
• :lucene:analysis:opennlp:check -x test passes (google-java-format, ecjLint, forbiddenApis, licenses), with the new test in place.
• A plain-Java probe of every symptom: backward walks 60, 33, 0, DONE and 32, 0, DONE; following(0) 33 (two sentences) and DONE (one sentence); preceding(38) 0; next(-2) 0.
• Earlier, in the hunt: the finder's and the reproducer's tests pass with it with assertions on and off (the adjudication's re-run with them on), the history probe passes with it on 7.3.0 and main, and the walk oracle's backward-walk and next(-k) checks report 0 failures and 0 crashes (its remaining next(k) failures are the separate LFV-622 bug).

Keep the trimmed starts and treat sentence 0's boundary as the begin index wherever a start is returned or compared: previous() (:127, :129), following() and moveToSentenceAt() for offsets below the first start (:88-112), preceding() (:157) and next(int) (:175).

Trade-offs

Four methods must agree, and fixing only previous() and following() removes the crashes but leaves preceding() and next(-n) returning the non-boundary 2. Same end result as A with more code. Not validated.

Make NLPSentenceDetectorOp.splitSentences (NLPSentenceDetectorOp.java:39-47) start the first span at 0.

Trade-offs

Changes the output of a public method in the exported tools package, whose callers may rely on the trimmed spans; the iterator is its only in-tree caller (OpenNLPSentenceBreakIterator.java:199). Not recommended, not validated.

Give the iterator a range that starts at the first non-whitespace character; offsets stay in the original text's coordinates because the class adds getBeginIndex() (:203):

int begin = 0;
while (begin < text.length()
    && (Character.isWhitespace(text.charAt(begin)) || Character.isSpaceChar(text.charAt(begin)))) {
  begin++;
}
bi.setText(new StringCharacterIterator(text, begin, text.length(), begin));

Trade-offs

The whitespace test must be at least as wide as OpenNLP's: the detector also trims U+00A0, for which Character.isWhitespace is false, hence the isSpaceChar term. The snippet is a method fragment (bi is the iterator, text the String) and needs import java.text.StringCharacterIterator. Offsets inside the skipped whitespace are then outside the range, so following() and preceding() there throw the documented IllegalArgumentException (:76-77, :137-138).

What the checks validated

On unfixed main this gives forward [2, 33, 60], backward [60, 33, 2] then DONE, and following(2) = 33, with no crash. The snippet exactly as printed was run with assertions on and off for the leading characters two spaces, newline plus tab, U+00A0, U+2003 and none: no crash, and every backward walk ends in DONE.

• Method (semi-formal). In the hunt's unit on the OpenNLP filters against the OpenNLP library, the finder ran the real OpenNLPSentenceBreakIterator against a stub sentence detector (the class only consumes the detector's spans): 84 configurations (1 to 14 sentences, start gaps 1 to 3, the first sentence at offset 0 or 2), following() and preceding() at every offset and next(k) from first(), 3,192 checks. The first signal: 84 crashes, all at offsets inside the leading gap, none without one.
• Walk oracle. An internal-consistency check that does not depend on the class's documented JDK-8015110 deviation: the backward walk must equal the reversed forward walk, and next(±k) must equal k single steps. Its backward-walk check crashed on all 42 configurations with a leading gap and on none without; with the fix, 0 failures and 0 crashes (its next(k) failures are the separate LFV-622 bug, which needs its own fix).
• Then a public-API JUnit with the real detector and the module's test model, and the checks below.