OpenNLP POS, chunker and lemmatizer filters emit nothing when an earlier filter removes the whole first sentence

The OpenNLP POS, chunker and lemmatizer filters read their input one sentence at a time through the shared helper SentenceAttributeExtractor, which starts at sentence index 0. When an earlier filter in the chain removed every token of the value's first sentence, the first token the helper sees belongs to sentence 1, the helper returns an empty first sentence, and the filter takes that as the end of the stream. The whole value then indexes no terms while its stored text stays intact: a nouns-only chain turns He did not come back. They sent him running in the evening. into [] instead of [they, evening]. A removed sentence in the middle is handled correctly. Every release from 9.5.0 to 10.6.0 (10.6.0 by reading) and main is affected; the 4-line fix in the helper was validated with the module's tests.

• Sentence index: OpenNLPTokenizer stores, for each token, the number of its sentence in SentenceAttribute: 0 for the first sentence, 1 for the second, and so on (OpenNLPTokenizer.java:31-34; the index starts at -1 at :45 and goes up by one for each detected sentence at :73).
• Sentence-aware filters: OpenNLPPOSFilter, OpenNLPChunkerFilter and OpenNLPLemmatizerFilter tag or rewrite whole sentences, so they first collect one sentence of tokens through SentenceAttributeExtractor, then emit its tokens one by one.
• Look-ahead: the helper reads one token past the end of a sentence to see that the index changed, and keeps that token for the next sentence.

1. SentenceAttributeExtractor starts with currSentence = 0 (SentenceAttributeExtractor.java:40; reset() sets it back to 0 at :78).
2. extractSentenceAttributes() (:51-65) reads tokens while currSentence == currSentenceTmp && hasNextToken (:57). Each token it reads is only stored as the look-ahead (:62); the previous look-ahead is added to the sentence (:59-61).
3. An earlier filter removed every token of sentence 0, so the first token the helper ever sees, They, has index 1. First call: 0 == 1 is false, the loop stops after one token, there is no previous look-ahead to add, and They becomes the look-ahead. The call returns an empty list.
4. The three filters end the stream when allSentencesProcessed() || nextSentence().isEmpty() is true (OpenNLPPOSFilter.java:52-56, OpenNLPChunkerFilter.java:55-59, OpenNLPLemmatizerFilter.java:66-70). More input is left, but the sentence is empty, so incrementToken() returns false: They and everything after it are lost, and the rest of the input is never read.
5. In the middle of a stream the look-ahead is never null and only a change of index is tested, so a skipped index (0, 0, 2, 2) still splits correctly. Only the initial value 0 is wrong.
6. The loss needs a surviving token in a later sentence: if the filter removes every token of a one-sentence value, empty output is correct. If the first two or more sentences are removed, the first index is 2 or more, with the same result.

For a value longer than the tokenizer's 1024-char buffer (SegmentingTokenizerBase.java:39), the blanked value is never read to its end, so end() reports a premature final offset, the amount read so far (SegmentingTokenizerBase.java:119-122), instead of its length: 1024 instead of 1504 in the reproducer's test; the differential run also recorded 1029 and 1049. In a multi-valued field the next value's offsets shift accordingly ([1054,1061) instead of [1539,1546)), pointing into the earlier value's text. CheckIndex stays clean.

• Who is exposed. Users of the lucene-analysis-opennlp module (factories opennlpPOS, opennlpChunker, opennlplemmatizer, or the classes) whose chain puts a token-removing filter before a sentence-aware OpenNLP filter: StopFilter, LengthFilter and a TypeTokenFilter on POS tags were executed; any filter that can drop every token of a sentence works. The module is opt-in, and no documented example, upstream test or third-party configuration found uses such a chain (they keep the OpenNLP stages together and put TypeTokenFilter last).
• What it takes. The filter must remove every token of the value's first sentence, punctuation included: a short opener such as Hi. or No. under a stop list or a minimum length, or any first sentence without a noun under a nouns-only filter. Once such a chain exists, it fires on every value of that shape.
• Effect. The whole value indexes no terms (as long as a later sentence keeps a token): no exception, no log line, CheckIndex clean, stored text intact. Deterministic, with assertions on and off, at any size. In a differential run on 404 generated documents, the POS filter's output in the chain tokenizer, StopFilter, POS differed from the library reference for 17 documents; with the fixes, for none.
• Query side. If the same chain analyzes queries, a query text whose first sentence is removed builds no query at all (QueryBuilder returns null).
• Offsets. Values longer than 1024 chars end with a premature final offset (1024 in the repros), which shifts the offsets of later values of a multi-valued field.
• Lasting. Documents indexed with an affected chain keep the empty field until reindexed.
• How likely. Rare: it needs a legal but undocumented chain shape, and nobody has reported it since 9.5.0 (2023).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[They, sent, him, running]> but was:<[]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.opennlp.TestChunkerAfterRemovedFirstSentence.testFirstSentenceRemoved(TestChunkerAfterRemovedFirstSentence.java:53)

• The contracts. OpenNLPPOSFilter.java:32: "Run OpenNLP POS tagger. Tags all terms in the TypeAttribute." OpenNLPChunkerFilter.java:32-36: "Prerequisite: the OpenNLPTokenizer and OpenNLPPOSFilter must precede this filter. Tags terms in the TypeAttribute, replacing the POS tags previously put there by OpenNLPPOSFilter." The lemmatizer "Runs OpenNLP dictionary-based and/or MaxEnt lemmatizers" (OpenNLPLemmatizerFilter.java:34) and replaces each non-keyword term by its lemma (:73-76). No javadoc, CHANGES or MIGRATE entry, code comment or test restricts which filters may sit between the stages. SentenceAttribute only "tracks what sentence a given token belongs to" and promises nothing about index 0.
• Filters between the stages are expected. The lemmatizer reads KeywordAttribute (OpenNLPLemmatizerFilter.java:49, :74, :87) so that keyword filters between POS and lemmatizer work. Upstream tests put KeywordRepeatFilter between the tokenizer and the POS filter (TestOpenNLPPOSFilterFactory.java:150-160) and between the POS filter and the lemmatizer (TestOpenNLPLemmatizerFilterFactory.java:230-238, :301-308, :323-345).
• The asymmetry. A fully removed middle sentence works, because the helper only tests for a change of index; the initial currSentence = 0 is the only assumption that breaks. Losing the whole value is not a trade-off anyone chose.
• History. #11734, "Fix repeating token sentence boundary bug", added the helper and fixed the 9.5.0 CHANGES entries GITHUB#11735 "KeywordRepeatFilter + OpenNLPLemmatizer always drops last token of a stream" and GITHUB#11771 "... sometimes arbitrarily exits token stream" (lucene/CHANGES.txt:3859-3863). Its review did not discuss a first sentence whose index is not 0.
• Nothing pins the current output. No upstream test puts a token-removing filter before a sentence-aware filter; with the fix all 33 upstream tests of the module pass.

The checkers built the strongest case for "unsupported chain": (1) no documented or example chain triggers it; the factory javadoc, the Solr reference guide, the upstream tests and the third-party configurations found keep the OpenNLP stages next to each other and put TypeTokenFilter after the last one; (2) the module overview says the tokenizer keeps punctuation because "This is required for the following taggers to operate properly" (overview.html:32-36), and the Solr reference guide ("OpenNLP Integration") says "the same tokenization using the same models must be used at runtime for optimal performance"; POS tagging and chunking use the sentence as context; (3) the chunker is excluded from the random-chain tests because "other filters must precede this one (see docs)" (OpenNLPChunkerFilter.java:37). So a maintainer could call removing tokens before the POS filter or the chunker unsupported, and lower tag quality there acceptable.

Why it fails: those notes are about tagging quality, not about losing every token, and nothing in them explains ending the stream for the whole value. For the dictionary lemmatizer the quality argument has nothing to hold on to in the repro: with fix A the nouns-only chain gives the same [they, evening] with and without the noun-free first sentence. Nothing restricts in-between filters, upstream tests use them, and the same code handles a removed middle sentence. The cross-model fact check rated the "intended" case moderate before the POS filter and before the chunker, weak before the dictionary lemmatizer.

Options A and B were each applied in a scratch worktree of main a9779ef3c193 and run through gradle with assertions on and off; the workaround needs no code change. Indexes built with an affected chain keep the empty fields until the documents are reindexed.

In extractSentenceAttributes() (SentenceAttributeExtractor.java:54-58), when prevAttributeSource == null (the first call after reset()), set currSentence to the first token's index before the comparison. This is the finder's fix.

What it changes. Only that first call. Streams whose first token has index 0 behave exactly as before; an empty stream still returns an empty list, so the filters still end it. All three filters, and any other user of the public helper, are fixed at once.

       hasNextToken = input.incrementToken();
       int currSentenceTmp = sentenceAtt.getSentenceIndex();
+      if (prevAttributeSource == null) {
+        // first token of the stream: its sentence is the current one, whatever its index
+        currSentence = currSentenceTmp;
+      }
       hasNext = (currSentence == currSentenceTmp && hasNextToken);

Trade-offs

• The smallest change, at the root cause. Indexed terms change only for values that were losing all their tokens.
• It does not touch the separate lemmatizer bug with sentence-final keyword tokens (a different fix site): in the exhaustive check fix A removes every first-sentence case and that bug's own fix removes the rest (both fixes: 0 of 55,987 lemmatizer cases).

What the checks validated

• The 3 new tests pass with assertions on and off (stock: 3 of 3 fail).
• All of :lucene:analysis:opennlp:test: 6 classes, 36 tests, 0 failures, assertions on and off (stock: only the 3 new tests fail).
• :lucene:analysis:opennlp:check -x test passes (google-java-format, ecjLint, forbiddenApis, licenses).
• Through the repro harness with the fix shadowed: the new test 3 of 3, the finder's test 3 of 3, the independent reproducer's 5 of 5 and a separate verification test pass.
• The hunt's exhaustive check: no first-sentence case left (728 POS, 728 chunker and 37,324 per lemmatizer variant before).

In OpenNLPPOSFilter, OpenNLPChunkerFilter and OpenNLPLemmatizerFilter, replace allSentencesProcessed() || nextSentence().isEmpty() by a loop that ends the stream only when allSentencesProcessed() is true and otherwise reads the next sentence while it is empty (do { if (allSentencesProcessed()) return false; } while (nextSentence().isEmpty());).

Trade-offs

Three call sites instead of one, and every other consumer of the helper must do the same; the helper still returns a misleading empty first sentence. It also retries after other empty batches, but each filter first runs its OpenNLP operation on the empty arrays (OpenNLPPOSFilter.java:70-71, OpenNLPChunkerFilter.java:76-77, OpenNLPLemmatizerFilter.java:94-96); that was validated only for the demonstrated case, with the bundled OpenNLP version. A and B can be combined; A alone covers every case found.

What the checks validated

• The 3 new tests pass with assertions on and off.
• All of :lucene:analysis:opennlp:test: 6 classes, 36 tests, 0 failures, assertions on and off.
• Not run through check or the exhaustive harness.

For example tokenizer, POS, lemmatizer, then TypeTokenFilter or StopFilter: the order the documented examples use. No code change.

Trade-offs

Not always possible: the chunker replaces the POS tags with chunk tags, so a POS-based TypeTokenFilter cannot simply move after it. Existing documents still need reindexing.

• Method (semi-formal): bounded-exhaustive translation validation of the stock main jar, in the hunt's unit on the OpenNLP filters against the OpenNLP library. The finder built an independent per-sentence reference that calls the OpenNLP operations directly, and formed the hypothesis by reading SentenceAttributeExtractor while doing so.
• Enumeration. Every sentence-index sequence over {0, 1, 2}^n for n <= 6 (and every keyword pattern for the lemmatizer): 1,093 cases each for POS and chunker, 55,987 per lemmatizer variant. Every case whose first token had a sentence index other than 0 gave empty output: 728 for POS, 728 for the chunker, 37,324 per lemmatizer variant, 0 unexplained; with fix A, 0. These count a diagnostic model space that also enumerates gaps and non-monotone index sequences the tokenizer does not emit; they say nothing about how often real text triggers the bug.
• Real text. A differential run of 404 generated documents through the tokenizer and 10 chains, with the library called directly as the reference, showed the same defect (the POS stage of tokenizer, StopFilter, POS differed for 17 of 404 documents); with both of the unit's fixes shadowed, 0 mismatches.
• Then a public-API JUnit; independent checkers followed (see "Independent checks").