OpenNLPLemmatizerFilter drops keyword-marked words that end a sentence

OpenNLPLemmatizerFilter leaves keyword-marked tokens unlemmatized, but since 9.5.0 it ends each sentence as soon as it has emitted one lemma per non-keyword token. Keyword tokens get no lemma, so protected words after the last ordinary word of a sentence are silently dropped: They sent him running in the evening with evening protected by KeywordMarkerFilter becomes [they, send, he, run, in, the], and a sentence made only of protected words keeps just its first word. The words are missing at index and at query time, with no error. It needs the opt-in OpenNLP module with a keyword marker in front of the lemmatizer, and it hits short text whose last words are protected and have no final punctuation. Introduced by #11734 (9.5.0); 9.4.2 kept the word; restoring the pre-9.5 end-of-sentence test fixes it.

• Keyword token: a token whose KeywordAttribute.isKeyword() is true. Filters such as KeywordMarkerFilter (a protected-word set or a pattern) and StemmerOverrideFilter set it so that stemmers and lemmatizers leave the word unchanged; the KeywordAttribute javadoc says stemming filters use it to "conditionally skip a term".
• OpenNLP chain: OpenNLPTokenizer splits text into sentences and tokens (each token carries its sentence index), OpenNLPPOSFilter adds part-of-speech tags, and OpenNLPLemmatizerFilter replaces each word by its dictionary form, one sentence at a time. All of it lives in the opt-in analysis-opennlp module and needs user-supplied models.

1. For each sentence, nextSentence() buffers every token, keyword or not, but sends only the non-keyword ones to the lemmatizer (OpenNLPLemmatizerFilter.java:86-90, :94 at a9779ef3c193). So lemmas holds one entry per non-keyword token (:52).
2. incrementToken() emits the buffered tokens in order and takes the next lemma only for a non-keyword token (:74-75).
3. The end-of-sentence test is lemmaNum >= lemmas.length (:64). It turns true as soon as the last non-keyword token is out, while keyword tokens after it are still buffered.
4. The next call then returns false (no more sentences) or loads the next sentence and replaces the token iterator (:95). Either way those keyword tokens are never emitted.
5. A sentence with no non-keyword token has lemmas.length == 0, so only its first token gets out.

With the module's test models and evening protected: 7 buffered tokens, 6 lemmas [they, send, he, run, in, the]. Calls 1-6 emit them; at call 7, 6 >= 6 ends the sentence, there is no more input, and the filter returns false with evening still in the iterator. The indexed terms are [he, in, run, send, the, they]. With a final . the output is [they, send, he, run, in, the, evening, .], because the . is the last non-keyword token. Acme Widget with both words protected gives [Acme]; I like Acme Widget gives [I, like].

The sibling filters count buffered tokens instead (OpenNLPPOSFilter.java:50, OpenNLPChunkerFilter.java:53: tokenNum >= sentenceTokenAttrs.size()); only the lemmatizer counts lemmas, because only it skips some tokens.

• Who is exposed. Chains built from OpenNLPTokenizer, OpenNLPPOSFilter and OpenNLPLemmatizerFilter (user-supplied models) with a filter that marks single words as keywords in front of the lemmatizer: KeywordMarkerFilter with protected words or a pattern, StemmerOverrideFilter, or a custom filter. The marker may sit before or after the POS filter (both executed). A typical use is keeping brand, product or person names out of lemmatization. SuggestStopFilter (suggest module) also marks a final stopword with no separator after it as a keyword (SuggestStopFilter.java:29-34, :108), so it would trigger the bug in front of the lemmatizer (read in the code, not executed).
• Not exposed. Chains without a keyword filter, and the documented KeywordRepeatFilter chain: it emits each token first as keyword and then as non-keyword (KeywordRepeatFilter.java:44-58), so the last token of a sentence always has a lemma. That chain lost 0 tokens over 404 generated documents. This holds with RemoveDuplicatesTokenFilter after the lemmatizer, as in the upstream tests; placed before it, that filter would remove the non-keyword copies and leave only keyword tokens (read in the code, not executed).
• What triggers it. A keyword token at the end of a sentence. Sentence-final punctuation is an ordinary token, so in prose only sentences that end in protected (or earlier removed) punctuation are hit. The everyday case is a field value or query without a final period, or a value made only of protected words. Each value of a multi-valued field is analyzed on its own.
• Effect. The index does not have the words (a stored copy of the value, if the field is stored, still shows them): term, phrase and all-words queries miss the document. When an inner sentence loses tokens, the later tokens move up (in the probe, they sits at position 4 instead of 6; offsets stay correct), so phrase queries can see words as adjacent that were not. On the query side, the classic QueryParser turns the phrase "running in the evening" into "run in the", which also matches They sent him running in the morning; QueryBuilder's AND query for Acme Widget becomes Acme. With splitOnWhitespace=true the classic parser analyzes each word alone and keeps them. Documents keep the missing terms until reindexed.
• Scale. Ordinary inputs, deterministic, every affected value. No crash, CheckIndex clean, no obvious slowdown in one noisy run (1019 ms vs 996 ms for its control, 20,000 sentences). In a 404-document corpus generated from a small vocabulary, through the keyword-marker chain, 6 documents lost 8 tokens.
• Since. 9.5.0 (#11734); 26 releases, 9.5.0 to 9.12.3 and 10.0.0 to 10.5.2, and main.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[they, send, he, run, in, the, evening]> but was:<[they, send, he, run, in, the]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.opennlp.TestLemmatizerSentenceFinalKeyword.testKeywordAtEndOfSentence(TestLemmatizerSentenceFinalKeyword.java:53)

• Design. The filter leaves keyword tokens out of lemmatization and copies them through unchanged (OpenNLPLemmatizerFilter.java:74-76, :86-90; the field comment at :52 reads "lemmas for non-keyword tokens"). Nothing in the class removes tokens on purpose.
• The note that added the feature (LUCENE-2899, GitHub #3973, migrated JIRA comment by sarowe): "Added KeywordAttribute awareness to OpenNLPLemmatizationFilter, so that lemmatization won't be performed on tokens with isKeyword()==true."
• KeywordAttribute javadoc: keyword-aware streams "can decide to modify a token"; stemmers "conditionally skip a term". Skipping means leaving the token as it is. It is not an explicit promise that every token is kept, but nothing in it suggests dropping keyword tokens.
• No chain rule. The class and factory javadoc name no restriction on the filters before the lemmatizer and never say tokens are removed.
• History. From 7.3.0 to 9.4.2 a sentence ended when its buffered tokens ran out (sentenceTokenAttrsIter == null || !sentenceTokenAttrsIter.hasNext()); 9.4.2 passes the repro. #11734 ("Fix repeating token sentence boundary bug", by kotman12, merged by dweiss in 2022) switched the lemmatizer to the lemma count; its review does not mention keyword tokens at a sentence end.
• Precedent. #11734 fixed GITHUB#11735 and GITHUB#11771, both lost tokens in this filter, both labeled type:bug on GitHub and listed in CHANGES 9.5.0 (under its "Optimizations" heading, lucene/CHANGES.txt:3854; entries :3859, :3862). The #11735 reporter wrote that a dropped last token "is usually not a big deal when that token is punctuation (most of the time) but can become especially problematic when the last bit of text of a stream has no punctuation": the same situation this bug creates for keyword tokens.
• Why the tests miss it. Every keyword test in TestOpenNLPLemmatizerFilterFactory (:230, :252, :274, :301, :323, :359) uses KeywordRepeatFilter, whose last token per sentence is never a keyword, and TestRandomChains skips the filter (@IgnoreRandomChains, OpenNLPLemmatizerFilter.java:45).

The strongest case for "works as designed" (checkers): keyword support here was motivated by the KeywordRepeatFilter pattern (index lemma and original side by side), every upstream keyword test uses it, and the class javadoc promises nothing about keywords, so a standalone marker is an untested, unsupported shape.

Why it fails: untested is not intended. The code reads KeywordAttribute whoever set it, and the design note states a general rule, keyword tokens are not lemmatized. (Granted: no javadoc formally promises one output token per input token; the case rests on the implementation, its history and the design note.) The same chain kept the words from 7.3.0 to 9.4.2, until #11734, whose purpose was sentence boundaries with repeated tokens. A lemmatizer has no reason to delete tokens, and upstream fixed two token-loss bugs in this filter as bugs. "Rare, because sentences end with punctuation" limits reach, not intent: short text without a final period is everyday input. Verdict: not intended, a regression.

One fix was built and validated with gradle in a scratch worktree at a9779ef3c193; the alternative and the workaround are for discussion.

Restores the end-of-sentence test of 7.3.0 to 9.4.2 (+4/-1 in OpenNLPLemmatizerFilter.java, with a two-line comment). Keyword tokens no longer decide when a sentence ends, so every buffered token is emitted.

  @Override
  public final boolean incrementToken() throws IOException {
-    boolean isEndOfCurrentSentence = lemmaNum >= lemmas.length;
+    // A sentence ends when all its buffered tokens have been emitted. Keyword tokens have no lemma,
+    // so counting lemmas would drop the keyword tokens that end a sentence.
+    boolean isEndOfCurrentSentence =
+        sentenceTokenAttrsIter == null || !sentenceTokenAttrsIter.hasNext();
     if (isEndOfCurrentSentence) {

Trade-offs

• Restores the 9.4.2 output for sentence-final keywords; an all-keyword sentence now keeps all its tokens, which no release did (9.4.2 emitted none).
• Existing indexes keep the missing terms, while the fixed query side keeps the protected words (the AND query Acme Widget needs both terms again), so such queries can stop matching old documents until the field is reindexed. Worth a CHANGES note (reasoning, not executed).
• Cost: no obvious slowdown; in one best-of-3 run of 20,000 sentences on a shared host, the fixed chain took 1024 ms and its control 981 ms. One noisy run, not a benchmark.
• The same line is in every release from 9.5.0, so the patch should apply to the maintained branches; only main was built and tested with gradle.

What the checks validated

• Gradle, JDK 25: the new test fails 4 of 6 on stock and passes 6 of 6 with the fix; with the fix, TestOpenNLPLemmatizerFilterFactory (12 upstream tests, including the KeywordRepeatFilter, early-exit and empty-field tests of #11734) and the new test pass, 18 of 18; the whole module passes, 6 suites and 39 tests (33 upstream + 6 new), with assertions on and off; :lucene:analysis:opennlp:check passes (ecjLint, forbiddenApis, google-java-format, tests).
• Standalone runner with the fix: the brief's test 6/6 (assertions on and off) and the finder's 3/3; the verifiers' runs: the independent reproducer's 12 tests plus the 12 upstream lemmatizer tests, 24/24, and the devil's advocate's 3/3.
• The bounded-exhaustive harness then reports no sentence-final-keyword mismatch (0 of 55,987 cases per lemmatizer variant).
• It does not bring back #11735 or #11771: it changes only the end-of-sentence test, keeps #11734's sentence grouping by SentenceAttributeExtractor (9.4.2 used the per-token end-of-sentence flag) and does not restore 9.4.2's early return false for a sentence without non-keyword tokens; the three lemmatizer tests #11734 added pass with it.

Add a tokenNum counter and end the sentence at tokenNum >= sentenceAttributes.size(), as OpenNLPPOSFilter.java:50 and OpenNLPChunkerFilter.java:53 do.

Trade-offs

Same semantics as the recommended fix and makes the three filters look alike; a few more lines (a field and its resets in nextSentence() and clear()).

Use KeywordRepeatFilter, with RemoveDuplicatesTokenFilter after the lemmatizer, instead of a standalone marker; it lost 0 tokens over 404 generated documents. Ending each value with punctuation also avoids the sentence-final case (executed), if the chain keeps the punctuation as an unmarked token, but not every inner sentence.

Trade-offs

Different behaviour: it indexes the lemma next to the original word for every token instead of protecting chosen words. Appending punctuation changes the tokens and does nothing for already indexed documents.

Method: bounded-exhaustive translation validation on the stock main jars (the hunt's "semi-formal" tier), hunt unit "OpenNLP filters versus the OpenNLP library", 2026-10-26.

• The hypothesis came first, from reading OpenNLPLemmatizerFilter while writing the model; the enumeration then confirmed it. So it is not a blind discovery.
• Reference: an independent per-sentence model that calls the OpenNLP ops directly. Inputs: every sentence-index sequence over {0,1,2}^n and every keyword pattern over {0,1}^n for n <= 6, 55,987 cases per lemmatizer variant (dictionary, MaxEnt, both). On main, 9,446 cases per variant match this bug's hypothesis exactly; the other mismatches belong to the sibling bug LFV-621 (a first sentence index other than 0); 0 cases are unexplained. With this fix shadowed, the class disappears; with both fixes, 0 mismatches remain.
• A generated-text differential run: 404 documents generated from a small fixed vocabulary, through OpenNLPTokenizer, with the library called directly as the reference. In the chain POS -> keyword marker -> lemmatizer, 6 documents lost 8 tokens; 0 with this fix and LFV-621's fix both shadowed (the only fixed run of the corpus).
• Then an AI finder wrote the public-API JUnit, and independent AI tools reproduced and judged it (see the checks).