SuggestStopFilter forgets the positions of the stop words it removed at the end of the input, so suggesters treat 'one of the ' like 'one '

SuggestStopFilter is the query-side stop filter for autocomplete. Its javadoc defines it as "Like StopFilter except it will not remove the last token if that token was not followed by some token separator" and recommends StopFilter in the index analyzer with this class in the query analyzer of the analyzing suggesters. When the typed text ends in stop words followed by a separator, such as one of the , it removes them like StopFilter, but its end() reports the input's own final position increment, 0, instead of adding the positions it removed; StopFilter reports 2. Terms, position increments, offsets and the final offset are identical. The suggesters turn that final increment into trailing holes of the lookup automaton, so with too few holes the lookup matches more entries: AnalyzingSuggester.lookup("one of the ") returns [one direction, one of the best] where a StopFilter query analyzer returns [one of the best]. AnalyzingSuggester and FuzzySuggester show it with two or more removed trailing stop words, the document suggester and FreeTextSuggester with one. In a bounded probe the result was always the result for the same text without its trailing stop words; in exactFirst mode (AnalyzingSuggester, FuzzySuggester) one entry whose text equals the typed text is dropped (with assertions on, the lookup throws an AssertionError). The end value has been wrong since 4.5.0 (LUCENE-3849); it has changed FreeTextSuggester results since that suggester's first release, 4.6.0, and the other suggesters' since 6.5.0 (LUCENE-7619). With a small fix (+6/-2 lines in SuggestStopFilter) the whole suggest test task has 0 failures.

• Position increment: each token says how many positions it moved on from the previous one. A removed stop word leaves a hole: in go to school, school has increment 2.
• Final position increment: after the last token, TokenStream#end() reports the positions skipped at the end of the input: "any skipped positions (such as those removed by a stopfilter) can be applied to the position increment" (TokenStream.java:152-154). StopFilter does this (FilteringTokenFilter.java:72-75): go to a the ends with final increment 3.
• SuggestStopFilter: "Like StopFilter except it will not remove the last token if that token was not followed by some token separator" (SuggestStopFilter.java:29-37). It keeps an unfinished last word (find the), so that a query a can still suggest apple (LUCENE-5165 CHANGES entry).
• Lookup automaton: AnalyzingSuggester turns the analyzed query into an automaton and matches it as a prefix of the analyzed entries. Each hole becomes an extra separator (AnalyzingSuggester.java:268-307); trailing holes come from the final position increment (TokenStreamToAutomaton.java:212-241).

// incrementToken(): the last token is a stop word and the input has no more tokens
clearAttributes();
input.end();
endState = captureState();                 // :96  the input's end state: final posInc 0
int finalEndOffset = offsetAtt.endOffset();
if (finalEndOffset > endOffset) {
  // OK there was a token separator after the stopword, so it was a stopword
  return false;                            // :102 skippedPositions and this word's posInc are lost
}

// end()                                      :61-71
if (endState == null) { super.end(); } else { restoreState(endState); }

// StopFilter, through FilteringTokenFilter.end()   :72-75
super.end();
posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement() + skippedPositions);

1. For one of the , one is returned. of is skipped (skippedPositions = 1). the is a stop word and the last token, so the filter calls input.end() and captures that state: final increment 0, final offset 11.
2. The final offset is past the end of the (10), so the was finished and is removed. The filter returns false without adding skippedPositions (1) or the's own increment (1). end() restores the captured state: final increment 0. StopFilter gives 2.
3. The index side stores one of the best as one, three separators, best (one separator per position step), and one direction as one, one separator, direction. The StopFilter query one of the is one plus two separators: a prefix of the first entry only. The SuggestStopFilter query is one plus one separator: a prefix of both.
4. AnalyzingSuggester and FuzzySuggester call setFinalOffsetGapAsHole(true) (AnalyzingSuggester.java:317-322): a final increment of 0 becomes 1 hole when the final offset lies past the end of the last token (TokenStreamToAutomaton.java:214-216; here 11 against 3, the gap of the ). That is right for one removed word and too few for two or more. The document suggester builds its automaton without that setting (ConcatenateGraphFilter.java:233-244), so there one removed word already matters.

Mid-stream the class does the bookkeeping: go to a the school gives school increment 4, which the upstream TestSuggestStopFilter.testMultipleStopWords asserts. Only the end-of-input branch drops it.

An exhaustive comparison with StopFilter over 11,718 inputs (every sequence of 0-5 words over go to a the school, 3 endings) found 4,686 differences, all in the final position increment only; 0 with the fix (compared: terms, position increments, offsets, final offset and final position increment).

• Who. Users of SuggestStopFilter in the query analyzer of AnalyzingSuggester, FuzzySuggester, the document suggester (PrefixCompletionQuery, FuzzyCompletionQuery) or FreeTextSuggester, directly or through SuggestStopFilterFactory (SPI name suggestStop). It is opt-in: no Lucene analyzer uses the class by default. The suggester defaults (exactFirst, preserved separators and position increments) are the configuration shown. preservePositionIncrements=false does not hide it but reverses it: the buggy lookup then misses single-token entries that a StopFilter query analyzer matches (AnalyzingSuggester with entries one direction, one of the best, oneplus: [one direction, one of the best] instead of [one direction, one of the best, oneplus]). AnalyzingInfixSuggester is not affected.
• Trigger. Ordinary typing: text that ends in removed stop words followed by a space, such as one of the . AnalyzingSuggester and FuzzySuggester need two or more such words (with one, their own trailing-space hole happens to be right); the document suggester needs one (one the matches like one), and so does FreeTextSuggester, which reads the end increment itself (FreeTextSuggester.java:474-541): one of returns the next-word predictions after one where a StopFilter query analyzer returns none.
• Effect. Extra suggestions. In a bounded probe (6 words, 3 of them stop words, texts of 1 to 4 words: 1,434 entries, 3,108 queries, 777 ending in removed stop words) the result with the bug always equals the result for the same text without its trailing stop words, 0 mismatches. Not every such query changes: 270 of the 777 differ from the fixed filter in AnalyzingSuggester and in FuzzySuggester, 528 (prefix) and 513 (fuzzy) in the document suggester; in FreeTextSuggester 657 of 2,868 queries differ. Contexts and document filters still apply to every hit.
• One entry can be lost. In exactFirst mode, an entry whose text equals the typed text, trailing space included (entry one of the , query one of the ), is dropped with assertions off: the probe's AnalyzingSuggester returns [one of the best, one direction, one the end], where StopFilter and the fix return [one of the , one of the best, one the end]. With assertions on the lookup throws an AssertionError at AnalyzingSuggester.java:808. FuzzySuggester, which inherits that code, does the same.
• Index time (not the documented setup). In an index analyzer the lost increment moves the next value of a multi-valued field one position earlier: foo the | bar with gap 0 puts bar at position 1 instead of 2, so the phrase "foo bar" matches.
• How likely. Plausible: in that setup such a lookup can be broadened, and the extra entries are the suggestions for the shorter text; they go away once a word that is not a stop word is typed. No report was found in the searches performed. No scale is needed.
• Workaround. None that keeps the class's purpose: preservePositionIncrements=false does not help (above) and ignores every hole; a StopFilter query analyzer is right for one of the but removes an unfinished last word (find the), the case SuggestStopFilter exists for. A filter after it cannot repair the value (the count of removed trailing words stays inside incrementToken()), and the class is final (SuggestStopFilter.java:38); an application would need its own copy of the class with the fix.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<2> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.search.suggest.analyzing.TestSuggestStopFilterTrailingHoles.testEndCountsRemovedTrailingStopWords(TestSuggestStopFilterTrailingHoles.java:38)

• One documented exception. "Like StopFilter except it will not remove the last token if that token was not followed by some token separator" (SuggestStopFilter.java:29-34). The LUCENE-5165 CHANGES entry says the same: a stop word "at the very end of the lookup query, and without any trailing token characters, will be preserved" (CHANGES.txt:13275-13280).
• StopFilter reports the positions, and a test pins it. FilteringTokenFilter.end() adds skippedPositions (:72-75); TestStopFilter.testEndStopword asserts final increment 1 for test of (TestStopFilter.java:172-189).
• end() is the channel. TokenStream#end: skipped positions "can be applied to the position increment" (TokenStream.java:152-154). LUCENE-3849: "TokenStreams now set the position increment in end(), so we can handle trailing holes" (CHANGES.txt:13337-13340). Its issue text (#4922) says "subclasses of FilteringTokenFilter (e.g. stopfilter) should do: super.end(); posIncAtt += skippedPositions;"; this class is a plain TokenFilter, but the commit made the same change in the test framework's MockTokenFilter, also a plain TokenFilter. Before the commit the committer wrote "I agree this bug is bad. It can also affects suggesters".
• The class already does it mid-stream (SuggestStopFilter.java:92, :114; testMultipleStopWords asserts +4), and the suggesters read the end increment: FreeTextSuggester since its first release, 4.6.0 (FreeTextSuggester.java:474-541), the analyzing suggesters since LUCENE-7619 (6.5.0).
• No test covers it. testEndIsStopWord (go to ) and testMultipleStopWordsEnd2 (go to a the ) use the assertTokenStreamContents overload that passes finalPosInc = null (TestSuggestStopFilter.java:47-64, 123-140).

Strongest case (an intended-behaviour checker and a reviewer from another model family, OpenAI GPT-5.6; rated moderate): TokenStream#end says skipped positions "can" be applied, not "must". The author of the class rewrote SuggestStopFilter.end() in the LUCENE-3849 commit itself, six days after adding the class, and did not add them. The upstream trailing-stop tests leave the final increment unchecked, and TestAnalyzingSuggester strips trailing holes from its oracle because "TokenStream.end() does not tell us any trailing holes, yet ... there is an issue open for this" (TestAnalyzingSuggester.java:900-902).

Why it fails: the class's only documented difference from StopFilter does not apply to text that ends in a separator. LUCENE-3849's issue text asks FilteringTokenFilter subclasses such as StopFilter to add skippedPositions in end(), and the same commit did so for FilteringTokenFilter and for the test framework's MockTokenFilter, a plain TokenFilter like this class. When it landed, no suggester read the end increment yet: FreeTextSuggester, which does, was added six weeks later (4.6.0), and TokenStreamToAutomaton looked only at the final offset until LUCENE-7619 (6.5.0). The indexer reads it too, but an index analyzer is not where the class is documented to go. No record of a decision about the value was found. The TestAnalyzingSuggester comment dates from 2012, before LUCENE-3849, and concerns that test's own filter: it names a known gap, not a design.

Fix A and fix B were built and run in a scratch worktree at a9779ef3c193 with the whole :lucene:suggest:test task (the two repro classes and the new asserts included); both give StopFilter's end state. Either fix also makes FreeTextSuggester follow StopFilter for such text. Both are behaviour-changing bug fixes: suggestion lists for such text change. The sibling end() bugs in other classes (LFV-354, LFV-542, LFV-532, LFV-488) have their own fix sites.

On the branch where the last stop word is followed by a separator, add skippedPositions + posInc to the input's final increment and only then capture endState; the unfinished-last-word branch captures it unchanged. end() stays as it is. The patch below plus two finalPosInc asserts in the existing TestSuggestStopFilter trailing-stop tests is fix-with-test.diff.

           clearAttributes();
           input.end();
-          endState = captureState();
           int finalEndOffset = offsetAtt.endOffset();
           assert finalEndOffset >= endOffset;
           if (finalEndOffset > endOffset) {
             // OK there was a token separator after the
-            // stopword, so it was a stopword
+            // stopword, so it was a stopword; like StopFilter,
+            // report the removed trailing positions in end()
+            posIncAtt.setPositionIncrement(
+                posIncAtt.getPositionIncrement() + skippedPositions + posInc);
+            endState = captureState();
             return false;
           } else {
+            endState = captureState();
             // No token separator after final token that
             // looked like a stop-word; don't filter it:
             restoreState(sav);

Trade-offs

• Smallest change: the stored end state itself is right, so any consumer that calls end() sees StopFilter's value.
• Changes suggestions only for text that ends in removed stop words and a separator; those now match StopFilter. FreeTextSuggester results for such text change the same way (one of gives [] instead of the bigrams after one).
• With preservePositionIncrements=false the change goes the other way: such a lookup then matches like a StopFilter query analyzer, which also includes single-token entries that start with the kept word (oneplus for one of the ).
• For very long runs of trailing stop words the lookup automaton grows like StopFilter's (40,004 states for 20,000 stop words instead of 6; single runs, not a benchmark); bounding query length is the application's job (SECURITY.md line 38).

What the checks validated

• Gradle on a9779ef3c193, seed DEADBEEF: fix-with-test.diff plus the two repro classes, all of :lucene:suggest:test: 33 classes, 240 tests, 0 failures, 7 skipped (upstream @Ignore and @Nightly), with assertions on and off (an independent fact check repeated both). That is 236 upstream tests (e.g. TestAnalyzingSuggester 23, TestFuzzySuggester 14, TestAnalyzingInfixSuggester 23, TestFreeTextSuggester 11, TestPrefixCompletionQuery 9) plus the 4 repro tests.
• :lucene:suggest:ecjLintMain, ecjLintTest and checkGoogleJavaFormat pass with fix-with-test.diff.
• With the standalone runner (fix as a shadow class): the finder's and the reproducer's tests pass, 46 tests of the four closest classes pass, and the exhaustive differential against StopFilter gives 11,718 of 11,718 equal.

Keep the capture where it is; store skippedPositions + posInc in a field on the separator branch, reset it in reset(), and add it after restoreState(endState) in end() (+5 lines).

   private State endState;
+  // positions of the stop words removed at the end of the input, reported in end() like StopFilter
+  private int trailingSkippedPositions;
   ...
     endState = null;
+    trailingSkippedPositions = 0;
   ...
       restoreState(endState);
+      posIncAtt.setPositionIncrement(posIncAtt.getPositionIncrement() + trailingSkippedPositions);
   ...
             // stopword, so it was a stopword
+            trailingSkippedPositions = skippedPositions + posInc;
             return false;

Trade-offs

Same result as A and reads like FilteringTokenFilter.end(), which adds its count after super.end(); one more field to reset.

What the checks validated

• Gradle on a9779ef3c193, seed DEADBEEF, with the same test asserts and repro classes: all of :lucene:suggest:test passes with assertions on (33 classes, 240 tests, 0 failures, 7 skipped); ecjLintMain, ecjLintTest and checkGoogleJavaFormat pass. With assertions off (run by an independent fact check): 33 classes, 240 tests, 0 failures, 7 skipped.

Pass finalPosInc in the two existing trailing-stop tests of TestSuggestStopFilter: 1 for go to (testEndIsStopWord) and 3 for go to a the (testMultipleStopWordsEnd2), with the assertTokenStreamContents overload that takes it. The finder's JUnit adds the suggester-level case.

Trade-offs

Pins the value at the filter level; the suggester-visible effect needs the repro's lookup test.

What the checks validated

• On stock main the two asserts fail (finalPosInc expected:<1> but was:<0>, expected:<3> but was:<0>; 2 of 6 TestSuggestStopFilter tests); with fix A or fix B they pass.

• Sweep (bounded-exhaustive translation validation of the stock jars). 434 selected component configurations (the token filters registered through SPI with their defaults, except 27 that need parameters or resources, plus parameter grids for the buffering and token-removing filters), behind 3 upstream chains, on 5,208 texts each (6.78 million analyses). The reference for each run is the end() state of the same chain without the component, plus the positions the component removed; it compares terms, position increments and lengths, offsets, and the final position increment and offset, not every attribute. Every suggestStop configuration with a matching stop word list showed final-position mismatches.
• Differential against the documented equivalence. Every sequence of 0-5 words over go to a the school with 3 endings (11,718 inputs) through StopFilter and SuggestStopFilter: 4,686 mismatches, all of one shape (only the final position increment differs; terms, position increments, offsets and the final offset equal; other attributes not compared); 0 with the fix (non-vacuous).
• Reduction. Code reading found the separator branch in incrementToken; the finder wrote the public-API JUnit with the AnalyzingSuggester lookup and ran it on main, 10.5.2 and 9.12.3, and a main() probe on 8.11.4.