NGramTokenFilter and EdgeNGramTokenFilter end() throw away the input's final position increment, so a trailing stop-word hole is lost

TokenStream#end is where a token stream reports the positions it skipped after its last token. StopFilter reports the removed last word of abc the as a final position increment of 1, and IndexWriter adds that number to the field position before the next value of a multi-valued field. NGramTokenFilter and EdgeNGramTokenFilter call super.end(), which leaves the input's value in the attribute, and then replace it with the increments of only the tokens they dropped themselves (shorter than minGram). So behind either filter abc the ends with 0 instead of 1, and abc x the with 1 instead of 2. With Lucene's default position increment gap of 0, the second value def of ["abc the", "def"] is indexed at position 1 instead of 2: the phrase abc def matches (1 hit, against 0 for the same chain without the n-gram filter) and the phrase that keeps the hole misses. Terms, scores, single-valued fields and the first value of the field are unchanged; a gap of 100 hides it from ordinary exact phrases. It came with LUCENE-8360 in 7.4.0, which fixed the opposite case; adding instead of replacing, as FilteringTokenFilter.end() and FixedShingleFilter.end() do, fixes it.

• Position increment: how many positions a token moves on from the previous one. A removed word leaves a hole: StopFilter gives the next token an increment of 2 instead of 1.
• Final position increment: holes after the last token have no next token to carry them, so the stream reports them in end() ("any skipped positions (such as those removed by a stopfilter) can be applied to the position increment", TokenStream.java:152-154). StopFilter reports the removed last word of abc the as 1.
• Multi-valued field: several values of one field in one document. IndexWriter adds each value's final increment, then the analyzer's position increment gap (0 by default), before the next value (IndexingChain.java:2105-2109, :2121-2122; Analyzer.java:300-311).

1. Both filters count, in curPosIncr, the increments of input tokens they drop themselves (shorter than minGram, without preserveOriginal), hand the sum to the next emitted token (a gram, or the preserved original) and reset it (NGramTokenFilter.java:108, :113-114, :132-133; EdgeNGramTokenFilter.java:100, :104-105, :117-118).
2. end() calls super.end(), which runs the input's end() and leaves the input's final increment in the attribute, and then posIncrAtt.setPositionIncrement(curPosIncr) (NGramTokenFilter.java:157-161, EdgeNGramTokenFilter.java:146-150). That replaces the input's hole with the filter's own drops.
3. The two counts are disjoint (curPosIncr only sees tokens the filter received), so the right value is their sum, as in FilteringTokenFilter.end() (FilteringTokenFilter.java:72-75).
4. Only end() is wrong: grams, their order, offsets, per-token increments, position lengths, types and keyword flags match a reference on all 24,106,264 generated inputs of the hunt's exhaustive harness, and a hole in the middle of the stream (abc the def) is kept.

Other readers of the final increment, by code reading only: TokenStreamToAutomaton turns it into trailing-hole states (TokenStreamToAutomaton.java:212-239), as used by ConcatenateGraphFilter for completion fields (with its default preservePositionIncrements=true); AnalyzingSuggester sets setFinalOffsetGapAsHole(true) (AnalyzingSuggester.java:320), which restores one lost trailing hole from the offset gap. Queries built by QueryBuilder (the query parsers' path) do not read the final increment, so those are unchanged; a suggester's lookup runs TokenStreamToAutomaton on the query text too (AnalyzingSuggester.java:885-890).

• Who is exposed. Chains with a filter that can remove the last tokens of a value (StopFilter, or another FilteringTokenFilter such as LengthFilter, CodepointCountFilter, TypeTokenFilter or KeepWordFilter) before NGramTokenFilter or EdgeNGramTokenFilter, built with the constructors, the nGram / edgeNGram factories or CustomAnalyzer, on a multi-valued field that indexes positions.
• What changes. Each later value of the field in the same document starts early by the trailing holes lost in the earlier values: normally one position per removed trailing word (precisely, the summed increments of the removed tokens, FilteringTokenFilter.java:58), and the shift adds up over successive values. Phrase, span and interval queries across that boundary match or miss differently from the same chain without the n-gram filter, in both directions (executed for phrases). At gap 0, phrases across a value boundary match by design (Analyzer.java:300-303), so the change is which cross-value phrases match, not whether any do.
• What does not change (executed on 300 random multi-valued documents, both filters, gaps 0 and 100): terms, frequencies, norms and BM25 term-query scores; single-valued fields and the first value of a multi-valued field (only later values move; a hole lost at the end of the last value has no effect); CheckIndex is clean.
• The gap. Lucene's Analyzer default is 0. The factories' own javadoc example and common search servers use 100 for text fields, which hides it from ordinary exact phrases (executed at gap 100 for the adjacent and the one-hole phrase). def is still at 101 instead of 102, so a phrase with explicit positions across the gap, or a sloppy, span or interval query that reaches that far, could still differ (not executed).
• Default vs opt-in. The n-gram filters are optional analysis-common components; once configured, ordinary text that ends in a stop word triggers it.
• Scale and versions. Deterministic, any index size, no error. Every release from 7.4.0 to 10.5.2 and main.
• How likely. Rare in visible results: n-gram fields are usually term-matched for autocomplete, and no report since 2018 was found in the recorded searches.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: final position increment after end() expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.analysis.ngram.TestNGramTokenFilterFinalPositionIncrement.testEndKeepsInputFinalPositionIncrement(TestNGramTokenFilterFinalPositionIncrement.java:36)

• The docs. "End of Input Cleanup": the components "must put accurate values into the token attributes to reflect the situation at the end of the field" (package-info.java:274-284). TokenStream#end: "any skipped positions (such as those removed by a stopfilter) can be applied to the position increment" (TokenStream.java:152-154). LUCENE-3849 added end() increments for exactly this: "TokenStreams now set the position increment in end(), so we can handle trailing holes" (CHANGES.txt:13337-13340).
• The class javadoc. NGramTokenFilter "emits all n-grams for the same token at the same position" (NGramTokenFilter.java:30). With 3-character tokens and grams of 3 it emits exactly its input, yet it changes indexed positions.
• Sibling filters add. FilteringTokenFilter.end() (:74) and FixedShingleFilter.end() (FixedShingleFilter.java:176-179, fixed this way by #16577, merged 2026-08) add to the input's value; per a verifier's grep, every other end() override in core and analysis-common that calls setPositionIncrement keeps or adds to it.
• The same filters keep mid-stream holes: abc the def keeps def two positions after abc. Only the trailing hole is lost.
• History and tests. LUCENE-8360 ("did not correctly set position increments in end()", CHANGES.txt:7498-7499) wrote the assignment to fix the opposite case, the filter's own dropped trailing tokens. Its tests (TestNGramTokenFilter.java:214-222, TestEdgeNGramTokenFilter.java:281-289) feed a whitespace tokenizer whose final increment is 0, so they cannot tell "set" from "add"; the issue's comments never mention the input's increment. With the fix all module tests pass (1,912, 18 skipped).

The checkers built the strongest case for "intended": (a) TokenStream#end only says skipped positions "can be" applied, which is permissive, and no javadoc states that the sum of increments must be preserved; (b) LUCENE-8360 wrote the assignment on purpose and its CHANGES entry calls the result correct; (c) in its motivating caller, ConditionalTokenFilter, the n-gram filter's input is a wrapper whose end() calls endAttributes(), so the input's increment there is always 0 and overwrite equals add; (d) n-gram fields are mostly single-valued and term-matched.

Why it fails: (a) the package docs say "must", and LUCENE-3849 exists to carry trailing holes; (b) the 8360 tests cannot distinguish the two behaviours and nothing in that issue discusses discarding the input's value; (c) explains why the change looked right there, not why a stop-word hole from a plain chain should vanish; the fix behaves identically in that caller; (d) is about severity, not intent. The open LUCENE-8361 position-preservation check never names the n-gram filters: its saved patch checks every chain, and only its discussion raises exceptions, for filters that summarise the whole stream and for truncating filters. Every checker rejected the case; the cross-model reviews rated it weak.

Option A was applied in a scratch worktree of main a9779ef3c193 and run through gradle with assertions on and off; option B is complementary and was not run.

In NGramTokenFilter.end() (NGramTokenFilter.java:160) and EdgeNGramTokenFilter.end() (EdgeNGramTokenFilter.java:149), replace setPositionIncrement(curPosIncr) by adding curPosIncr to the value super.end() left in the attribute.

Why it cannot double count. curPosIncr only accumulates increments of tokens that input.incrementToken() returned; the input's final increment exists only after input.end(), inside super.end().

What it changes. Only the final increment, and only when the input reports one. With an input whose final increment is 0 (a plain tokenizer, or ConditionalTokenFilter's wrapper) the result is the same as today; the LUCENE-8360 case (seventeen one two three four, grams 8..8) still gives 4.

   @Override
   public void end() throws IOException {
     super.end();
-    posIncrAtt.setPositionIncrement(curPosIncr);
+    // keep the input's final increment (e.g. a trailing stop-word hole) and add the positions of
+    // the tokens this filter dropped after the last emitted token
+    posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement() + curPosIncr);
   }

Trade-offs

• Indexes built before the fix keep the old positions for later values until reindexed, so phrase results across value boundaries change after an upgrade and reindex; that is the fix.
• No API change; negligible cost: one attribute read and one addition per end() call.

What the checks validated

• The 3 new tests pass, with the 88 upstream ngram.* and TestConditionalTokenFilter tests: 91 of 91, assertions on and off (stock: the 3 new tests fail).
• All of :lucene:analysis:common:test with assertions on: 340 classes, 1,912 tests, 18 skipped, 0 failures (the upstream testEndPositionIncrement tests, the LUCENE-8360 case, included).
• :lucene:analysis:common:check -x test passes (google-java-format, ecjLint, forbiddenApis, licenses).
• Through the repro harness: the new test 3 of 3 with the fix shadowed on a9779ef3c193; earlier on ea57360912ed the finder's repro 4 of 4, the reproducer's suite 6 of 6, the devil's advocate's probe 3 of 3, and 59 upstream n-gram and ConditionalTokenFilter tests; the LUCENE-8360 control still gives 4.

LUCENE-8361 (#9408, open since 2018) proposes that TestRandomChains check that the sum of position increments, the final one included, is preserved through a chain. Its saved patch checks every chain, with no exemption list; its discussion raises exceptions for filters that summarise the whole stream and for truncating filters. By reading its patch, it would probably flag this bug; whether it also flags the sibling bugs in ConditionalTokenFilter, ConcatenatingTokenStream and WordDelimiterGraphFilter was not checked.

Trade-offs

Larger scope: any exemptions must be agreed, and it finds bugs rather than fixing them; not a substitute for option A.

Say in the class javadocs that the filters reset the final increment to their own drops.

Trade-offs

Contradicts the package docs' "must", TokenStream#end and the sibling filters that add, and keeps the wrong positions in multi-valued fields.

• Method (semi-formal): bounded-exhaustive translation validation of the stock main jar, in the hunt's unit on analysis filters against their javadoc. References written from the class javadocs and the TokenStream#end contract were compared token by token, and on the end-of-stream state, with the real NGramTokenFilter and EdgeNGramTokenFilter.
• Inputs. 12,053,132 generated inputs per filter: every term of up to 4 code points over 5 code points (including a supplementary character and a combining mark), and every sequence of up to 3 tokens over 7 terms with position increments 0/1/2, position lengths 1/2, the keyword flag, a source final increment of 0 or 1 and several gram parameter sets. These are synthetic streams; some, such as an empty term or a first token with increment 0, no real chain produces.
• Signal. 6,026,566 mismatches per filter, all in the final increment and exactly the inputs whose source reported 1; grams, order, offsets, increments, position lengths, types and keyword flags all matched. The harness killed three planted n-gram mutants: two that change the generated grams (by its exhaustive suite) and one that skips the curPosIncr reset (by its separate reuse check).
• Root cause by reading the two end() overrides; the finder then wrote the public-API JUnit and the fix, and independent checkers followed (see "Independent checks").