WordDelimiterGraphFilter loses the hole of a dropped last token, so the next value of a field starts one position early

WordDelimiterGraphFilter drops a token made only of delimiters, such as --, and leaves a hole for it, like a stop word removed by StopFilter: foo -- bar gives bar a position increment of 2, which upstream's tests assert for LUCENE / SOLR. It keeps the dropped token's increment in a counter, accumPosInc, that only the next emitted token applies, and it has no end() override. So when the dropped token is the last one, the hole is lost: foo -- ends with a final position increment of 0 instead of 1. The same happens to a stop-word hole such a token carries (foo the -- behind a StopFilter: 0 instead of 2) and to a last token whose parts the flags all switch off (foo 12-34 with GENERATE_WORD_PARTS only).

It shows wherever the final increment is used. In a multi-valued field with Lucene's default position increment gap of 0, the next value starts one position early: PhraseQuery(foo, bar) matches the values foo -- | bar while the StopFilter values foo the | bar do not, and the phrase QueryBuilder builds for foo -- bar matches the single value foo -- bar but not the two values. A ShingleFilter after the filter loses its trailing filler shingle. It has been there since the filter was added in 6.5.0 (LUCENE-7619, 2017). A small end() override, the same shape as FilteringTokenFilter.end(), fixes it; with it, the new tests and all 1,914 tests of the analysis module pass.

• Position increment: how many positions a token moves forward from the previous one; 1 for the next word, 2 when one word between them was removed (a hole).
• Final position increment: the increment a token stream reports in end(), after its last token. It carries holes left at the end of a value. The indexer adds it to the field's position before the next value of the same field.
• Swallowed token: an input token for which WordDelimiterGraphFilter emits nothing: a token made only of delimiters (--, /, &) with PRESERVE_ORIGINAL off, the default, or a token whose parts the flags all switch off (12-34 with only GENERATE_WORD_PARTS).

1. incrementToken() adds every input token's increment to accumPosInc (WordDelimiterGraphFilter.java:359), "so that we preserve incoming holes" (:187-188); only an IGNORE_KEYWORDS keyword returns before that line, unchanged (:353-355, the sibling bug).
2. A token made only of delimiters is dropped with continue (:373-380, "swallow this token, creating a hole"); a token whose parts are all switched off buffers nothing (:384, :387) and is dropped the same way.
3. Only a later emitted token applies accumPosInc (:368-369, :422-423); reset() clears it (:434-442).
4. The class has no end() override. The inherited TokenFilter.end() only calls input.end(), so the final increment is the input's and the pending accumPosInc is never reported.
5. IndexingChain adds the final increment to the field position after end() (IndexingChain.java:2105-2109) and then the analyzer's gap before the next value (:2121-2123; Analyzer.getPositionIncrementGap returns 0 by default). So the next value starts one position early per lost hole: one per trailing dropped token, plus each stop-word hole they carry.

WDGF is WordDelimiterGraphFilter with the factory's default flags. The filter's own terms, offsets, token counts and positions inside a value do not change; only the final increment does. When the filter is last in the query analyzer, queries do not change either: QueryBuilder reads only per-token increments. A later filter that reads the final increment, such as ShingleFilter (ShingleFilter.java:379-382), turns it into tokens, so there the terms differ too (foo _), at index and at query time. A value that ends in a stop word alone keeps its hole, because the inherited end() passes the StopFilter's value through.

• Who is exposed. Applications that index a multi-valued text field through WordDelimiterGraphFilter (directly, through the wordDelimiterGraph factory, or through a search server's word-delimiter-graph filter built on it) with a small position increment gap, and run phrase, span or interval queries that can span two values. Also chains with a ShingleFilter after the filter (the trailing filler shingle is lost). Suggesters that keep trailing holes see a different final increment too (by reading; not executed).
• Default vs opt-in. The filter is an optional analysis-common filter; once added, its default flags trigger the bug. With PRESERVE_ORIGINAL on, a punctuation-only token is kept, not dropped.
• What it takes. A value whose last token the filter drops: a spaced punctuation-only token (--, /, &) or a token whose parts the flags all switch off, optionally after removed stop words; and a following value of the same field, or a later consumer of the final increment.
• What happens. The next value starts one position early per lost hole (one per trailing dropped token, plus carried stop-word holes; no cap). Phrase and proximity queries across the boundary match where the StopFilter chain does not ("foo bar" on foo -- | bar) or miss where the same text in one value matches ("foo ? bar"). Positions inside one value do not change; in a single-valued field only a later consumer of the final increment sees a difference: a ShingleFilter loses foo _, and TokenStreamToAutomaton turns a positive final increment into a trailing hole (TokenStreamToAutomaton.java:214-232, by reading).
• Large gaps hide it. Common search servers default text fields to a gap of 100; then only phrases whose slop is close to the gap see it. Over slops 0 to 250, on the values foo -- | bar against the StopFilter chain, PhraseQuery(foo, bar) differs only at slop 100 (100-101 with two lost holes), and the phrase QueryBuilder builds for foo -- bar (body:"foo ? bar") only at slop 99.
• Scale and versions. Deterministic and independent of index size; no error, CheckIndex clean. Every release from 6.5.0 to 10.6.0 (10.6.0 by reading) and main.
• How likely. Rare: the wrong final increment is common, but users see it only across value boundaries with a small gap, and no report was found in the recorded searches.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: finalPosInc expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:436)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:544)
       org.apache.lucene.analysis.miscellaneous.TestWordDelimiterGraphFilterTrailingHole.testTrailingPunctuationOnlyTokenLeavesHole(TestWordDelimiterGraphFilterTrailingHole.java:34)

• The package docs say "must". "End of Input Cleanup": "The components of the token stream (the tokenizer and the token filters) must put accurate values into the token attributes to reflect the situation at the end of the field" (package-info.java:274-284). TokenStream#end: "any skipped positions (such as those removed by a stopfilter) can be applied to the position increment" (TokenStream.java:152-154).
• LUCENE-3849 fixed exactly this symptom as a bug (4.5.0, Bug Fixes: "TokenStreams now set the position increment in end(), so we can handle trailing holes"). Its report was a phrase across the values of a multi-valued field with gap 0, after a value ending in removed stop words.
• The same omission was fixed as a bug in four later changes: LUCENE-8360 (NGram filters, 7.4.0), LUCENE-8376/8371 (ConditionalTokenFilter, FixedShingleFilter, 7.5.0), LUCENE-8650 (ConcatenatingTokenStream, 7.7.0) and #16577 (FixedShingleFilter, 10.5.2; reported for a wrong mid-stream increment), whose fix also adds an end() of the same shape: a pending counter added to the final increment.
• The filter makes this hole on purpose everywhere else. Upstream asserts LUCENE / SOLR -> +1, +2 (TestWordDelimiterGraphFilter.java:328-338, with other flags; the drop does not depend on them), that an incoming increment of 10 on a dropped / "is carried over" (:392-414), and that a stop word before a split word "should add a gap here" (:451-461). A leading -- foo gives foo +2. The code says "preserve incoming holes" (WordDelimiterGraphFilter.java:187) and "swallow this token, creating a hole" (:373).
• The history says holes were the point. LUCENE-7619, which added the filter: "I'm also trying to produce holes properly (removes logic from the current WDF that swallows a hole when whole token is just delimiters)". The deprecated filter's code asks for "proper hole adjustment (FilteringTokenFilter-like)" (WordDelimiterFilter.java:264-265).
• Nothing pins the current output. No upstream test asserts the filter's final increment, and it is excluded from the random-chain tests (@IgnoreRandomChains, WordDelimiterGraphFilter.java:87). With the fix, the 28 upstream tests of the filter and the 16 of the deprecated one pass, and all 1,914 tests of the analysis module pass.
• The weakest of the three triggers is a token whose parts the flags all switch off: no upstream comment or test says it must leave a hole. The support is the filter's own behaviour: in the middle, foo 12-34 bar (word parts only) gives bar +2, so the filter already treats it as a hole and drops it only at the end. The cross-model fact check rated the intended case moderate for this trigger and weak for the other two.

The checkers built the strongest case for "intended": (a) TokenStream#end says skipped positions "can be applied", which reads as optional; (b) no upstream test checks the filter's final increment; (c) the deprecated WordDelimiterFilter cancels the increment of a plain punctuation-only token, so "no hole" is the older behaviour; (d) the class javadoc says the filter "cannot consume an input token graph correctly", so a StopFilter in front could be called unsupported; (e) at gap 0 Lucene already lets a phrase run across values (foo | bar matches "foo bar"), so one position more or less is cosmetic.

Why it fails: (a) the package docs say "must", and the project fixed missing final increments as bugs in LUCENE-3849 and four later changes (the last, #16577, added the same kind of end() while fixing a mid-stream increment). (b) Untested is not specified; nothing asserts a final 0. (c) This filter was written to replace exactly that logic (LUCENE-7619), and it makes the hole in the middle and at the start, so dropping it only at the end is inconsistent with the class itself. (d) A hole is not a graph (position length stays 1), and upstream's own test puts a StopFilter in front of the filter and asserts the stop word's gap. (e) The issue is not that values touch, but that the same text gets different positions depending on where a value ends: the phrase foo -- bar matches it as one value and misses it as two, while StopFilter is consistent. Every checker rejected the case; the cross-model reviewer rated it weak, the independent reproducer weak to moderate.

Option A was applied in a scratch worktree of main a9779ef3c193 and run through gradle with assertions on and off. Option B (with the IGNORE_KEYWORDS sibling fix) was run through the repro harness with shadowed classes; option C was not run.

Add end() to WordDelimiterGraphFilter, next to reset() (WordDelimiterGraphFilter.java:434): call super.end(), then add accumPosInc to the position increment. The finder's and the reproducer's fix sketches are the same method.

What it changes, by code. accumPosInc is non-zero at end() when the last input tokens emitted nothing, and in one more case: a dropped token followed only by IGNORE_KEYWORDS keywords, which are emitted without touching the counter (:353-355, the sibling bug). Every other emitted token resets it to 0 (:369, :378, :423). So output changes for values that end in dropped tokens, and in that keyword case: with solr a keyword, lucene / solr ends with 1 instead of 0 under this fix alone (executed by the fact check); option B fixes it.

New output. foo -- and foo 12-34 (word parts only) end with 1, foo the -- with 2; the values foo -- | bar put bar at position 2, so the phrase foo -- bar finds the document and "foo bar" does not; a ShingleFilter after the filter emits foo _ for foo --, as for foo the behind a StopFilter.

End-composition controls (a print-only probe, not in the 5-test class; stock and with the fix shadowed): no double count. foo -- the behind a StopFilter ends with 2 (stock 1); foo -- -- and -- -- with 2 (stock 0); a value of only -- with 1 (stock 0), like StopFilter's 1 for the; a -- whose incoming increment is 0 adds nothing (final 0 with and without the fix).

  @Override
  public void end() throws IOException {
    super.end();
    // report the holes of swallowed trailing tokens (and the incoming holes they carried)
    posIncAttribute.setPositionIncrement(posIncAttribute.getPositionIncrement() + accumPosInc);
  }

Trade-offs

• The next value's positions move by the lost holes, so documents indexed before the fix keep the old positions until reindexed; query parsing does not change when the filter is last in the query analyzer (with a ShingleFilter after it, query-side filler shingles change the same way as index-side ones, by reading).
• On its own it leaves the IGNORE_KEYWORDS sibling bug: a hole pending in front of a last keyword token is reported at the end instead of being lost (executed by the fact check); option B puts it on the keyword.

What the checks validated

• The 5 new tests pass; upstream TestWordDelimiterGraphFilter (28) and TestWordDelimiterFilter (16) pass; assertions on and off (stock: the 4 defect tests fail).
• All of :lucene:analysis:common:test with assertions on: 340 classes, 1,914 tests, 18 skipped, 0 failures.
• :lucene:analysis:common:check -x test passes (google-java-format, ecjLint, forbiddenApis, licenses).
• Through the repro harness with the class shadowed: the new test 5 of 5 (assertions on and off), the finder's repro 4 of 4, the independent reproducer's suite 9 of 9, the devil's advocate's phrase test 3 of 3; earlier runs: upstream 28 of 28, the security probe 8 of 8, and the same override on 6.5.0 and 10.5.2 4 of 4.

The sibling bug's recommended fix applies the pending accumPosInc to an IGNORE_KEYWORDS keyword before its early return (WordDelimiterGraphFilter.java:353-355). Both patches apply cleanly together. A composition probe (lucene / solr with solr a keyword must give solr +2 and a final 0; lucene solr / a final 1; lucene / solr / both) fails 3 of 3 on stock, 2 of 3 with either fix alone, and passes 3 of 3 with both; with both, the new test (5/5), the sibling's test (3/3), upstream TestWordDelimiterGraphFilter (28/28) and the three older repros pass.

Trade-offs

None beyond the two fixes' own; it is the only option that puts every dropped token's hole where StopFilter would put it. One issue with two fixes, or two cross-referenced issues.

What the checks validated

Through the repro harness with both fixes shadowed, assertions on; not run through gradle or the whole module together.

The deprecated filter also keeps an accumPosInc (WordDelimiterFilter.java:183, :247) and has no end(). A verifier observed that it drops a stop-word hole carried by a last dropped token (foo the -- final 0, while foo the -- bar gives bar +3), and a probe shows it also drops the hole of a last token whose parts the flags all switch off (foo 12-34 with word parts only: final 0, while foo 12-34 bar gives bar +2). The same override would report 2 and 1 there and leave a plain trailing -- at 0, matching that filter's own behaviour in the middle (by reading). It needs its own validation: the deprecated filter has the same IGNORE_KEYWORDS early return before the accumulation (WordDelimiterFilter.java:241-243), so an end()-only patch can move a pending hole there too.

Trade-offs

The filter is deprecated, and open PR #202 proposes removing it; not run.

• Method (semi-formal): bounded-exhaustive translation validation of the stock main jar, in the hunt's unit on analysis graph filters. Every whitespace-separated text up to length 7 over a, A, 1, - and space, and up to length 6 over a, 1, -, ', s and space, with keyword-marker and protected-word variants, times all 1,024 flag combinations: 99,998,720 and 57,329,664 checks.
• Oracle, a hand-written rule that encodes the intended trailing-hole invariant (not a formal specification), independent of the code: the final position increment must equal the sum of the increments of the trailing input tokens that produced no output.
• Signal. 1,764,524 and 3,247,808 violations of that rule; the only other category in those runs was the IGNORE_KEYWORDS sibling bug.
• Root cause by reading the class: no end() override, so the pending accumPosInc is dropped. The finder then wrote the public-API JUnit and the fix sketch; independent checkers followed (see "Independent checks").