WordDelimiterGraphFilter with IGNORE_KEYWORDS moves a dropped delimiter's hole past the next keyword, so exact phrases across it miss or over-match

WordDelimiterGraphFilter drops a token made only of delimiters, such as the / in lucene / solr rocks under the whitespace tokenization its javadoc recommends, and leaves a hole, so solr gets position increment 2 (WordDelimiterGraphFilter.java:373-376; asserted upstream for LUCENE / SOLR). With the opt-in IGNORE_KEYWORDS flag and solr marked as a keyword, the increments become {1, 1, 2}: the keyword slides back into the hole and the gap reappears between solr and rocks, two words that are adjacent in the text. The keyword early return (:353-355) runs before the line that collects pending holes (:359), so the hole skips the keyword and lands on the next non-keyword token. The protected-word bypass in the same method keeps the hole, and so does the keyword mark without the flag. With the same analyzer at index and query time, exact phrase, span and interval queries across the keyword miss ("solr rocks": 0 of 1) or match across the dropped token ("lucene solr": 2 instead of 0). It arrived with the flag in 7.4.0 (LUCENE-8265) and fails on every release run from 7.4.0 to 10.5.2 and on main. A 3-line fix (4 after formatting) applies the pending hole to the keyword.

• Position increment: how many positions a token moves forward from the previous one. 1 means adjacent; 2 means one position is empty, a hole. Exact phrase queries need adjacent positions.
• Dropped (swallowed) token: WordDelimiterGraphFilter emits nothing for a token made only of delimiters, such as a standalone /, - or & under a whitespace tokenizer, or for a token whose parts the flags do not generate. It leaves a hole instead ("swallow this token, creating a hole", :373).
• Keyword: a token whose KeywordAttribute is set by a marker such as SetKeywordMarkerFilter. With IGNORE_KEYWORDS the filter passes it through without splitting it (:148).

1. accumPosInc (:187-188, "so that we preserve incoming holes") collects the increment of every non-keyword input token (:359). Plain and protected words (:368-369) and word parts (:422-423) take the whole amount and reset it.
2. A dropped token leaves its increment pending: a delimiter-only token continues at :376, and a delimited token with no parts under the flags falls through at :430. That pending amount is the hole.
3. The keyword return at :353-355 comes before :359. The keyword leaves with its own increment (1), and the pending amount goes to the next non-keyword token on top of its own.
4. lucene / solr rocks: lucene +1 (position 0), / dropped (pending 1), solr +1 (position 1, should be 2), rocks 1 + 1 = +2 (position 3). Only the keyword moves, back into the hole.

Same shift for - and &, for a token with no parts under the flags (1-2 a-b solr rocks with number parts only), for two keywords in a row ({1, 1, 1, 2} instead of {1, 2, 1, 1}), for a larger incoming hole (solr +1 and rocks +11 instead of +11 and +1), and for two dropped tokens in a row (lucene / - solr rocks: solr at 1 instead of 3). A trailing keyword (lucene / solr) ends one position earlier than without the mark. Not affected: PRESERVE_ORIGINAL (the / is kept, no hole), a hole from a StopFilter right before the keyword, a keyword before the /, and the deprecated WordDelimiterFilter (it leaves no hole for /).

• Who is exposed. Users of WordDelimiterGraphFilter with IGNORE_KEYWORDS (the Java flag since 7.4.0; the factory parameter ignoreKeywords since 9.9.0, default 0), a keyword-marking filter in front (KeywordMarkerFilter, SetKeywordMarkerFilter, PatternKeywordMarkerFilter, or anything that sets KeywordAttribute), and PRESERVE_ORIGINAL off (the default).
• What it takes. A marked token directly after a token the filter drops. Under the whitespace tokenization the class javadoc recommends, a spaced /, - or & is such a token. The fraction example in LUCENE-8265's description hits it: it wanted fractions marked as keywords "so that 1/2 does not become 12", and 1/2 - 3/4 inch then loses the phrase "3/4 inch".
• What the damage looks like. In the ordinary linear chain, wrong positions in the index and no error. Exact phrase, PhraseQuery, span and interval queries across the spot miss ("solr rocks": 0 of 1) or match across the dropped token ("lucene solr": 2 instead of 0). A phrase that quotes the whole text, / included, still matches, because query analysis shifts the same way. Terms, offsets, and term and boolean queries are unaffected. A phrase slop of 1 hides a one-position shift but not a larger one: with two dropped tokens (lucene / - solr rocks), "solr rocks"~1 still misses and "lucene solr"~1 still matches. In a multi-valued field the pending hole does not carry into the next value, but a phrase across the value boundary sees the shifted positions of the value before it (values lucene / solr and rocks: "lucene solr rocks" matches).
• Scale. Deterministic, per occurrence, any index size, assertions on or off. Each keyword after a dropped token moves back by that token's increment (1, or more if a hole came with it or several tokens were dropped).
• How likely. Plausible rather than common: opt-in, the keyword must directly follow such a token, only exact positional queries across that spot change, and no report was found.
• After a fix. Positions are written at index time, so documents indexed before the fix keep the shifted positions until they are reindexed.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: posIncrement 1 term=solr expected:<2> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:274)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:699)
       org.apache.lucene.analysis.miscellaneous.TestWdgfIgnoreKeywordsHole.testHoleStaysBeforeKeyword(TestWdgfIgnoreKeywordsHole.java:41)

• The contract. Class javadoc WordDelimiterGraphFilter.java:38-40: "producing a correct token graph so that e.g. PhraseQuery can work correctly when this filter is used in the search-time analyzer". :187: accumPosInc is "used for accumulating position increment gaps so that we preserve incoming holes". :373: "word of simply delimiters: swallow this token, creating a hole". :148: the flag "Suppresses processing terms with KeywordAttribute#isKeyword()=true", which speaks of the keyword term, not of a hole that belongs to an earlier token.
• The analysis package rule. package-info.java:303-305: "a filter that filters out tokens must increment the position increment in order not to generate corrupt tokenstream graphs"; its StopFilter example (:307-326) adds the accumulated increment to the next returned token, whatever it is.
• Upstream tests. TestWordDelimiterGraphFilter.java:328-338 ("in this case, works as expected."): LUCENE / SOLR -> {1, 2}; :352-361: LUCENE / NUTCH SOLR -> {1, 2, 1}, a word after the dropped / keeps the hole. (NUTCH is protected but has no delimiter, so it takes the plain-word half of :366 and the protected set is never consulted; a protected word with a delimiter keeps the hole too: lucene / wi-fi rocks with wi-fi protected gives {1, 2, 1}, executed.) testKeywordFilter (:466-481), the only test added with the flag, has no dropped token, so nothing pins the current behaviour.
• History. LUCENE-7619 added the filter to "produce holes properly" (2017). LUCENE-8265 (2018) added the flag; its discussion is about overloading KeywordAttribute, never positions. LUCENE-8395 (#9442, fixed in 7.5.0) fixed a misplaced hole in the same bookkeeping on the PRESERVE_ORIGINAL branch, so a wrongly placed hole in this filter has been treated as a bug before. None of the javadoc, tests, CHANGES entries and recorded discussions that were reviewed says the shifted increments are by design.

The strongest case for "intended" (the cross-model reviews rated it moderate, the devil's-advocate verifier weak): IGNORE_KEYWORDS is a literal pass-through. The javadoc says processing is suppressed, CHANGES says the option lets the filter "skip tokens marked with KeywordAttribute", the deprecated WordDelimiterFilter has the same early return, and returning the keyword with its own increment of 1 is exactly "untouched".

Why it fails: the pending increment belongs to an earlier token that the filter itself removed, and the package rule puts it on the next returned token, here the keyword. The current code does not leave positions alone either: the keyword moves from 2 to 1 and a gap opens between two adjacent words. The protected-word bypass, the static sibling the keyword option was requested as an alternative to (LUCENE-7342), keeps the hole, and so does the same keyword with the flag off. The deprecated WordDelimiterFilter (used instead of this filter) and ProtectedTermFilter around this filter give {1, 1, 1}: they drop the hole consistently and move no gap, so they are no precedent for {1, 1, 2}.

How strong the case is depends on the sub-case: weak for a keyword followed by another token (the main claim); moderate for a trailing keyword, because the filter reports no hole at the end of the stream in any configuration (the separate bug LFV-354), so the claim there is only the lost +1; strong for a stacked synonym in front of the filter, which feeds it a token graph it says it cannot consume, so that chain is left out of the claim.

Two equivalent shapes were built in a scratch worktree of main a9779ef3c193; A is the smallest and was run against the whole module. Workarounds for affected releases follow.

The keyword is still returned without splitting or catenation; it only takes the increment that the filter owes for tokens it dropped just before it, as plain and protected words already do (:368-369).

if (has(IGNORE_KEYWORDS) && keywordAttribute.isKeyword()) {
  // apply any hole left by previously swallowed tokens to this keyword token
  posIncAttribute.setPositionIncrement(
      accumPosInc + posIncAttribute.getPositionIncrement());
  accumPosInc = 0;
  return true;
}

Trade-offs

• No change when no hole is pending: the keyword keeps its own increment.
• With the fix, two dropped tokens in a row and a two-valued field also give the positions of the chain without the mark (executed).
• Behaviour change only for the affected inputs: positions become those of the chain without the mark, so phrases that matched across the dropped token ("lucene solr") stop matching, which is the intent.
• Existing indexes keep the old positions until reindexed; query-side analysis changes at once.
• A stacked keyword (incoming increment 0) right after a dropped token now takes the hole too.
• The same lines exist on every maintained branch, so the patch should apply as is; only main was built.

What the checks validated

• Stock: 32 tests, 4 failed (this doc's 3 and the upstream-suite method), assertions on and off.
• Fix: the same 32 tests pass, assertions on and off; the whole analysis-common module, 340 classes, 1,913 tests, 0 failures, 18 skipped, assertions on and off; check -x test (formatting, lint, forbidden APIs) passes.
• Earlier, with the finder's equivalent fix: the finder's test 3 of 3, the independent reproducer's 7 of 7 and the security probe's 6 of 6 pass.

Move accumPosInc += posIncAttribute.getPositionIncrement(); above the keyword test and, in the keyword branch, set the increment to accumPosInc and reset it, exactly like the protected-word branch.

Trade-offs

Same behaviour as the recommended option; reads as one rule (every returned token takes the pending increment) at the cost of a slightly larger diff. Only the 32 targeted tests were run (assertions on), not the whole module.

List the words as protected words (protWords, factory protected=) instead of keyword-marking them: that path keeps the hole (executed: lucene / wi-fi rocks with wi-fi protected gives {1, 2, 1}). It does not help in addition to the mark: the keyword return comes first, and a word that is both keyword-marked and protected still gives {1, 1, 2} (executed). For a word without a delimiter, simply do not keyword-mark it.

Trade-offs

Only for a fixed word list, not for pattern-based marks such as fractions. Alternatives: PRESERVE_ORIGINAL keeps the delimiter token, so no hole is pending, but it indexes punctuation tokens and changes the output for every split word; a phrase slop of 1 hides a one-position shift (not a larger one) but loses exactness everywhere.

Method: bounded-exhaustive translation validation on the stock main jar (the hunt's "semi-formal" tier). A hunt unit, analysis graph filters, 2026-10-26.

• Oracle: the output for a whitespace-separated text equals the outputs of its tokens analysed one by one, concatenated, with a hole for every token the filter drops.
• Inputs: every text over {a, A, 1, -, space} up to length 7 (99,998,720 checks) and over {a, 1, -, ', s, space} up to length 6 (57,329,664 checks), each with all 1,024 flag combinations and with keyword and protected-word sets.
• Result: 47,044 and 72,588 mid-stream position mismatches (category M1), every one with IGNORE_KEYWORDS and a keyword token. The split-category rerun printed no line for the same mismatch without a keyword, and the harness prints only categories it has counted, so that count was 0. The same runs reported 1,764,524 and 3,247,808 trailing-hole mismatches (category M3), many without a keyword: that is the separate bug LFV-354. Offsets were consistent.
• An AI finder then read incrementToken for the root cause and minimised the signal by hand into a public-API JUnit; independent AI tools reproduced and judged it (see the checks). No model checker, SMT solver or prover found or verified this bug.