ShingleFilter with minShingleSize >= 3 ends the stream early after a hole near the end, so the last word(s) are never indexed

ShingleFilter builds shingles (word n-grams) and, by default, also outputs every input word as a unigram. A removed stop word leaves a hole, which the filter fills with the token _. With minShingleSize 3 or more, a hole followed by only 1 to minShingleSize-2 words at the end of a value ends the stream early: incrementToken() skips the filler-only unigram, finds too few entries for a shingle and returns false while real words are still in its window. "a cup of tea" is indexed without "tea", "the cat" without any token, and term queries for those words miss the document. Nothing throws. It has been in every release since 3.1.0 (LUCENE-2400, 2010); minShingleSize 2, the default, is not affected.

• Shingle: a word n-gram, for example cup _ tea. minShingleSize and maxShingleSize bound its length (both default 2). Unigram: a single input word, output too when outputUnigrams is true (the default).
• Hole and filler: a position with no token, for example where StopFilter removed "of". ShingleFilter puts a filler token _ there so shingles keep their spacing; LUCENE-2400 (2010) made it skip shingles and unigrams that are only fillers.
• Window: the filter keeps up to maxShingleSize entries (words and fillers) and builds one unigram or shingle from the head of the window per call.

1. incrementToken() makes one attempt per call and returns whether that attempt produced a token (ShingleFilter.java:262-315, return tokenAvailable at :314).
2. When the window starts with a filler, the filler-only unigram is skipped by jumping to the next gram size, minShingleSize (:285-288).
3. Near the end of the value the window can hold fewer than minShingleSize entries (words and fillers both count). Then nothing is built either (:293), and the call returns false, which TokenStream documents as "end of stream" (TokenStream.java:138). The consumer stops, although the words after the filler are still in the window.
4. Before the input is exhausted the window is full, so a filler-led window still yields a shingle and the next call outputs the word; with minShingleSize 2 the jump lands on a 2-entry shingle such as _ tea, which fits. That is why only minShingleSize >= 3 and only the end of a value are affected.

Stock main a9779ef3c193, WhitespaceTokenizer + StopFilter {a, of, the} + ShingleFilter, words as term@position:

Controls give every word: the same "a cup of tea" with min 2, "cup tea" (no hole), "war of the worlds of" (with min 3, a trailing stop word adds a trailing filler and brings worlds@3 back; with min 4 one trailing stop word is not always enough: "one of two the" still loses two) and "please divide the sentence into shingles" (a hole mid-value). After indexing "a cup of tea", TermQuery(title:tea) counts 0 and title:cup 1.

• Entry points. new ShingleFilter(ts, min, max), ShingleFilterFactory (SPI name shingle, argument minShingleSize) and ShingleAnalyzerWrapper(analyzer, min, max); executed with new ShingleAnalyzerWrapper(new EnglishAnalyzer(), 3, 3): "the lord of the rings" has no unigram ring. Downstream shingle filters that pass a minimum shingle size through inherit it.
• Opt-in. minShingleSize defaults to 2, which is not affected. Within the opt-in nothing else is needed: outputUnigrams defaults to true. Unigrams off with outputUnigramsIfNoShingles on is hit too.
• No scale, no crafted input. One short ordinary value, every time: "a cup of tea", "war of the worlds", "the cat". At most minShingleSize-2 words per value are lost; with min 3 a two-word value such as "the cat" is not findable through that field at all.
• Silent. No exception, with assertions on or off; CheckIndex is clean; the stored text still shows the words. Term, match and phrase-of-unigram queries for them miss the document, and indexes built before a fix keep the gap until reindexed.
• Not affected. FixedShingleFilter (it never outputs unigrams, FixedShingleFilter.java:34), minShingleSize 2, holes in the middle of a value, and values whose last hole is followed by at least minShingleSize-1 words.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[cat]> but was:<[]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.shingle.TestShingleFilterHoleBeforeLastWord.testUnigramAfterRemovedStopWord(TestShingleFilterHoleBeforeLastWord.java:43)

• Javadoc. setOutputUnigramsIfNoShingles (ShingleFilter.java:200-201, and ShingleAnalyzerWrapper.java:70-71): "Note that if outputUnigrams==true, then unigrams are always output, regardless of whether any shingles are available." setMinShingleSize (:228): "The unigram output option is independent of the min shingle size." setOutputUnigrams (:184-185): the output contains "the input tokens (unigrams) as well as shingles". The class javadoc (:40-41) says the filter "handles position increments > 1 by inserting filler tokens".
• outputUnigramsIfNoShingles. A narrower promise: unigrams "for those times when no shingles are available (because there are fewer than minShingleSize tokens in the input stream)" (ShingleFilter.java:196-197). "the cat" is such a value: shiftInputWindow() switches to unigrams for it (:439-444), and the call then ends the stream before outputting one.
• End of stream. TokenStream.incrementToken() returns "false for end of stream" (TokenStream.java:138); ShingleFilter returns false with unread input tokens in its window.
• Tests. testTriGramFilterMinTriGram (min 3, max 3) asserts the trailing unigrams into and shingles where no trigram fits; every upstream test with holes uses minShingleSize 2, and the min 3 and 4 tests have no holes, so no upstream test asserts the combination.
• History. LUCENE-2400 (#3475, 2010) changed the filter to "don't output all-filler shingles and unigrams". Precedents in the same package treat missing output at the end of a stream as a bug: LUCENE-5180 (#6244, trailing holes) and LUCENE-8719 (#9765, FixedShingleFilter missed output at the end of a stream, the same "too short, so stop" pattern).
• Legal trigger. Validated settings (minShingleSize 3, maxShingleSize 3), an ordinary StopFilter, public constructors.

Strongest case for "intended": LUCENE-2400 deliberately skips filler-led unigrams, and near the end a filler-led window shorter than minShingleSize has nothing left to output, so the tail could be called "no more n-grams", an old best-effort limit of fillers combined with minShingleSize >= 3.

Why it fails: the skipped token is the filler, not the word after it; the javadoc promises unigrams "always" and "independent of the min shingle size"; the same word is output with min 2, mid-value, and (in the min 3 example) when a trailing stop word follows it; "the cat" loses every token; no test, javadoc, issue or CHANGES entry allows dropping real words. The devil's advocate itself, the independent reproducer and the adjudication concluded real; the cross-model reviewer's intended-behaviour lens said real and rated the "intended" case weak, and its refute-lens review said "not real" only because it could not run code.

Option A was run through gradle in a scratch worktree at a9779ef3c193; the others are untested alternatives or workarounds.

shingle-skip-empty-position.diff (the finder's sketch with reworded comments): the old body of incrementToken() becomes a private nextGram(), and incrementToken() loops while a position yields no token and the window still has entries, resetting gramSize so the next attempt shifts the window. The first token after skipped positions gets them added to its position increment, so tea lands at position 3, as with minShingleSize 2.

int skippedPositions = 0;
while (true) {
  if (nextGram()) { // the old incrementToken() body
    if (skippedPositions > 0) {
      posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement() + skippedPositions);
    }
    return true;
  }
  if (inputWindow.isEmpty()) {
    return false;
  }
  gramSize.reset(); // no token at this position: move on to the next one
  skippedPositions++;
}

Trade-offs

• Output changed only where words were lost in the bounded exhaustive check and the test suites (observed, not proved); end() still reports the input's final position increment. No token starts at the skipped filler position, which maintainers may want to discuss; the test pins the positions.
• Terminates: each iteration resets gramSize, so the next attempt shifts the window, and the window empties once the input is exhausted. Cost: at most a window's worth of extra attempts at the end of a value; a 1.75-million-character timing probe ran at 0.85x the time of the min 2 control (one loaded host). No benchmark beyond that.
• Indexes built before the fix keep the missing terms until reindexed. The open refactor PR #13390 touches the same file and may conflict textually.

What the checks validated

• The new test: 5 of 5 pass (4 of 5 fail on stock), assertions on and off.
• The shingle package with the finder's 4 tests added: 6 classes, 93 tests (84 upstream), 0 failures, assertions on and off.
• The whole analysis-common module: 341 classes, 1,918 tests (1,909 upstream + 9 added), 0 failures, 18 skipped, assertions on and off; :lucene:analysis:common:check -x test passes.
• TestRandomChains (a separate module, :lucene:analysis.tests), run by the fact check with the fix: 2 tests pass, seed DEADBEEF, assertions on. One seed, and a random chain need not include ShingleFilter: a single data point, not coverage.
• The independent reproducer's 8-test JUnit passes 8 of 8 with the fix (5 of 8 fail on stock); in the hunt, the exhaustive check gave 0 mismatches with this change (14,884 without), against the finder's own reference.

Inside the existing method, when the head is a filler and the window has fewer than minShingleSize entries, shift the window and try again, keeping the position increment.

Trade-offs

Narrower, but it duplicates the shift logic; the recommended loop already covers this case and the outputUnigramsIfNoShingles case. Not run.

Say in the javadoc that trailing words after a hole may be dropped when minShingleSize >= 3.

Trade-offs

Contradicts the existing promise that unigrams are always output, independent of the min shingle size, and leaves values like "the cat" unsearchable.

Use minShingleSize 2 (every word is output; executed as the control), or index the unigrams in a separate field whose analyzer has no ShingleFilter and query both fields.

Trade-offs

Each application has to know about it. The separate-field workaround was not run.

Method: bounded-exhaustive translation validation (a hunt unit, analysis pattern, shingle and compound filters, 2026-10-26, main ea57360912ed). No model checker, SMT solver or prover was involved.

• The finder (an AI tool) wrote a reference built on ShingleFilter's documented unigram and filler behaviour (javadoc and option docs), with the other attribute details (filler cap, position lengths, offsets, types, end() state) taken from the implementation.
• Its harness (ShingleTV.java) ran the real filter against it on every token stream of 0-5 tokens with position increments 1-3, final position increments 0-2, minShingleSize 2-4, maxShingleSize min..5, both unigram options, 2 separators and 2 fillers: 157,248 checks, 14,884 mismatches, every one with minShingleSize >= 3, a hole and a missing trailing unigram; 0 with the fix. The check is bounded to those sizes: strong testing, not a proof. A registry re-run reproduced the log byte for byte; it uses the same reference, so it is not independent. The same log lists 3,237 differences between FixedShingleFilter and ShingleFilter, a separate question and not part of this report.
• Then a public-API JUnit, a cross-model review, independent verifiers, an adjudication, a 69-release sweep with a commit bracket, a prior-report search and a security screen (see Independent checks).