ConcatenatingTokenStream loses a source's trailing position increment (for example a removed stop word) when the next source produces no tokens, so later tokens are indexed too close and phrase queries over-match

Finding LFV-488Area: analysis / common miscellaneous (ConcatenatingTokenStream#incrementToken and #end: carry-over of child final position increments)Priority: lowImpact: silent wrong resultsNovelty: new

Finding page for LFV-488

AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.

What it shows

You build one field from several token streams with ConcatenatingTokenStream, for example the values 'alpha the', '' and 'beta', each run through a StopFilter that removes 'the'. You expect the hole left by the removed 'the' to survive, as it does when no empty value sits in between and as IndexWriter keeps it for the same values in a multi-valued field, because LUCENE-8650 (7.7.0) made the class carry each source's final position increment over to the next token. Instead each exhausted source overwrites the pending increment instead of adding to it, and end() reports only the last source's own increment, so a source with no tokens erases the hole: 'beta' is indexed directly after 'alpha', the exact phrase "alpha beta" matches this document but not the single value 'alpha the beta', and with an empty last source end() reports 0 instead of 1.

The test shows that ConcatenatingTokenStream loses the hole of a removed trailing stop word when the next source is empty: ('alpha the' with 'the' stopped, empty, 'beta').

Expected vs actual

Expected position increments [1, 2] (as without the empty source); actual [1, 1].

Files

• src/org/apache/lucene/analysis/miscellaneous/TestConcatenatingEmptySourceHoles.java (JUnit test)

Run it

• Tested on Lucene main e1faeba08bde (2026-11-02): still reproduces.
• Tested on Lucene main 645d4fe14a20 (2026-10-28): reproduced.
• Recorded outcome of run 1 on Lucene main 645d4fe14a20: 1 test(s) run, 1 failed.
• How to read the outcome: a failing test is the finding reproducing (the failure message states expected against actual).
• Seeds: a rerun with the same seed replays the same random choices. Assertions off (-da -dsa -Dtests.asserts=false) is the production-like setting.

# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.analysis.miscellaneous.TestConcatenatingEmptySourceHoles

Gradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).

Feedback

Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.

Snapshot of 2026-11-03. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.