MemoryIndex drops the final position increment of each value of a multi-valued field (and the gaps after leading values with no tokens), so phrase and other position queries disagree with IndexWriter, including Lucene Monitor's single-document matching

Finding LFV-340Area: memory / MemoryIndex term inversion (multi-valued analyzed fields: positions and offsets carried from one value to the next); reached in-tree through monitor single-document batchesPriority: mediumImpact: silent wrong resultsNovelty: new

Finding page for LFV-340

AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.

What it shows

You add a document whose text field has two values, 'quick the' and 'fox', to a MemoryIndex with an analyzer that removes the stop word 'the', and expect the positions of a one-document IndexWriter index: IndexWriter keeps the hole the removed trailing 'the' leaves (TokenStream.end() reports it as a final position increment), so 'fox' is at position 2 and the phrase "quick fox" does not match. MemoryIndex.storeTerms never adds that final increment, puts 'fox' at position 1 and matches the phrase; the reverse also happens (the input of upstream's TestIndexWriter#testStopwordsPosIncHole matches a phrase on IndexWriter but not on MemoryIndex), and values that produce no tokens before the first token skip the position and offset gaps ([the | quick fox] with gap 100: IndexWriter quick at position 101, offsets 4-9; MemoryIndex position 0, offsets 0-5). Lucene Monitor inherits it: a stored phrase query matches the document when it is matched alone (MemoryIndex path) but not when the same document is in a two-document batch (IndexWriter path).

The test shows that a field with values "quick the" and "fox" (StandardAnalyzer with stop word "the") gets the phrase "quick fox" wrong in MemoryIndex: IndexWriter keeps the hole that the removed trailing "the" leaves (the final position increment from TokenStream.end(), so fox is at position 2), and MemoryIndex drops it (fox at 1).

Expected vs actual

Expected the IndexWriter result, 0 hits for PhraseQuery("body", "quick", "fox"). Actual: MemoryIndex gives 1 hit (expected:<0> but was:<1>).

Files

Run it

# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/memory/build/libs/lucene-memory-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.index.memory.TestMemoryIndexValueEndHole

Gradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).

Feedback

Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.

Snapshot of 2026-11-04. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.