MemoryIndex drops the final position increment between the values of a multi-valued field, so phrase queries disagree with IndexWriter

When a document's analyzed text field has several values, IndexWriter starts each value after the previous value's final position increment, the one TokenStream#end() reports; a stop filter puts the positions of the words it removed at the end of a value there. MemoryIndex.storeTerms never reads it: it starts the next value at "last token position + gap". So after a value that ends in a removed word, every later value sits lower than in a one-document IndexWriter index of the same document, and phrase, sloppy-phrase and span queries that cross the value boundary give different answers, in both directions: [quick the | fox] matches "quick fox" only on MemoryIndex, and the input of upstream's testStopwordsPosIncHole matches "just ? test" only on IndexWriter. A second, smaller inconsistency: MemoryIndex skips the position and offset gaps after leading values that produce no tokens. Lucene Monitor matches a single document through MemoryIndex and a batch through IndexWriter, so the same stored phrase query matches a document alone but not in a batch. A divergence since 4.5.0 (the gap part since 4.1.0); default settings, and one document is enough. A fix that carries IndexingChain's accounting into storeTerms passes the repro and the memory, monitor and highlighter suites.

• Multi-valued field: a document with several fields of the same name, for example two TextField("body", ...). Each value is analyzed on its own; positions and offsets continue from one value to the next.
• Position hole: a stop filter removes the from quick the fox but keeps its position, so fox is at 2, not 1. A phrase "quick fox" therefore does not match. When the removed word is the last word of a value, the filter reports the hole when the value ends, as the final position increment of TokenStream#end().
• Gaps: Analyzer#getPositionIncrementGap (default 0) and #getOffsetGap (default 1) are added between values. At gap 0 the values behave like one long value (the analysis package docs: "treat all these sections as one large section").
• MemoryIndex: a one-document, in-memory index, documented as a substitute for a RAM-resident Directory. Lucene Monitor stores queries and matches incoming documents; it indexes a single document into a MemoryIndex and a batch into an IndexWriter.

1. A FilteringTokenFilter (StopFilter, LengthFilter, ...) adds the positions of tokens removed after the last kept one to the position increment it reports in end() (FilteringTokenFilter.java:72-75; TokenStream#end javadoc, TokenStream.java:152-154).
2. IndexWriter uses it: after stream.end() (IndexingChain.java:2105) it adds that increment and the final offset (:2109-2110), then the gaps for every analyzed value, with or without tokens (:2121-2124).
3. MemoryIndex.storeTerms does not: it starts a value at lastPosition + positionIncrementGap and lastOffset + offsetGap, and only once the field has produced a token (info.numTokens > 0, MemoryIndex.java:994-999); after stream.end() it records the last kept token's position (:1045-1049). The increment end() reported is never read.
4. After a value that ends in removed words, later values are lower by the size of that hole; after each leading value with no tokens, lower by the gap plus its hole, and their offsets by that value's final offset plus the offset gap. Positions stay in order, so only distances across a boundary and absolute positions change. Term queries and Boolean queries made only of term clauses gave the same hit counts in 108,000 random checks per run (executed); a Boolean query with a phrase clause inherits the phrase difference.
5. Monitor: DocumentBatch.of sends one document to a MemoryIndex and two or more to an IndexWriter (DocumentBatch.java:55-63, :66-93, :101-114), so its results depend on the batch size.

With a large gap such as 100 most exact-phrase results do not change, because the shift is only the size of the hole; sloppy phrases with slop near the gap and absolute-position queries such as SpanFirstQuery still can. An empty value in the middle of a field is handled the same on both sides.

• Who. Users of MemoryIndex.fromDocument, repeated addField calls or Lucene Monitor's single-document match, with multi-valued analyzed text fields and an analyzer that removes tokens: any stop filter (the stock EnglishAnalyzer has one by default; StandardAnalyzer only when given a stop set), a LengthFilter, or a custom FilteringTokenFilter. The gap part needs only an empty or all-stop-word leading value and a positive gap.
• Effect. Silent wrong match decisions for position-sensitive queries across a value boundary, in both directions (a false positive and a false negative, both executed), plus wrong positions, offsets and term-vector positions read from the MemoryIndex reader. No exception or crash in any run, and nothing is persisted; the effect stays within the document. Default scores are not affected; a custom Similarity that reads FieldInvertState#getPosition() could score differently, because MemoryIndex passes the last kept token's position (MemoryIndex.java:1380-1393) and IndexWriter the position after the last end() and gap (read in the code, not run).
• Monitor. A document's result depends on whether it is matched alone (MemoryIndex) or in a batch (IndexWriter): 1 match alone, 0 in a batch of two. Monitor documents no such difference.
• Default vs opt-in. Default MemoryIndex settings, the default Analyzer gaps (0 and 1) and stock analysis components; no opt-in feature.
• Limits. With a large gap such as 100 most exact-phrase results are unchanged, because the shift is only the size of the hole; sloppy phrases with slop near the gap and absolute-position queries still can differ (executed: "alpha omega"~100 on alpha the + omega). Offsets differ only after leading tokenless values. No report was found in the 13 years since 4.5.0.
• Scale. Needs no scale: one document triggers it, deterministically, every time the shape occurs.
• Not affected in-tree: the highlighters build their MemoryIndex from one combined stream per field; the UnifiedHighlighter's MultiValueTokenStream adds end()'s increment and the gap itself (read in the code).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: hits of "quick fox", IndexWriter vs MemoryIndex expected:<0> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.index.memory.TestMemoryIndexValueEndHole.testTrailingStopWordHoleBetweenValues(TestMemoryIndexValueEndHole.java:56)

• The contract. TokenStream#end (TokenStream.java:152-154): "any skipped positions (such as those removed by a stopfilter) can be applied to the position increment". LUCENE-3849 (#4922), a 4.5.0 Bug Fixes entry ("TokenStreams now set the position increment in end(), so we can handle trailing holes", CHANGES.txt:13337), made IndexWriter use it for exactly this multi-valued case; in that issue a committer wrote: “The "bug" is if a field instance ends with a stopword, that accumulated 'hole' is lost.” TestIndexWriter#testStopwordsPosIncHole (TestIndexWriter.java:1899-1927) asserts it, and its input fails on MemoryIndex.
• MemoryIndex is documented and tested as an IndexWriter substitute. Class javadoc: "a replacement/substitute for RAM-resident Directory implementations" (MemoryIndex.java:106); addField: "Equivalent to adding a tokenized, indexed ... Field" (:683-684, :697-698, :713-714), with gaps "if fields with the same name are added more than once" (:722-724). TestMemoryIndexAgainstDirectory "Verifies that Lucene MemoryIndex and RAM-resident Directory have the same behavior" (:96-97) and already compares a two-valued field's term vectors with IndexWriter (:713-753), for values without holes. With the precedents this sets a strong expectation of equal query results, not a formal promise of identical postings.
• Precedents. LUCENE-4880 ("Fix MemoryIndex to consume empty terms ... consistent with IndexWriter") and LUCENE-5444 ("offsets were corrupted if multiple fields with the same name were added", the same multi-value code) fixed earlier differences.
• Self-disagreement. At gap 0, quick the fox as one value puts fox at 2, split into quick the + fox at 1; the quick puts quick at 1, the + quick at 0 (executed). The analysis package docs say values at the default gap are treated "as one large section".
• History: an omission. Multi-value support came with LUCENE-4515 (4.1.0), nine months before LUCENE-3849, which changed 30 files and not lucene/memory. Executed: both sides agree on 4.1.0-4.4.0 and differ from 4.5.0.
• Legal trigger. Public API, a stock StopFilter, the default gap; IndexWriter accepts the same document.
• Monitor swaps MemoryIndex in for IndexWriter purely by batch size, for speed ("MemoryIndex benchmarks as better performing than RAMDirectory for this case", DocumentBatch.java:101-102).

The strongest case for "intended": Analyzer#getPositionIncrementGap is "Invoked before indexing a IndexableField instance if terms have already been added to that field" and #getOffsetGap "is only called if the field produced at least one token for indexing" (Analyzer.java:297-323). MemoryIndex's numTokens > 0 guard matches that text word for word, so the leading-tokenless part could be called documented; and "replacement/substitute" describes a purpose, not identical postings. Weaker points a reviewer may raise: TokenStream#end says skipped positions "can be applied", not that every consumer must apply them; the addField(String, TokenStream, int, int) overload lets the caller choose the gaps; MemoryIndex already differs from IndexWriter in accepted ways (it keeps freqs and positions and returns term vectors for every indexed field, TestMemoryIndex#testKeywordWithoutTokenStream, #testFieldsOnlyReturnsIndexedFields); positions that have held since 4.5.0 would change; and at a gap of 100 the practical effect is small.

Why it fails: those javadocs say nothing about end()'s increment, so they do not cover the headline at all. For the gap part they are stale: IndexWriter has applied the gaps after empty values since LUCENE-2529 / LUCENE-2668 (3.1.0, CHANGES.txt:17578), and TestIndexWriter#testPositionIncrementGapEmptyField (:768-800) expects position 100 after an empty first value at gap 100. The guard descends from the comment "ensure infos.numTokens > 0 invariant; needed for correct operation of terms()" in MemoryIndex's original 2005 contribution (8f9e2a15e78); LUCENE-4515 left it as unchanged context and only made its new carry branch depend on it, and LUCENE-7091 kept it so that doc-values-only additions add no gap: no change was a decision about gaps. On the weaker points: IndexWriter has applied end()'s increment since LUCENE-3849 and its test asserts it, so "can" is not an opt-out for an index documented as a substitute for it; caller-chosen gaps do not help, because end()'s increment is ignored whatever gap is passed; the accepted differences add information IndexWriter does not keep rather than changing positions it does keep; the behaviour change belongs in a CHANGES entry. Verdict: not intended; high confidence for the headline, medium for the gap part. File the gap part as a secondary consistency fix and correct the two Analyzer javadocs in the same change.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle; the alternatives were not run. Fix the two stale Analyzer javadocs in the same change.

Workarounds today (not run): in Monitor, match a single document in a batch with a dummy second document (the IndexWriter path); or, at gap 0, join a field's values into one value before handing it to MemoryIndex (one value keeps the hole, executed in the self-consistency check).

Info gets nextValuePosition = -1 and nextValueOffset = 0. A value starts there; after stream.end() the next start is pos + posIncrAttribute.getPositionIncrement() and offset + offsetAtt.endOffset(), plus positionIncrementGap and offsetGap when the value is tokenized, whether or not it produced tokens. applyGaps is true on the addField(String, String, Analyzer) and addField(String, TokenStream, int, int) paths and field.fieldType().tokenized() on addField(IndexableField, Analyzer). With an analyzer that is IndexingChain's condition tokenized() && analyzer != null (IndexingChain.java:1972). With no analyzer it is not: IndexingChain then adds no gaps, while MemoryIndex already uses the Analyzer default gaps 0 and 1 (MemoryIndex.java:631-635) and the fix keeps applying them, so fromDocument(doc, null) follows an IndexWriter whose analyzer keeps the default gaps, not one configured with no analyzer. Executed on pre-analyzed values (offsets of the last token): [empty | a] no-analyzer IndexWriter 0-1, default-gap IndexWriter 1-2, stock MemoryIndex 0-1, with the fix 1-2; [a | b] 1-2, 2-3, 2-3, 2-3; [a | empty | b] 1-2, 3-4, 3-4, 3-4. Stock matches the default-gap IndexWriter in 2 of 3 cases, the fix in 3 of 3; the hunt variant matches the no-analyzer one in 3 of 3 but moves offsets stock gives today. Say in the PR which reference it follows. Info.lastOffset is removed: nothing reads it any more, and ecjLint rejects the unused field (it failed the first validation round). lastPosition stays (norms and the keyword-term path read it). Doc-values-only additions never reach storeTerms, so they add no gap (the LUCENE-7091 point). A second diff adds both tests; a PR also needs a CHANGES entry.

int pos = info.nextValuePosition;   // was: -1, or lastPosition + gap if numTokens > 0
int offset = info.nextValueOffset;  // was: 0, or lastOffset + offsetGap if numTokens > 0
// ... tokens as before; then, after stream.end():
int next = pos + posIncrAttribute.getPositionIncrement();
int nextOffset = offset + offsetAtt.endOffset();
if (applyGaps) {
  next += positionIncrementGap;
  nextOffset += offsetGap;
}
info.nextValuePosition = next;
info.nextValueOffset = nextOffset;

Trade-offs

• One more int per field (three instead of two); no extra work per token. No benchmark was run.
• Behaviour change by design: later values of multi-valued fields move to IndexWriter's positions, so stored Monitor or percolator phrase rules can match differently after an upgrade.
• The keyword-term path (storeTerm, fake positions from lastPosition++) is untouched, but it does not update the new carry: a token-stream value after a binary-term value (reachable only through a custom IndexableField whose tokenStream() returns null) now starts from the carried value position (or -1), not from lastPosition + gap (read in the code, not tested).
• Norms still get the last kept token's position (unchanged); only a custom Similarity that reads FieldInvertState#getPosition() would notice.

What the checks validated

• Stock main a9779ef3c193: the 4 repro tests fail with assertions on and off and in every one of 10 iterations (33 of 33 memory, 10 of 10 Monitor executions); the whole memory (57 tests) and monitor (182) suites fail only on them.
• With the fix: the repro passes with assertions on and off and over 10 iterations (seed C0FFEE34); lucene/memory 57 tests, lucene/monitor 182 (1 nightly test skipped, as on stock) and lucene/highlighter 722, 0 failures.
• ecjLint (memory, monitor), checkGoogleJavaFormat and :lucene:memory:renderJavadoc pass, tidy changes nothing, and both diffs apply cleanly to a9779ef3c193. The first round, which kept Info.lastOffset, passed every test but failed ecjLint ("The value of the field MemoryIndex.Info.lastOffset is not used").
• Not run: other modules' suites; the analyzer-less pre-analyzed path beyond the 3-case probe above; a field that mixes binary-term and token-stream values; custom Similarity implementations; no benchmark.

getPositionIncrementGap: replace "Invoked before indexing a IndexableField instance if terms have already been added to that field" with wording about the value, for example "Returns the position increment gap added between two consecutive instances of the field, also when the earlier instance produced no tokens." Do not say when it is called: MemoryIndex reads both gaps before it consumes a value (MemoryIndex.java:516-521, :627-634), IndexWriter after (IndexingChain.java:2121-2124). getOffsetGap: drop "This method is only called if the field produced at least one token for indexing."

Trade-offs

Documentation only; describes what IndexWriter has done since LUCENE-2529 / LUCENE-2668 (3.1.0) and what the recommended fix makes MemoryIndex do.

Add posIncrAttribute.getPositionIncrement() after end() to the carried position (and the final offset to the carried offset), and keep the guard.

Trade-offs

Smaller and keeps the Analyzer javadoc literally true, but leaves the + quick fox different from IndexWriter (101 vs 0 at gap 100); if done by changing lastPosition, that field no longer is the last token's position (it feeds FieldInvertState for norms).

The same change except on the addField(IndexableField, null) path, where it applies no gaps at all, like an IndexWriter with no analyzer, and so drops the offset gap 1 that stock MemoryIndex puts between pre-analyzed values (executed: [a | b] gives b offsets 1-2 instead of 2-3).

Trade-offs

Copies IndexingChain's condition literally, but changes the analyzer-less path, which no in-tree test covers, and keeps the now-unused lastOffset field, which ecjLint rejects (seen in this doc's first validation round, which kept it too).

What the checks validated

On ea57360912ed, shadowing MemoryIndex: the hunt's repro 3 of 3 pass, the upstream TestMemoryIndex, TestMemoryIndexAgainstDirectory and TestSlicedIntBlockPool 51 of 51 pass, the independent reproducer's 8 tests pass, and in the exhaustive scopes (180,544 cases) every position, offset, term-vector, query-match and query-score mismatch is gone, while two unrelated classes stay at their stock counts (OFFSET_MINUS1, offsets other than -1 for fields indexed without offsets: 36,032 in T2 and 5,232 in SIM; 216 IW_UNEXPECTED_THROW); the random scope (3,000 documents) reports no mismatch.

A caveat in the MemoryIndex and Monitor javadocs.

Trade-offs

Leaves wrong positions and Monitor's single-vs-batch disagreement in place.

Method: bounded-exhaustive translation validation on the real jar (semi-formal tier), in a hunt unit on MemoryIndex equivalence.

• Scope: every pair of canned token streams of up to 2 tokens over {a, b}, with position increments {0, 1, 2} and a final position increment {0, 1}, times 3 index options, norms on or off, offsets on or off, and position gap {0, 3}: 177,504 two-value cases plus 40 single-value cases (177,544), each indexed into a one-document IndexWriter index and a MemoryIndex and compared through the public LeafReader and a fixed query set.
• Result on main: 56,448 position, 832 offset, 41,664 term-vector position, 19,232 query-match and 26,928 query-score mismatches; 0 of each with the fix shadowed in (two unrelated classes stay unchanged). A random baseline with real analyzers (3,000 documents) found 2,553 position, 1,430 offset and 39 query-match mismatches, 0 with the fix.
• Then public-API JUnit tests on the stock jar, each asserting the IndexWriter value first. Found 2026-10-26 on main ea57360912ed; then checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6) and an adjudication.