ConcatenatingTokenStream loses a source's trailing hole (a removed stop word) when the next source produces no tokens

ConcatenatingTokenStream joins several token streams into one field. Since LUCENE-8650 (7.7.0) it carries each source's final position increment, for example the hole left by a stop word that a StopFilter removed at the end of a value, over to the first token of the next source. It stores that increment with an assignment for every finished source (ConcatenatingTokenStream.java:92), while the same loop adds offsets (:94), and end() reads only the last source (:115). So a source that emits no tokens, such as an empty value or one made only of stop words, overwrites the pending hole, and an empty last source makes end() report 0. With the values "alpha the", "" and "beta", beta is indexed at position 1 instead of 2 (the multi-valued field and the concatenation without the empty value give 2), and the exact phrase "alpha beta" gets 1 hit instead of 0. Nothing throws and CheckIndex is clean. Reach is narrow: the class has no factory and no in-tree caller, and no downstream use was found. The fix is to add the increments up in a pending counter (+8/-4 lines), validated with the module's whole test suite.

• Position increment: how many positions a token moves on from the previous one. A removed stop word leaves a hole: the next token gets increment 2 instead of 1.
• Final position increment: what a stream reports from end() after its last token, for a hole at its very end ("alpha the" with the removed reports 1).
• Source: one of the streams ConcatenatingTokenStream joins. A source with no tokens is an empty value, a whitespace-only value, or one made only of removed words.

// ConcatenatingTokenStream.java:89-97 at a9779ef3c193, incrementToken(): skip finished sources
while (sources[currentSource].incrementToken() == false) {
  if (currentSource >= sources.length - 1) return false;                              // :90
  sources[currentSource].end();
  initialPositionIncrement = sourceIncrements[currentSource].getPositionIncrement();   // :92  '=' overwrites
  OffsetAttribute att = sourceOffsets[currentSource];
  if (att != null) offsetIncrement += att.endOffset();                                 // :94  offsets use '+='
  currentSource++;
  newSource = true;
}
// :103-106 the stored value is added to the first token of the next source that has one
// :115 end(): int finalPosInc = sourceIncrements[currentSource].getPositionIncrement();  (last source only)

1. alpha comes out with increment 1 (position 0). Source 0 ends; its end() reports 1 (the removed the), stored at :92.
2. Source 1 ("") has no token; its end() reports 0, and :92 replaces the pending 1 with 0. Its offset, 0, is added to the running offset 9.
3. Source 2 emits beta with 1 + 0 = 1: position 1 instead of 2.

In a run of finished sources, the sum of the final increments of all of them except the last is lost: two empty values in a row lose the same hole, and foo+hole followed by two sources with no tokens and final increments 2 and 1 gives bar 2 instead of 5. If the last source is empty, incrementToken() returns at :90 and end() reads only that source, so end() reports 0. An empty first value ("", final increment 0) is handled correctly; a first value made only of stop words follows the same rule (concat("the", "the", "bar") gives bar increment 2, the single text 3). The assignment and a pending counter that starts at 0 and is cleared once applied agree whenever every source has a token, which is why the LUCENE-8650 test passes.

The expected column of the two index rows was run as a multi-valued field (position gap 0). For the stream rows it is what the indexing chain would give for the same values (read from IndexingChain.java:2105-2109) and what the fix gives (executed); the first row is also what the class gives today.

• Who is hit. Code that joins several values into one field with this public analysis-common class (not marked experimental or expert), or uses it in its own analysis or query code (not run), when a value that ends in a removed token is followed by one or more values with no tokens. Such values are ordinary (an empty subtitle, a value made only of stop words), but the class is rarely used.
• Reach. No analysis factory or SPI name, so CustomAnalyzer and schema-driven setups cannot use it; no caller outside its own file and test; read-only code search of default branches found no use in Elasticsearch, OpenSearch or Solr. No prior report was found in the documented searches.
• What goes wrong. Every token after the empty value moves closer by the lost hole. An exact phrase across a removed stop word matches ("alpha beta": 1 hit instead of 0), and a phrase that needs the hole stops matching (QueryBuilder turns foo the bar into "foo ? bar", which then misses the document). Only PhraseQuery was run; spans and intervals read the same positions.
• Empty last value. end() reports 0 instead of the hole. When such a stream is one value of a multi-valued field, the next value starts closer; the position gap between values is kept (a value ending in three removed words, "x the the the", then an empty source, then a second value, with gap 100: 101 instead of 104).
• Silent and stored. No exception and CheckIndex clean; on main with assertions on and off; the diagram probe indexed with the default codec, the gradle tests with the test framework's randomized codec. The wrong positions are in the index, so documents indexed before a fix keep them until reindexed.
• Size. The tokens after a run of values with no tokens move closer by the final increments of every finished source in the run except the last: with one such value, the removed trailing tokens of the value before it; with several in a row, more. Positions only move closer and stay in order. No scale needed: one document, three values.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: posIncrement 1 term=beta expected:<2> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:274)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:699)
       org.apache.lucene.analysis.miscellaneous.TestConcatenatingEmptySourceHoles.testEmptySourceKeepsPendingHole(TestConcatenatingEmptySourceHoles.java:34)

• CHANGES, Lucene 7.7.0 (CHANGES.txt:6928-6930): "LUCENE-8650: ConcatenatingTokenStream did not correctly clear its state in reset(), and was not propagating final position increments from its child streams correctly." In #9696 the committer wrote that the patch "also handles final position increments (to allow for things like stop words being removed at the end of a tokenstream)". A stop-word-only value is exactly that case.
• The upstream test testOffsetGaps (TestConcatenatingTokenStream.java:63-83) expects a child's final increment 2 to be added to the next child's first token: posIncs {1, 1, 3, 1} (:76). Both children have tokens, and its end() value 2 (:79) is the last child's own, so neither half of this bug is tested. testBasic has put an EmptyTokenStream between children since the class was added (:44), so empty sources are legal input.
• The indexing chain (IndexingChain.java:2105-2109) calls end() and adds the final increment for every value, including a value with no tokens, then the position gap (:2122). Executed: the multi-valued field "alpha the", "", "beta" puts beta at 2; the concatenation of the same streams at 1.
• The analysis docs: TokenStream.end() says skipped positions "(such as those removed by a stopfilter) can be applied to the position increment" (TokenStream.java:152-154); the package docs say the components "must put accurate values into the token attributes to reflect the situation at the end of the field" (analysis/package-info.java:276-281). An empty last source makes end() report 0 although a hole is pending.
• History: LUCENE-2529 and LUCENE-2668 (2010) fixed, as bugs, empty or stop-word-only values of multi-valued fields that shifted later positions or offsets in the indexing chain; LUCENE-3849 moved trailing holes into end() so that multi-valued fields keep them.

"Positions across an empty source are unspecified." The class javadoc promises "as if all the inputs were from a single source" only for offsets, end() says "can", not "must", and no test combines an empty source with a pending hole. It fails because LUCENE-8650 made carrying child final increments the stated and tested behaviour, the same loop adds offsets but overwrites positions, and no comment, test or discussion drops the increments of empty sources on purpose.

"Phrases across sources are loose anyway: the class adds no position gap." No gap is a separate, deliberate choice. The defect is that the same tokens get different positions depending on whether an empty value sits between them: concat("alpha the", "beta") puts beta at 2, concat("alpha the", "", "beta") at 1. An empty source should change nothing.

"What a stop-word-only middle value should add is a design question." Possibly; summing its own hole gives the positions of the multi-valued field and of the single text ([1, 3] for foo the + the + bar), which is the natural reading. The case is strongest for an empty value (final increment 0), which should change nothing, and for the empty last value's end(); it is weakest for how several stop-word-only values combine, which no document spells out. Verdict of every verifier and the adjudication: not intended (an opinion; the contracts are above).

Option A was validated through gradle; B is the minimal-looking variant a reviewer might suggest, C an alternative a maintainer might raise, D a workaround for users today.

Rename initialPositionIncrement to pendingPositionIncrement (starting at 0), add to it instead of assigning, clear it once it is applied to a token, add what is left in end(), and clear it in reset(). The regression test: the four methods of the repro, added to TestConcatenatingTokenStream.

-      initialPositionIncrement = sourceIncrements[currentSource].getPositionIncrement();
+      pendingPositionIncrement += sourceIncrements[currentSource].getPositionIncrement();
 ...
-      posIncAtt.setPositionIncrement(posInc + initialPositionIncrement);
+      posIncAtt.setPositionIncrement(posInc + pendingPositionIncrement);
+      pendingPositionIncrement = 0;
 ...
-    int finalPosInc = sourceIncrements[currentSource].getPositionIncrement();
+    int finalPosInc =
+        pendingPositionIncrement + sourceIncrements[currentSource].getPositionIncrement();
 ...
+    pendingPositionIncrement = 0;   // in reset()

Trade-offs

• Positions change only for inputs that hit the bug (a pending hole followed by a source with no tokens, or an empty last source); when every source has a token, the assignment and the pending counter give the same values, and the upstream tests pass unchanged.
• No extra work per token. Documents indexed before the fix keep their wrong positions until reindexed; the CHANGES entry should say so and reference LUCENE-8650.
• A stop-word-only middle value adds its own hole too ([1, 3]), matching a multi-valued field; a maintainer may want that spelled out in the javadoc.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25.0.3, seed DEADBEEF: :lucene:analysis:common:test with the fix and the three repro classes added passes with assertions on and off: 342 classes, 1,923 tests, 0 failures, 18 skipped. That is the module's 1,909 upstream tests (among them TestConcatenatingTokenStream 4 of 4) plus the 14 repro tests, which fail 10 of 14 on stock main.
• Static checks: :lucene:analysis:common:check -x test with the fix and the doc's test passes (checkGoogleJavaFormat, ecjLint, forbiddenApis, checkLicenses; error-prone is skipped in this build).
• Earlier, with the standalone runner: the finder's version of the same fix passes the finder's 4, the independent reproducer's 6 and the security probe's 6 tests; the reproducer wrote the same fix independently.
• The fact check's probe also ran fix A on two sources with no tokens in a row, an empty first value, a stop-word-only middle value, an empty last value and reuse after reset(): every value matches the multi-valued field. An all-empty stream and early or repeated end() calls were not run.

The one-character change a reviewer may suggest first. The field starts at 1 (line 48) and reset() never clears it, so a bare += also breaks inputs where every source has a token.

Trade-offs

• Executed (a main() probe on the stock classes with each variant): += alone gives the control concat(foo+hole, bar) [1, 3] instead of [1, 2], the testOffsetGaps input {1, 1, 4, 1} instead of {1, 1, 3, 1} (the upstream test would fail), and three plain one-token sources [1, 2, 2].
• Adding the clear once applied and the rest in end() still gives the control [1, 3] and testOffsetGaps {1, 1, 4, 1}, gives end() 2 instead of 1 for an empty last source, and leaks state across reset() (the next document's x, y get [1, 3]).
• Only with all four changes (start at 0, clear once applied, add the rest in end(), clear in reset()) is every case right, and that is option A.

A javadoc note that positions are not carried across a source with no tokens.

Trade-offs

• Keeps the silent shift and the mismatch with multi-valued fields, and contradicts what LUCENE-8650 set out to fix.
• Not run.

Index the values as a multi-valued field instead: the indexing chain keeps the holes (executed: beta at 2). Dropping empty values before concatenating avoids the empty case, but not a value made only of stop words.

Trade-offs

• A multi-valued field applies the analyzer's position and offset gaps between values (0 by default for the position gap), which may differ from what the application wanted from one concatenated stream.

Method: code reading (the study's "non-formal" tier), hunt unit "attribute and char-array contracts", main ea57360912ed, 2026-10-26.

• The unit's end-to-end attribute and end-state harness ran ConcatenatingTokenStream with one source only, which passes; it pointed the search at end() and position handling in classes that copy attributes between streams.
• Reading the class found the assignment at line 92 next to the offset += at line 94; a probe with an empty middle source confirmed the lost hole.
• An AI finder then wrote the public-API test, traced the cause and wrote a fix sketch; independent AI tools reproduced, argued and judged it (see the checks).