TeeSinkTokenFilter: a keyword marker on a sink stops stemming every word after the first protected one

TeeSinkTokenFilter lets one analysis chain feed several fields: the tee saves each token's state and a sink replays it later, where more filters can be added. A sink has its own copy of the tee's attributes, and SinkTokenStream.incrementToken() only calls restoreState(), never clearAttributes(). So an attribute that only the sink's own filters add is never reset between tokens. With SetKeywordMarkerFilter protecting quick before PorterStemFilter on a sink, The quick foxes jumped is indexed as [the, quick, foxes, jumped] instead of [the, quick, fox, jump], and stemmed:fox finds nothing. Silent, deterministic, in every release from 2.9.0 to 10.6.0 (10.6.0 by reading) and main; the fix adds clearAttributes() to SinkTokenStream.incrementToken(), plus a separate super.end() hardening in end().

• Attributes. A token stream describes the current token through attribute objects: the term text, offsets, position, type, and optional flags such as KeywordAttribute ("do not stem this word"). All filters of one chain share one set of attribute objects.
• Clearing. The source at the start of a chain resets every attribute before each token ("Tokenizers must call clearAttributes() in incrementToken()", a rule LUCENE-1801 applied to every token source). Filters that only ever switch a flag on rely on that reset.
• Tee and sink. TeeSinkTokenFilter saves each token's state with captureState(); each sink from newSinkTokenStream() replays the saved states with restoreState(). The class javadoc wraps sinks in further filters (new EntityDetect(sink1)).

1. newSinkTokenStream() gives each sink its own attribute source, a copy of the tee's attributes (new SinkTokenStream(this.cloneAttributes(), cachedStates), TeeSinkTokenFilter.java:69).
2. SetKeywordMarkerFilter and PorterStemFilter wrapped around the sink add KeywordAttribute to that copy. The tee never has it, so none of the tee's saved states (:85) contains it.
3. SinkTokenStream.incrementToken() only calls restoreState(state) (:121). restoreState "does not affect attributes of the targetStream that are not contained in this state"; to reset them "the caller should first call TokenStream#clearAttributes() on the targetStream" (AttributeSource.java:312-316). The sink never does.
4. KeywordMarkerFilter only ever sets the flag to true (KeywordMarkerFilter.java:45-46), and PorterStemFilter skips keyword tokens (PorterStemFilter.java:64).
5. So: the (flag false) is stemmed; quick is protected and the flag turns on; foxes and jumped get their terms restored but keep the flag, and are not stemmed.

On the tee side the same filters are fine: they share the tokenizer's attributes, which the tokenizer clears for every token. Attributes the tee already has when the sink is created (term, type, offsets, position increment and length) are in every saved state and are restored correctly.

The quick foxes jumped: the sink chain gives [the, quick, foxes, jumped], the same chain on the tokenizer [the, quick, fox, jump]. Indexed with the tee as plain and the sink as stemmed (the documented order), stemmed:fox and stemmed:jump find 0 documents and plain:foxes finds 1. Adding KeywordAttribute to the tee before creating the sink gives the right terms. The test framework's own check reports "clearAttributes() was not called correctly in TokenStream chain at token 0" on a plain sink.

• Who is exposed. Only chains built in Java: TeeSinkTokenFilter has no analysis factory (Lucene's factory test lists it as a "crazy component" without one), so CustomAnalyzer and factory-based schemas cannot reach it. A read-only code search found no use in Elasticsearch, OpenSearch or Solr (default branches only; private applications are unknown).
• What triggers it. A sink-side filter that sets an attribute on some tokens only and relies on the chain's source to reset it. Executed: SetKeywordMarkerFilter, PatternKeywordMarkerFilter or StemmerOverrideFilter before PorterStemFilter, EnglishMinimalStemFilter ([cat, exempt, dogs] instead of [cat, exempt, dog]) or KStemFilter; StemmerOverrideFilter {dogs->canine, mice->mouse} also blocks its own second override ([canine, running, mice, walking] instead of [canine, run, mouse, walk]); PatternTypingFilter (digit tokens) + DropIfFlaggedFilter keeps only [order] of order 42 shipped to alice instead of [order, shipped, to, alice]; NumericPayloadTokenFilter gives later tokens the number's payload.
• Scale. In every value with a flagged token, the flagged token itself is right, but the attribute stays stale on all later tokens, and the output is wrong wherever a later filter depends on its default (here every later word the stemmer would change); deterministic, any size, assertions on or off. If one tee/sink chain is reused for several documents, the flag carries into the next document (foxes jumped indexed as [foxes, jumped]).
• Effect. Wrong terms or payloads in the sink field only; queries on it miss documents (or over-match on payloads). The tee's field and stored values are right, CheckIndex is clean, and documents stay wrong until reindexed.
• Not affected. The same filters on the tee side; attributes added to the tee before the sink is created; sink-side filters that set their attribute on every token.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[quick, fox]> but was:<[quick, foxes]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.sinks.TestTeeSinkSinkOnlyAttribute.testKeywordMarkerOnSink(TestTeeSinkSinkOnlyAttribute.java:55)

• Contract. AttributeSource.restoreState (AttributeSource.java:312-316): attributes not in the state "remain unchanged"; to reset them "the caller should first call TokenStream#clearAttributes() on the targetStream". The package docs: "Tokenizers must call AttributeSource#clearAttributes() in incrementToken()" (package-info.java:404-405). LUCENE-1801 (#2876, 2009) applied that rule to "All Tokenizers/TokenStreams that are source of tokens"; a reviewer there noted that restoreState() alone is not enough. SinkTokenStream already existed and was not changed.
• Documented use. Class javadoc: sinks "will further do additional token filtering" (TeeSinkTokenFilter.java:49-51, example new EntityDetect(sink1) at :41). CHANGES 5.5.0 (API Changes), LUCENE-6973: "If you wish to filter the sinks, you can wrap them with any other TokenFilter (e.g. a FilteringTokenFilter)." PorterStemFilter alone adds KeywordAttribute to the sink.
• Tests. TestTeeSinkTokenFilter.testGeneral adds CheckClearAttributesAttribute to the tee and to the sink (:123-124), so the check value rides in the saved states. No upstream test puts an attribute on the sink only. With it on the sink only the check fails; that is supporting evidence, and the stock filters' wrong terms are the stronger evidence.
• end(). TokenStream.end(): "If you override this method, always call super.end()" (TokenStream.java:156); SinkTokenStream.end() does not, so a sink-only attribute keeps its last-token value after end(). This is a separate, smaller contract gap: the wrong terms come from incrementToken() alone.
• History. The restore-only incrementToken() came with the class (LUCENE-1693, 2009) and survived the LUCENE-6973 rewrite (2016, first released in 5.5.0); no commit ever added clearAttributes() to the sink.

Strongest case (rated weak by the intended-lens reviewer from another model family, the independent reproducer and the devil's advocate; moderate by the refute-lens reviewer and by the cross-model fact check of this doc): the package rule names only Tokenizers; restoreState says a reset "might be desirable"; CachingTokenFilter also restores without clearing and its end() also skips super.end(); the 2009 design wanted a sink with "exactly the same attributes as the TeeSinkTokenFilter"; a committer wrote on dev@ in 2014 that "clearAttributes() is not needed before restoreState()"; the Lucene.NET port suppressed its warning about the missing base.End() as intended.

Why it fails: CachingTokenFilter shares its chain's attributes, so filters after it are in its states (the same keyword chain over a CachingTokenFilter replay is correct), while a sink has a separate copy by construction. The javadoc puts the reset on the caller; the package rule's wording names Tokenizers, and LUCENE-1801 applied it to every TokenStream that is a source of tokens, which a sink is. The "same attributes" remark was about creating the sink; the same javadoc wraps sinks in filters. The 2014 remark was about a filter whose state covers the whole shared chain. The Lucene.NET rationale ("calling base.End() would clear the attributes that RestoreState then sets") does not hold when super.end() runs before restoreState. Verdict: a defect. Expect the reply "add the attribute to the tee first": it works, but nothing documents it.

Option A was built and validated with gradle in a scratch worktree; option B was run through the standalone runner; option C was not run.

Two calls in SinkTokenStream (TeeSinkTokenFilter.java:121, :126): each sink token starts from cleared attributes, as on any tokenizer-rooted chain, and end() runs endAttributes() before restoring the tee's final state. The first call fixes the bug. The second is a separate end() contract fix: without it a sink-only attribute keeps its last-token value after end() (executed: a reviewer's testSinkEndResetsSinkOnlyAttribute fails on stock and passes with the fix); no in-tree consumer impact of that part was shown.

       AttributeSource.State state = it.next();
+      // the sink is the source of its own chain: reset the attributes the state does not hold
+      clearAttributes();
       restoreState(state);
       return true;
     }
 
     @Override
     public void end() throws IOException {
+      super.end(); // endAttributes(), also for attributes that only this sink's chain added
       State finalState = cachedStates.getFinalState();

Trade-offs

• Tee attributes do not change: every one is in every saved state and is restored right after the clear (assuming, as TokenStream requires, that the tee's attributes are all added before tokens are captured). Only sink-only attributes change, to their per-token default.
• Cost: one clearAttributes() per sink token; not benchmarked.
• A sink-side filter that relied on its attribute persisting across tokens would change; none is known.
• Existing indexes keep their wrong terms until reindexed.
• In a path the model treats as a caller error (consumeAllTokens() without the tee's end(), so no final state exists), the sink's end-of-stream values are wrong with or without the fix (3,579 of 4,092 model inputs on stock, 4,089 with it).
• The same lines are in every 9.x and 10.x release, and the +3-line diff applies cleanly (git apply --check) to the 10.5.2 and 9.12.3 sources; only main was built with gradle.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test fails 3 of 5 on stock and passes 5 of 5 with the fix; TestTeeSinkTokenFilter (3 upstream tests) and the new test pass (8 of 8).
• The whole lucene/analysis/common module: 340 suites, 1914 tests, 0 failures (18 skipped), with assertions on and off; :lucene:analysis:common:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• Standalone runner with the fix shadowed: the new test 5/5 (on and off), the independent reproducer's 5 tests 5/5, the follow-up's 4-test effect probe 4/4. The finder's exhaustive model (same logic) drops from 4,032 of 4,092 mismatching inputs to 0, and the documented-use scenarios stay at 0.

Fixes every wrong term and payload seen: the independent reproducer's 5 tests pass, 4 of 5 of the new test, and the follow-up's 4-test effect probe (payloads, dropped tokens, reused chain, deny rule) 4 of 4.

Trade-offs

Sink-only attributes keep their last-token value after end(), and the framework's end() check still fails on such a sink.

What the checks validated

Standalone runner on main: the new test's only failure is the framework's end() check, "super.end()/clearAttributes() was not called correctly in end()".

Say in the class javadoc that sink-side filters' attributes are never reset and must be added to the tee before newSinkTokenStream().

Trade-offs

No code change, so every existing chain stays wrong; stock filters such as PorterStemFilter add KeywordAttribute without the user asking.

Method: bounded-exhaustive translation validation (semi-formal tier). A reference model of the tee and its sinks was compared with the stock jar on every input up to a small scope: 0 to 4 tokens, position increments 1 or 2, offset gaps 0 or 1, final position increment 0 or 1, final offset slack 0 or 1, and 1 to 3 sinks.

• The hypothesis came from reading SinkTokenStream while modelling the sibling bug (an attribute added to the tee after a sink exists makes restoreState throw).
• New scenario F wraps each sink in a filter that adds KeywordAttribute and sets it on one token only, relying on the source's clear, and compares the result with the same filter run on the input itself. Stock main: 4,032 of 4,092 inputs mismatch; the documented-use scenarios A, C and D: 0.
• The finding was reduced to a public-API JUnit on the stock main jars, then checked by a reviewer from another model family (OpenAI GPT-5.6) in lenses, an independent reproducer, a devil's advocate, an adjudication and an audit.