AttributeSource.removeAllAttributes() leaves the cached State chain in place, so captureState(), copyTo() and hashCode() still see the removed attributes

AttributeSource, the base class of every TokenStream, keeps its attributes in two maps and caches the attribute instances as a linked State chain. addAttributeImpl() drops the chain whenever the attribute set changes. removeAllAttributes() clears both maps but not the chain, so once something has walked the chain (for example clearAttributes(), which tokenizers are documented to call per token), the methods that walk it keep seeing the removed instances until the next addAttribute().

With one CharTermAttribute set to "foo": restoreState(captureState()) on the same source throws IllegalArgumentException, copyTo(new AttributeSource()) throws, copyTo(target) overwrites the target's term "keep" with "foo", reflectAsString(false) prints term=foo,bytes=[66 6f 6f], and hashCode() is 110307 although equals() says the source equals a new AttributeSource (hash 0). Nothing in Lucene calls the method; at the only known caller, every consumer checked adds an attribute before it looks, so the impact is small. Present since the method was added in 5.5.0; the fix is one line, currentState[0] = null, the same invalidation addAttributeImpl() does.

• :62-66: "These two maps must always be in sync!!!": attributes (interface to instance), attributeImpls (class to instance), and private final State[] currentState, the cached chain. AttributeSource(AttributeSource input) shares all three with its input (:79-85), so a filter and its tokenizer see the same chain.
• :252-266: getCurrentState() returns currentState[0] as is whenever it is non-null (:253-255); it rebuilds the chain from attributeImpls only when the cache is null and hasAttributes() is true.
• :188-202: addAttributeImpl() sets this.currentState[0] = null for every new interface, with the comment "invalidate state to force recomputation in captureState()" (:196-197).
• :288-292: removeAllAttributes() is attributes.clear(); attributeImpls.clear();. currentState[0] is kept.
• Chain walkers, which see the removed instances: getAttributeImplsIterator() (:116), clearAttributes() (:272-276), endAttributes() (:282-286, called by TokenStream.end()), captureState() (:300-303), hashCode() (:335-341), reflectWith() (:418-422, behind reflectAsString() and toString()), copyTo() (:457-468).
• Map-based methods, which are right: hasAttributes(), hasAttribute(), getAttribute(), addAttribute(); equals() checks hasAttributes() first (:350, :372); cloneAttributes() is guarded by hasAttributes() (:434).
• restoreState(captureState()) throws because captureState() clones the stale chain and restoreState() looks each impl up in the now empty map (:322-327). restoreState() itself is fine.

It opens when any chain walker ran before removeAllAttributes(), and closes at the next addAttribute(), or at an addAttributeImpl() that registers at least one Attribute interface: after the removal every interface is new, so the first one added nulls the chain. An impl that implements no Attribute sub-interface registers nothing and leaves the stale chain (:149-169, :193-200). If nothing walked the chain before the removal, the source behaves like an empty one.

With the one-line fix every row gives the right-hand value.

• Trigger. Any chain walker (clearAttributes(), endAttributes(), captureState(), hashCode(), toString()/reflection, the impls iterator) before removeAllAttributes(), then any chain walker before the next addAttribute(). A filter built on the source with AttributeSource(input) shares the window.
• Not on any Lucene path. The only references to removeAllAttributes() in Lucene's sources are its declaration and TestAttributeSource.testRemoveAllAttributes. A read-only code search found no caller in Elasticsearch or OpenSearch. The only known caller is the pre-analyzed field tokenizer of Solr 9.x. Its parsers add attributes again for any value that has tokens (a value with no token list adds none), and every consumer that was checked adds an attribute before it observes the stream, which closes the window; Solr main removed that field type (SOLR-17839).
• Default versus opt-in. No setting is involved: an explicit call to a rarely used method plus a legal call order.
• Scale. None. One attribute and one clearAttributes() before the removal are enough.
• Effect. Wrong return values and an IllegalArgumentException on that in-memory source; copyTo() can write removed values into another source. No index, query or search effect: indexing runs gave the same outcome with and without the fix, and CheckIndex was clean.
• The hashCode() symptom needs non-default values. If the attributes are cleared right before the removal, the stale impls hash to 0 and the mismatch hides; the restoreState() and copyTo() failures do not depend on the values.
• Likelihood: rare. The usual remove-then-re-add pattern closes the window at once, and nobody has reported it since 2016.

new AttributeSource() gets a CharTermAttribute; clearAttributes() builds the cached chain, as a tokenizer does for every token; removeAllAttributes() empties the maps; restoreState(captureState()) then fails because the captured state still holds the removed CharTermAttributeImpl. Deterministic, single thread; LuceneTestCase is only the base class, and the package matches TestAttributeSource without using package-private API. It fails the same way with assertions on and off.

Regression test (for the recommended fix): this same class, or its one method added to TestAttributeSource next to testRemoveAllAttributes; it passes with the fix. The comment line is longer than 100 columns, so ./gradlew tidy rewraps it.

cp $REPRO/TestRemoveAllAttributesCachedState.java $LUCENE/lucene/core/src/test/org/apache/lucene/util/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestRemoveAllAttributesCachedState -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (environment lines trimmed; -Ptests.asserts=false identical)
   >     java.lang.IllegalArgumentException: State contains AttributeImpl of type org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl that is not in in this AttributeSource
   >         at __randomizedtesting.SeedInfo.seed([DEADBEEF:2B2D639CF8B7BF22]:0)
   >         at org.apache.lucene.util.AttributeSource.restoreState(AttributeSource.java:326)
   >         at org.apache.lucene.util.TestRemoveAllAttributesCachedState.testRestoreOwnStateAfterRemoveAllAttributes(TestRemoveAllAttributesCachedState.java:30)

:lucene:core:test (FAILURE): 1 test, 1  failure

# standalone, seed DEADBEEF: main 645d4fe14a20 and 10.6.0, -ea and -da, fail with the same exception; with the fix: pass

• The method's own javadoc: "Removes all attributes and their implementations from this AttributeSource" (AttributeSource.java:288). Afterwards the impls iterator, reflection and toString() still list the removed implementation.
• captureState(): "The return value can be passed to restoreState to restore the state of this or another AttributeSource" (:294-297). On the same emptied source that throws.
• copyTo(): "The given instance has to provide all Attributes this instance contains" (:450-452). The source contains none, yet copyTo(new AttributeSource()) throws and a matching target receives the removed values.
• clearAttributes(): "Resets all Attributes in this AttributeSource" (:268-271); it resets instances that are no longer in it.
• Object contract: equal objects must have equal hash codes; at the same moment the source equals a new AttributeSource but hashes to 110307, not 0.
• The cache is meant to follow the attribute set: addAttributeImpl() invalidates it (:196-197). It has done so since LUCENE-1693 (31a5f0edcc4, Michael Busch, 2009; the comment at :196); LUCENE-3042 (49ab81b316c, Uwe Schindler, 2011) made the cache a shared State[1] array and changed the reset to this.currentState[0] = null (:197).
• Precedent: LUCENE-3042 (#4115, fixed 2011) was this same cache missing an invalidation on another path (attributes added to an already consumed chain). Its CHANGES entry calls it "This bug", and it was backported to 2.9, 3.0 and 3.1.
• Intent of the method: SOLR-4619 added it "to support reuse of token streams with variable attributes, like PreAnalyzedTokenizer", listed under New Features for 5.5.0. Neither the discussion nor the patch mentions the cache; the only test, TestAttributeSource.testRemoveAllAttributes (TestAttributeSource.java:214-264), checks hasAttribute() and hasAttributes() only.
• Nothing says otherwise: no javadoc, test, CHANGES or MIGRATE entry accepts the stale state.

The strongest case for "intended": the org.apache.lucene.util package-info says "APIs in this package are primarily internal and may change between releases"; the TokenStream javadoc says attributes "must be added during instantiation", and committers called a tokenizer with no attributes before reset() "a bug really" (Robert Muir, SOLR-4619) and adding an attribute in incrementToken() "completely wrong" (Alan Woodward, LUCENE-8610 / #9656); nothing in Lucene calls the method; and the intended use, remove then re-add, never sees the stale chain.

Why it fails: the package-info sentence was added in 2026 by a javadoc-only change (#15719), long after the method shipped as a public, final, unmarked New Feature; the repro uses a plain AttributeSource, which the TokenStream rule does not cover, and no AttributeSource javadoc forbids using the source after the removal; the ordinary reading of the method's one-sentence javadoc and of captureState()'s strongly implies that the cache follows the attribute set, and the Object contract is explicit (the defect is that equal objects have unequal hash codes at the same moment, not that a mutable object's hash changes). These points argue for low impact, which the rating already reflects, not for intended behaviour. Every verifier, the cross-model lenses and the adjudication rated the intended argument weak.

One fix was validated in a scratch worktree of 645d4fe14a20; the alternatives are described but not run.

One statement at the end of removeAllAttributes(), the same one addAttributeImpl() uses. The finder wrote it, and an independent reproducer wrote the same line on its own.

   public final void removeAllAttributes() {
     attributes.clear();
     attributeImpls.clear();
+    // invalidate the cached state chain, as addAttributeImpl() does when the attribute set changes
+    this.currentState[0] = null;
   }

Trade-offs

• None found. Sources built with AttributeSource(input) share the array, so they are fixed too. No cost on Lucene's in-tree paths (the method has no Lucene caller); one array write per explicit removal. Code that observes the source after the removal and before the next addAttribute() now sees an empty source. Not changed: an impls iterator obtained before the removal still walks the old chain (:117-120), and a State captured before the removal keeps the old values, because it is a deep copy by design (:298-302); neither is this bug. Add a CHANGES entry.

What the checks validated

Validated on 2026-11-01 with gradle (--no-daemon) in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3:

• stock: the repro fails with assertions on and off;
• with the fix (+2 lines in AttributeSource.java): the repro passes with assertions on and off; TestAttributeSource (7 tests, including testRemoveAllAttributes) and every core analysis test class (org.apache.lucene.analysis.*, 21 classes) pass with seeds DEADBEEF and CAFEBABE: 22 classes, 175 tests, 0 failures;
• checkGoogleJavaFormat: the patched AttributeSource.java is clean; only the repro test's long comment line is flagged, and ./gradlew tidy rewraps it into two lines;
• standalone, with the fix shadowed: the repro passes with assertions on and off, and the probe gives the empty source's value in every row of the table above.

Earlier hunt runs with the same line: the finder's 4 tests and upstream TestAttributeSource (7 of 7) pass; an independent reproducer's 8-test JUnit passes 8 of 8; the copyTo() probe keeps the target's "keep".

Return null from getCurrentState() when !hasAttributes(), or rebuild a stale chain.

Trade-offs

Adds a check to a method that clearAttributes() calls for every token, to guard against the one method that forgot the invalidation; the recommended fix changes only that method.

The method has no Lucene caller, and its only known caller was removed from Solr main; upstream may prefer to deprecate it in 10.x and remove it on main with a MIGRATE note.

Trade-offs

Does not fix released branches unless combined with the recommended one-line fix, which could ship first.

State that the source must not be used before the next addAttribute().

Trade-offs

Leaves the javadoc contracts and the Object contract broken.

Method: non-formal (code reading, then a randomized contract checker and executed tests).

• Finder. The hunt unit for attribute and char-array contracts read AttributeSource and noticed that removeAllAttributes() changes the attribute set without the currentState[0] = null that addAttributeImpl() performs.
• Contract checker. The unit's randomized property-based checker built 3,000 random attribute sets over 3 attribute factories and ran 134,973 contract checks. It reported 4 distinct violations, all after removeAllAttributes() and all this root cause: captureState() non-null, the impls iterator non-empty, reflection non-empty, and equals() true while hashCode() differs.
• Repro. A public-API JUnit test then failed on main with assertions on and off and on 10.5.2, and passed with the one-line fix; the later minimal repro keeps one of its four checks, restoreState(captureState()).