After an AttributeFactory throws, IndexWriter can index later documents of the field with the previous stream's positions and offsets

For each field, FieldInvertState caches the TokenStream it last inverted together with handles to that stream's attributes, and skips the lookup when the same stream object comes back, which is the normal case because an analyzer hands back the same reused TokenStream object (per thread) each time. setAttributeSource stores the new stream as the cached identity first (FieldInvertState.java:98 at 5adff78bb168) and only then calls addAttribute for TermFrequencyAttribute, PositionIncrementAttribute and OffsetAttribute (:107-109), which asks the stream's AttributeFactory for any attribute the stream lacks. If that factory throws, IndexWriter rejects the document as a non-aborting exception, as documented, but the cache now names the analyzer's stream while the frequency, position, offset and payload handles still belong to the previous stream of that field. Every later document of the field in the same in-memory segment that gets the same stream back reuses those handles, until the next flush or until the field inverts a different TokenStream object, and the documents mis-indexed in that window keep their wrong positions: after a pre-analyzed value that ended in two removed stop words, "a b" is indexed at positions 1 and 3 instead of 0 and 1 and PhraseQuery f:"a b" finds nothing, with no error. With no earlier stream, later documents fail with a NullPointerException instead. Every release examined since 4.9.0 (2014) has it. The trigger is an application-written factory that throws, which no non-test factory in Lucene's source does, so the priority is Low.

1. Getting the stream. IndexingChain.java:1981 calls field.tokenStream(analyzer, tokenStream): for a string value that is the analyzer's reused stream (analysis/Analyzer.java:183-195, document/Field.java:493-498), for a TokenStream value the field's own stream. :1984 calls invertState.setAttributeSource(stream).
2. The identity is stored first (the fix site). FieldInvertState.java:97 compares the stream with the cached one; :98 stores the new stream; :106 refreshes termAttribute; :107 calls addAttribute(TermFrequencyAttribute.class). StandardTokenizer creates its term, offset, position-increment and type attributes itself (analysis/standard/StandardTokenizer.java:140-144) but not the frequency attribute, so AttributeSource.addAttribute asks the factory (util/AttributeSource.java:218), which throws. :108-110 never run.
3. The document is rejected, the cache is not. The exception is non-aborting: only this document is marked deleted (DocumentsWriterPerThread.java:392-398; the runs show the rejection with the writer still open). Left behind: identity = the analyzer's stream B, termAttribute = B's, but termFreqAttribute, posIncrAttribute, offsetAttribute and payloadAttribute = the first document's stream A, whose end() left increment 2 and offsets 9-9 (analysis/FilteringTokenFilter.java:72-74, analysis/standard/StandardTokenizer.java:178-184). A probe that reads this state by reflection on main prints exactly that.
4. Later documents read the old values. The next document gets the same stream B; :97 sees the cached identity and returns early. Every token reads A's frozen values: IndexingChain.java:1996 (increment 2) and :2035-2036 (offsets 9-9). From position = -1 (FieldInvertState.java:84), a lands at 1 and b at 3.
5. Until the next flush, or a different stream. There is one FieldInvertState per field per in-memory segment, that is per DocumentsWriterPerThread (IndexingChain.java:1416, :1905-1909). A different TokenStream object for the field fails the identity check at :97 and refreshes the handles (executed: a document with its own stream "c d" between the failure and the next "a b" gives PhraseQuery f:"a b" count 1). After a commit the next document is indexed correctly, but the document mis-indexed before it keeps positions 1 and 3 (executed).

"Later documents" means later documents of the field that get the same stream back. Not run: an earlier stream whose end() leaves increment exactly 1 (one trailing removed stop word); by reading (FilteringTokenFilter.java:72-74), positions then come out right and only the offsets are stale.

• Who. An analyzer whose tokenizer was built with an application-written AttributeFactory that throws when IndexWriter requests TermFrequencyAttribute (before 7.0.0: PositionIncrementAttribute), once or always.
• Reach limits (by reading). The factory is asked only for an attribute the stream does not have yet (AttributeSource.java:209-221). A factory built on the packed token factory, the default, is never asked here, because PackedTokenAttributeImpl already implements all three attributes (PackedTokenAttributeImpl.java:29-40); one that creates a separate implementation per attribute, as one built on AttributeFactory.DEFAULT_ATTRIBUTE_FACTORY does (the repro's), is asked for TermFrequencyAttribute. No non-test AttributeFactory in the Lucene source at 5adff78bb168 throws for these attributes (the default ones in AttributeFactory.java and the two collation factories). The silent variant also needs the field's cache to hold another stream's handles when the factory fails: in the repro, an earlier document in the same in-memory segment that passed its own TokenStream for that field (by reading, another thread's analyzer stream used earlier in the same pooled in-memory segment would do the same; not run).
• Effect. Silent wrong positions (phrase, span and interval queries miss the documents) and wrong offsets (highlighting), or later documents of the field rejected with a misleading error. New documents are affected until the next flush or until the field inverts a different TokenStream object; the documents already indexed with wrong positions keep them after the flush (executed). The failing document itself is handled correctly.
• Scale. Three small documents, one thread. Likelihood contrived: no matching public report was found in the documented searches, although every release examined since 4.9.0 (2014) has it.

TestFieldInvertStateAttributeFactoryFailure (one method, extends LuceneTestCase, package org.apache.lucene.index of the lucene/core tests, public API only). Document 1 is a TextField whose value is StopFilter(StandardTokenizer("x the the")); document 2 is "a b" through an analyzer whose StandardTokenizer(factory) throws IllegalStateException the first time TermFrequencyAttribute is requested (expectThrows); document 3 is the same "a b" document. The test asserts that PhraseQuery f:"a b" counts 1.

The result does not depend on the seed: this file fails with expected:<1> but was:<0> on main with DEADBEEF (assertions on and off), 5EEDF151 and A0D17001 (on) and 5EEDF152 (off), and on the 9.1.0, 9.12.3, 10.5.2 and 10.6.0 release jars. The comment line is 102 columns, so checkGoogleJavaFormat flags it and wraps its last word (formatting only). As the regression test, the same method fits in TestFieldInvertState or TestIndexWriterExceptions.

cp $REPRO/TestFieldInvertStateAttributeFactoryFailure.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestFieldInvertStateAttributeFactoryFailure -Ptests.seed=DEADBEEF

# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF -Ptests.asserts=false (assertions on: the same assertion at the same line)
TestFieldInvertStateAttributeFactoryFailure > testFactoryFailureDoesNotShiftLaterPositions FAILED
    java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:B9F9175C71685E69]:0)
        ...
        at org.apache.lucene.index.TestFieldInvertStateAttributeFactoryFailure.testFactoryFailureDoesNotShiftLaterPositions(TestFieldInvertStateAttributeFactoryFailure.java:72)
BUILD FAILED

# with fix A: BUILD SUCCESSFUL with assertions on and off

# postings of the accepted "a b" document, offsets indexed (state probe, assertions off)
main:            PhraseQuery f:"a b" count=0 | a pos 1 off 9-9 | b pos 3 off 9-9
main with fix A: PhraseQuery f:"a b" count=1 | a pos 0 off 0-1 | b pos 1 off 2-3

• Contract. DocumentsWriter.java:76-79: after a non-aborting exception "we immediately mark the document as deleted so that the document is always atomically ("all or none") added to the index." DocumentsWriterPerThread.java:392-398: an exception during analysis "causes the doc to be rejected but won't cause the DWPT to be stale". IndexingChain.java:1988-1994 says the same for an exception in the token loop ("in stream.next below"); this throw comes just before that loop, from setAttributeSource at :1984, and the runs show it is handled the same way. IndexWriter.java:1465: "if an Exception is hit ... the index will be consistent".
• Tests. TestIndexWriterExceptions.testExceptionFromTokenStream (LUCENE-1072, :493-494) throws from a TokenStream, then: "Make sure we can add another normal document" (:541).
• History. The cache came with LUCENE-5611 (#6673). Its first, inline form in DefaultIndexingChain.invert stored the identity as its last statement (67ebc26fa97, kept by f0414ff3af7). After the review request "Can we factor this into FieldInvertState?" (Robert Muir on #6673), e7b59c5a211 moved the block into the setter with the identity first; no comment in the thread discusses the setter's order or a throw from its lookups (its commit notices about non-aborting exceptions concern other code). The parent commit does not reproduce; e7b59c5a211 does (executed).

The strongest case for "not a bug": a throwing AttributeFactory is a broken component. The factory javadoc names only UndeclaredThrowableException and says "attributes should not throw or declare checked exceptions" (AttributeFactory.java:28-35), and #9656 (LUCENE-8610), an NPE from the same cache, was closed Not A Bug as the tokenizer's fault.

Why it fails: that javadoc sentence is about attribute constructors, and createAttributeInstance is a public extension point that IndexWriter itself calls. In #9656 the stream broke the TokenStream contract by adding attributes lazily; here every stream follows the contract, and the failure is reported to the caller, who then continues as LUCENE-1072's test does. IndexWriter documents that such a failure drops one document, not that it changes how later ones are indexed. No javadoc explicitly promises isolation after a throwing AttributeFactory, so the case is that the behaviour is inconsistent with the non-aborting design above, not that it breaks an explicit contract. That bears on likelihood, which is why the priority is Low, not on intent. A credibility audit anticipated the "broken component" objection and still kept the finding, with its small fix and single test.

Fix A was built and checked through gradle on 5adff78bb168; B passed the independent reproducer's own test; C was not built.

After a failure the next call with the same stream fetches the handles again: a factory that failed once now works, and one that always fails rejects every document loudly instead of mis-indexing them.

      // clear the identity first, so that a failing addAttribute below leaves no stale handles
      this.attributeSource = null;
      termAttribute = null;
      termFreqAttribute = null;
      posIncrAttribute = null;
      offsetAttribute = null;
      payloadAttribute = null;
      if (attributeSource != null) {
        termAttribute = attributeSource.getAttribute(TermToBytesRefAttribute.class);
        termFreqAttribute = attributeSource.addAttribute(TermFrequencyAttribute.class);
        posIncrAttribute = attributeSource.addAttribute(PositionIncrementAttribute.class);
        offsetAttribute = attributeSource.addAttribute(OffsetAttribute.class);
        payloadAttribute = attributeSource.getAttribute(PayloadAttribute.class);
        this.attributeSource = attributeSource;

Trade-offs

The fast path (same stream, :97) is unchanged; a new stream costs six extra null stores (the identity and the five handles). No handle of the previous stream survives a failed call.

What the checks validated

• Gradle in a scratch worktree at 5adff78bb168, seed DEADBEEF: the repro fails without the fix (assertions on and off) and passes with it (on and off).
• With the fix: TestFieldInvertState (2), TestIndexWriterExceptions (40, 3 skipped), TestIndexWriterExceptions2 (1), TestDocumentWriter (15), TestPostingsOffsets (12), TestPayloads (7) and the repro: 78 tests, 0 failures. checkGoogleJavaFormat flags only the repro's comment line. The full core suite was not run.
• Earlier, a separate re-run with the same change passed; the cross-model reviewers' equivalent change passed TestFieldInvertState and TestIndexWriterExceptions2 (refute lens) and TestFieldInvertState and TestIndexWriterExceptions.testAddDocsNonAbortingException (intended lens).

Two lines: this.attributeSource = null; before the lookups, this.attributeSource = attributeSource; after them.

Trade-offs

It leaves the previous stream's handles in place after a failure while the cached identity is null, and by reading one path still reads them: a later non-tokenized (TERM) value of the same field calls setAttributeSource(null) (IndexingChain.java:2146), which then returns early, and FreqProxTermsWriterPerField.start copies termFreqAttribute (:78-80). The field schema compares index options, not tokenization (IndexingChain.java:2278-2288), so for a field without positions that sequence is accepted. Not run. A clears the handles and has no such gap.

What the checks validated

Passed the independent reproducer's own test (expected:<0> but was:<1> without it); not run through gradle.

As the cache's original inline form did.

Trade-offs

After a failure the identity keeps the previous stream while termAttribute already points at the new one, so if that previous stream object is passed again, its term attribute is wrong. Not built.

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

• The finder checked setAttributeSource against the obligation that a call which throws leaves no cache a later call with the same source relies on. The identity is assigned at :98, before the addAttribute calls at :107-109, which reach the user's AttributeFactory.
• The finder then wrote the public-API JUnit test (positions 1 and 3 instead of 0 and 1), two probes (the positions, and the NullPointerException variant with no earlier stream) and the fix that makes the test pass.
• An independent reproducer, two executed cross-model reviews, a devil's-advocate verifier and an adjudication reproduced it with their own tests ("Prior reports, history and checks").