DelimitedPayloadTokenFilterFactory shares one custom PayloadEncoder across all threads, so an encoder that reuses its BytesRef stores other documents' payloads (since 4.0)

DelimitedPayloadTokenFilterFactory (SPI name delimitedPayload) accepts encoder=float|integer|identity or the fully qualified name of your own PayloadEncoder class. For a class name, inform() creates one instance (DelimitedPayloadTokenFilterFactory.java:82) and create() passes that same instance to every DelimitedPayloadTokenFilter it builds (:69-71). An Analyzer keeps one set of components per thread (Analyzer.java:91-96), and CustomAnalyzer calls create() once per thread on its single factory (CustomAnalyzer.java:161-168), so every thread calls the same encoder object. The filter stores the returned BytesRef by reference (DelimitedPayloadTokenFilter.java:59) and the indexing chain copies it a moment later (FreqProxTermsWriterPerField.java:88-92). An encoder that reuses one BytesRef is correct when only one token stream uses it, because the consumer copies the payload bytes before it asks that stream for the next token; through the factory every thread uses the same encoder, and another thread's encode() overwrites the bytes first. With 4 threads x 5,000 addDocument calls, 384 to 7,153 of 20,000 documents silently stored a wrong payload (another document's, or a torn mix of bytes) across all runs, and CheckIndex stayed clean. With a variable-length encoder the commit's flush throws and the IndexWriter closes itself, losing every document since the last commit. PayloadEncoder, the factory and the factory lifecycle docs say nothing about sharing or threads, while PhoneticFilterFactory and SnowballPorterFilterFactory, the siblings that load a class by name, create one instance per create(). Nothing Lucene ships is affected: the built-in encoders return a new BytesRef per call, so this is a documentation and hardening defect (impact class minor). Affected: every release from 4.0.0-BETA to 10.6.0 (10.6.0 by reading), and main. Creating a new custom encoder per create() fixes it; with that change the whole :lucene:analysis:common:test suite passes.

• DelimitedPayloadTokenFilter: splits a token such as word|1.0 at the delimiter, keeps word as the term and turns 1.0 into the token's payload (bytes stored at that position in the index) with a PayloadEncoder. Built-in encoders: float, integer, identity; any other value of the factory's encoder argument is loaded as a class name.
• Per-thread components: an Analyzer builds its tokenizer and filters once per thread and reuses them (Analyzer.java:91-96). CustomAnalyzer builds them by calling create() on the same factory objects in every thread (CustomAnalyzer.java:161-168).
• Payload ownership: a filter hands the payload to the consumer as a BytesRef in its PayloadAttribute. The consumer reads it before it calls incrementToken() again, and a filter that keeps a token for later gets a deep copy (captureState() clones the attributes; PayloadAttributeImpl.clone() and copyTo() use BytesRef.deepCopyOf), so a producer may reuse one BytesRef within its own stream; Lucene's own test filter PayloadSetter does ("reuse the payload / byte[]"). Between two incrementToken() calls the attributes belong to the caller (TokenStream.java:124-127).

1. inform() creates the custom encoder once: encoder = loader.newInstance(encoderClass, PayloadEncoder.class); (DelimitedPayloadTokenFilterFactory.java:82, field at :51).
2. create() passes that same object to every filter: return new DelimitedPayloadTokenFilter(input, delimiter, encoder); (:69-71). With 4 indexing threads, 4 filters share one encoder.
3. The filter stores the returned BytesRef by reference: payAtt.setPayload(encoder.encode(buffer, i + 1, (length - (i + 1)))); (DelimitedPayloadTokenFilter.java:59).
4. The indexing chain copies the payload right after incrementToken() returns, in FreqProxTermsWriterPerField.writeProx: it writes payload.length (:91) and then the bytes (:92). In between, thread B's encode() on the same object has rewritten the shared bytes, so A's document stores B's value, or a torn mix of bytes (the delivered test once stored 2.2959E-41 for t0, whose payload is 0.0).
5. With a variable-length encoder (BytesRefBuilder.copyChars grows the array, then sets the length), the length can change between :91 and :92; the in-memory position stream gets out of step and the flush that decodes it fails in FreqProxFields$FreqProxPostingsEnum.nextPosition. A flush exception is an aborting exception: the flush aborts and fires a tragic event (DocumentsWriterPerThread.java:585-587, :594-605), and on a tragic event the writer rolls back and closes (IndexWriter.java:6098-6106), so every document since the last commit is lost. The length change between the two lines is read from the code, not instrumented.

New JUnit class through gradle on main, 13 runs each (assertions off twice, on once, and 10 repetitions): the latch test sees 2.0 instead of 1.0 every time; 4 threads x 5,000 addDocument give 693 to 2,992 of 20,000 documents with a wrong payload; the variable-length test's commit() fails 13 of 13 with the writer closed. Across all earlier runs on main and releases: 384 to 7,153 of 20,000. In the security screen's 30 variable-length rounds, addDocument never threw, the commit failed every time, all 12,000 new documents were lost, and the earlier commit stayed CheckIndex-clean. Controls (one encoder per filter; the built-in float encoder through the factory): 0 wrong in every run.

The concurrent user can also be query parsing with the same Analyzer object: two threads analysing q|666.0 with QueryBuilder made 170 to 828 of 10,000 concurrently indexed documents store 666.0, and variable-length query payloads closed the writer in 11 of 15 rounds. These counts reflect the probe's load (two threads analysing queries continuously while two threads indexed): they show the path is reachable, not how often it happens in practice.

• Who is exposed. Users of encoder=<your class>: through CustomAnalyzer with delimitedPayload, the SPI name, or a Solr schema. Their encode() must keep per-instance state, such as a reused BytesRef or BytesRefBuilder, a natural allocation-saving choice that is safe while only one token stream uses the encoder.
• What it takes. More than one thread using the analyzer: IndexWriter is "completely thread safe" and its indexing threads share the one analyzer of its config (IndexWriter.java:167-168); query parsing with the same Analyzer object counts too. One thread is enough when two filters from the same factory are consumed interleaved (the independent reproducer's testTwoFiltersFromOneFactory, direct factory use: stream A reads 2.0, not 1.0). No special input and no scale, assertions on or off, 4 threads. The tests set no codec, IndexWriterConfig or JVM option (a plain new IndexWriterConfig(analyzer)); through gradle the test framework chose its randomized environment (recorded: codec Asserting(Lucene104)) and the build's default test JVM options, and the security probe, a plain main() with the default codec, failed the same way.
• What the damage looks like. Fixed-length encoder: wrong payloads in the index with no exception, and CheckIndex clean, so payload scoring and payload checks use wrong values. Variable-length encoder: the next flush throws, the writer closes itself and every document since the last commit is lost; in the security screen it happened again in every round, each with a new writer and directory. No memory corruption; every surviving commit was CheckIndex-clean.
• Not affected. The built-in encoders (a new BytesRef per call); custom encoders that return a new BytesRef per call and keep no other mutable state; the filter built through its public constructor with one encoder per filter; one thread that consumes one token stream at a time, as a single indexing thread does (probe: 0 of 20,000 wrong). Elasticsearch and OpenSearch accept only the built-in encodings (read from their code, not run).
• How likely. Rare (registry): it needs a user-written stateful encoder loaded by class name; the only public custom encoders found that are used through this factory allocate per call, and the saved searches found no report since the factory was added in 2009.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: payload of x|1.0 expected:<1.0> but was:<2.0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.apache.lucene.analysis.payloads.TestDelimitedPayloadFactoryEncoderSharing.testFiltersFromOneFactoryKeepTheirOwnPayloads(TestDelimitedPayloadFactoryEncoderSharing.java:57)

• No text states the precondition. PayloadEncoder's javadoc is "Mainly for use with the DelimitedPayloadTokenFilter, converts char buffers to BytesRef. NOTE: This interface is subject to change"; encode says "Convert a char array to a BytesRef / @return encoded BytesRef" (PayloadEncoder.java:21-35). The factory javadoc shows only encoder="float" (DelimitedPayloadTokenFilterFactory.java:25-38), and AbstractAnalysisFactory says only "Consumer calls create() to obtain instances" (AbstractAnalysisFactory.java:45-52). A grep for "thread" finds nothing in these files or in TokenFilterFactory, ResourceLoader(Aware), CustomAnalyzer and PayloadAttribute(Impl); Analyzer mentions threads only for its own per-thread reuse of components (Analyzer.java:91-96), not for factories or the objects they hold.
• The custom-class path was designed in. The commit that added the factory (SOLR-1285, b5620d86146, 2009) documented it in the example schema: "Fully Qualified class name implementing PayloadEncoder Encoder must have a no arg constructor." Thread safety is not in that list.
• Reuse is an accepted pattern inside one stream. The consumer reads the payload before the next incrementToken(), a filter that keeps a token gets a deep copy (captureState(); PayloadAttributeImpl.clone() and copyTo()), and TestAnalyzers.PayloadSetter reuses one BytesRef ("reuse the payload / byte[]").
• The token-stream contract gives the attributes to the caller. TokenStream.incrementToken(): "The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it" (TokenStream.java:124-127). Between two calls the payload belongs to that stream's consumer; a shared encoder lets a different stream rewrite its bytes in that window.
• Siblings create one instance per filter. PhoneticFilterFactory.getEncoder(): "Must be thread-safe." "Unfortunately, Commons-Codec doesn't offer any thread-safe guarantees so we must play it safe and instantiate every time." (PhoneticFilterFactory.java:160-166). That came from SOLR-3424, "PhoneticFilterFactory thread-safety bug" (9707be81c1c, 2012), where a committer wrote that user-provided classes "may not be thread safe (unless this is explicitely specified in their spec / javadocs - which isn't)". SnowballPorterFilterFactory resolves the class once and creates a new stemmer per create() (:83, :94).
• At least one comparable factory states its sharing assumption. Word2VecSynonymProviderFactory: "Assumes synonymProvider implementations are thread-safe." (:29). Factories that share plain data (stop sets, maps) do not say so.
• Precedent for the bug class. LUCENE-2943 (#4017, 2011): an ICU Collator shared across threads' token streams, fixed by cloning per stream.

Every checker built the strongest case for "intended": objects a factory builds in inform() are conventionally shared by all its filters and so by all threads (stop sets, SynonymMap, dictionaries); the built-in encoders return a new BytesRef per call and are safe to share, and the interface reads as a pure function; the factory javadoc never advertises the custom-class path; so a stateful encoder is the user's bug, and a committer may close it as user error or answer with one javadoc sentence.

Why it fails as a reason to do nothing: those shared objects are immutable data, while an encoder is user code with an output buffer; no text tells the user it is shared; the custom path was documented with a precondition list that leaves thread safety out; the 2012 sibling fix decided the same question the other way; and at least one comparable factory states its sharing assumption. It is still why the row is minor: there is no broken contract, only a missing one, and a javadoc-only fix is a legitimate answer (option B). Verdict: not intended as written; a documentation and hardening defect. The one "not real" review had two grounds: it could not run its own repro, so it voted not real on procedure, and it judged the class-name path outside the documented inputs because the factory javadoc shows only encoder="float" (the 2009 schema comment speaks against that); it agreed on the mechanism.

Options A and A2 were applied in a scratch worktree of a9779ef3c193 and run through gradle; the worktree was removed afterwards. Option B is documentation only. A workaround for users today: return a new BytesRef from encode(), or build the filter in your own Analyzer.createComponents with new DelimitedPayloadTokenFilter(input, '|', new MyEncoder()).

+16/-1 in DelimitedPayloadTokenFilterFactory: inform() still creates one instance through the ResourceLoader (early errors) and remembers its class; create() passes the shared built-in encoder, or a new instance of the custom class.

   private PayloadEncoder encoder;
+  // set for a custom encoder class: create() then makes a new instance for every filter, because
+  // the filters of all threads using the analyzer would otherwise share one, and nothing requires a
+  // PayloadEncoder implementation to be thread-safe
+  private Class<? extends PayloadEncoder> customEncoderClass;
@@
   public DelimitedPayloadTokenFilter create(TokenStream input) {
-    return new DelimitedPayloadTokenFilter(input, delimiter, encoder);
+    return new DelimitedPayloadTokenFilter(
+        input, delimiter, customEncoderClass == null ? encoder : newCustomEncoder());
+  }
+
+  private PayloadEncoder newCustomEncoder() {
+    try {
+      return customEncoderClass.getConstructor().newInstance();
+    } catch (ReflectiveOperationException e) {
+      throw new IllegalArgumentException("Cannot create payload encoder: " + encoderClass, e);
+    }
   }
@@
     } else {
+      // one instance now, so that a bad class name fails here
       encoder = loader.newInstance(encoderClass, PayloadEncoder.class);
+      customEncoderClass = encoder.getClass();
     }

Trade-offs

• One reflective construction per create(): once per thread and analyzer with CustomAnalyzer (the default GLOBAL_REUSE_STRATEGY), not per token or document. PhoneticFilterFactory's comment calls the same pattern "negligible".
• getConstructor().newInstance() is what the default ResourceLoader.newInstance does (ResourceLoader.java:33-40) and what both siblings do, but it bypasses a custom ResourceLoader that overrides newInstance; option A2 keeps that hook.
• A custom encoder that is expensive to construct, or that relies on being one shared instance, changes behaviour.

What the checks validated

• TestDelimitedPayloadFactorySharedEncoder passes with assertions off, and 50 of 50 with -Ptests.iters=10 and assertions on (stock: 30 of 50 fail). With the existing payload tests: 6 classes, 15 tests, 0 failures.
• All of :lucene:analysis:common:test with assertions on: 340 classes, 1,914 tests, 18 skipped, 0 failures, 0 errors.
• :lucene:analysis:common:check -x test passes (checkGoogleJavaFormat, ecjLint, forbiddenApis) with the new test file.
• Earlier: the original hunt test (4 of 4) and the independent reproducer's 5 tests (5 of 5, also with a new seed) pass with the same approach, and so does the version-portable check.

Keep the ResourceLoader given to inform() and call loader.newInstance(encoderClass, PayloadEncoder.class) in create() for a custom class (+11/-1).

Trade-offs

Honours a custom ResourceLoader's instantiation (a hunt reviewer asked for this), but the factory, and so the analyzer, keeps the loader (possibly with a class loader) reachable for its lifetime and uses it after inform(), which the lifecycle text does not describe. How Solr's loader behaves when called after core initialisation was not checked.

What the checks validated

The payload tests (org.apache.lucene.analysis.payloads.*, the new class included): 6 classes, 15 tests, 0 failures; checkGoogleJavaFormat passes. The module suite, check and repetitions were run only for A.

If maintainers prefer one shared instance, the javadoc of PayloadEncoder and of the factory must say more than "thread-safe": the factory shares one instance of a custom class among all its filters and so all threads; encode() must be safe to call concurrently and must return a new BytesRef, or one that is never modified afterwards, on every call, because the filter keeps the returned reference until the consumer has copied it. A synchronized encode() that returns a shared scratch BytesRef is still broken.

Trade-offs

No code and no cost, but users with a stateful encoder keep the silent failure until they read the javadoc; the new test would then document misuse instead of failing. Not run.

payAtt.setPayload(BytesRef.deepCopyOf(encoder.encode(...))).

Trade-offs

One more allocation per token for the built-in encoders, and it does not help when two threads run encode() on the same scratch buffer at the same time: the copy itself can see the other thread's writes (reasoned from the code, not run). A ThreadLocal encoder in the factory would cover the threads but adds state and still shares one encoder between two filters on one thread; one encoder per filter covers both.

Method: a shared-state inventory plus a directed test (registry tier semi-formal: no formal tool produced the counterexample).

• The hunt unit on shared state in analysis components built a bytecode inventory of the factory and Analyzer fields that reach token-stream components. It listed DelimitedPayloadTokenFilterFactory.encoder, read in create(), and classed it safe because the three built-in encoders are safe to share.
• An independent reviewer from another model family (OpenAI GPT-5.6), asked to refute that negative result, named the hole: a closure over the shipped classes cannot cover a user-named encoder class whose interface promises nothing about threads.
• The finder then wrote the public-API JUnit with a deterministic latch interleaving and concurrent IndexWriter indexing, measured the hit rate (20 of 20 with assertions off; the per-filter control 0 of 20), ran it with assertions on and off, on 10.5.2 and on 9.12.3, and checked the per-create() fix.