OpenNLP analysis factories cache models and dictionaries JVM-wide by file name, so a second configuration silently uses another configuration's same-named file

Lucene's OpenNLP analysis factories (the openNlp tokenizer and the openNlppos, openNlpChunker and openNlpLemmatizer filters) load their models and lemma dictionaries through OpenNLPOpsFactory, which keeps seven static maps, one per resource type, keyed only by the resource name. The ResourceLoader a factory receives in inform() is read only when the name is not yet cached; the factory then fetches the object by name in create(). So within one JVM the first configuration that loads a file name decides what every later configuration gets under that name: an analyzer whose own lemmas.txt maps dogs to dogB outputs dogA; with two directories holding different tok.bin models, the second analyzer tokenizes with the first one's model; and a configuration that lacks the file still builds. Nothing raises an error, and it lasts until the JVM restarts or the application calls the public clearModels(). Caching by name is deliberate (LUCENE-2899, to share multi-megabyte models), so this is a documentation defect: no javadoc says that the names form one namespace for the whole JVM, while CustomAnalyzer.builder(Path) says it "loads all resources from the given file system base directory". Present since 7.3.0; affected through 10.6.0 and main.

• ResourceLoader: what an analysis factory reads its files through; CustomAnalyzer.builder(Path) uses a FilesystemResourceLoader for one directory, builder(ResourceLoader) uses the one you pass.
• OpenNLP factories: OpenNLPTokenizerFactory (openNlp, sentence and tokenizer models), OpenNLPPOSFilterFactory (openNlppos), OpenNLPChunkerFilterFactory (openNlpChunker) and OpenNLPLemmatizerFilterFactory (openNlpLemmatizer, a lemma dictionary and/or a lemmatizer model). All live in the opt-in analysis-opennlp module.
• OpenNLPOpsFactory: the module's model cache. Its class javadoc says only "Supply OpenNLP Named Entity Recognizer Cache model file objects. Assumes model files are thread-safe." (tools/OpenNLPOpsFactory.java:33-36).

1. OpenNLPOpsFactory keeps seven static maps (sentence, tokenizer, POS, chunker, NER and lemmatizer models, lemma dictionaries) keyed only by the resource name (:38-48).
2. Each loader-taking getter reads from the loader only on a miss. getLemmatizerDictionary(dictionaryFile, loader) (:156-166) does lemmaDictionaries.get(dictionaryFile); only if that is null does it call loader.openResource(dictionaryFile) and put the result. On a hit the loader is never used. The model getters are the same (:59-69, :80-90, :97-107, :114-124, :131-141, :168-178).
3. The factories' inform(loader) only fills the maps ("register models in cache with file/resource names", OpenNLPLemmatizerFilterFactory.java:97), and create() fetches by name with no loader (OpenNLPOpsFactory.getLemmatizer(dictionaryFile, lemmatizerModelFile), :88). The tokenizer, POS and chunker factories do the same. No factory keeps the object its own loader produced.
4. clearModels() (:180-188) is the only way to empty the maps; it is public, but nothing in Lucene calls it. So the first object loaded under a name stays for the life of the JVM, unless the application calls it.
5. With configurations built one after another, as in the repro, the first one decides. The getters do get, load, then put, which is not atomic: if two configurations load the same name at the same moment, both read their own file and the last put wins (the populate race committer magibney noted on PR #380).

• Lemma dictionary: A's lemmas.txt is dogs<TAB>word<TAB>dogA, B's is dogs<TAB>word<TAB>dogB (the whitespace tokenizer's type word serves as the POS tag). B's analyzer emits dogA; A's emits dogA, as it should. Main with assertions on and off, 10.6.0, 10.5.2, 10.0.0, 9.12.3 and 9.1.0.
• Tokenizer model, two real directories via CustomAnalyzer.builder(Path), each with a valid sent.bin and tok.bin: B's model keeps periods attached ([Hello, world., This, is, a, test.], 6 tokens, its output with an empty cache); after A is built, B's analyzer emits A's [Hello, world, ., This, is, a, test, .], 8 tokens. 13 releases from 7.3.0 to 10.5.2 and main.
• Missing file: a directory without lemmas.txt throws IOException when built first, but builds without error, using A's dictionary, once A has loaded that name.

• Who is exposed. Only processes that build OpenNLP analyzers from more than one configuration, and only when two configurations ship different contents under the same file name: for example a retrained model kept under its old name in a new configuration, or per-language directories that reuse generic names such as tok.bin or lemmas.txt. The Solr reference guide's examples prefix the model names with the language (en-sent.bin, en-tokenizer.bin), which makes model collisions less likely; its three OpenNLP lemmatizer examples, however, name the dictionary plain lemmas.txt, the name this repro uses (and the name in the LUCENE-10171 report), so configurations copied from the guide share the dictionary name.
• Silent and lasting. No error and no log line. The second configuration indexes documents with the first one's tokens, tags or lemmas and analyzes its queries the same way, until the JVM restarts or the application calls clearModels(); the documents keep the wrong terms until reindexed.
• Missing files are masked. A configuration without the file builds if another one loaded that name first, and fails if it is built first, so a broken configuration can pass in one process and fail in another, depending on load order.
• Scale. Ordinary: one shared file name and two configurations. No unusual parameter is needed.
• Same pattern, by reading only (not executed): Word2VecSynonymProviderFactory in analysis-common keeps a static map keyed by the model file name and reads from the loader only on a miss.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<dog[B]> but was:<dog[A]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:573)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:637)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:792)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:777)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:862)
       org.apache.lucene.analysis.opennlp.TestOpenNLPLemmaDictionarySecondLoader.testSameDictionaryNameFromSecondLoader(TestOpenNLPLemmaDictionarySecondLoader.java:31)

• What users read. CustomAnalyzer.builder(Path): "loads all resources from the given file system base directory" (CustomAnalyzer.java:109-116); builder(ResourceLoader): "loads all resources using the given ResourceLoader" (:118-124); ResourceLoaderAware.inform: "Initializes this component with the provided ResourceLoader" (ResourceLoaderAware.java:29-33; weak on its own, it promises no isolation).
• What the module says. The OpenNLPOpsFactory class javadoc mentions a cache of model objects, but not its key, its JVM-wide scope or that the loader is ignored on a hit. The factory javadocs, the module overview and package docs, CHANGES.txt, MIGRATE.md and the Solr reference guide's OpenNLP section say nothing about it.
• The input is legal. Public API, SPI names and valid files; the cross-model reviewer rated the inputs in contract, and no checker argued misuse or a test artifact.
• What kind of defect. The mechanism is intended, so the defect is the missing documentation (a content-keyed cache is an optional follow-up), not a violation of the ResourceLoaderAware contract.

The intended-behaviour checker built the strongest case for "by design": (1) in LUCENE-2899 (#3973, 2012) the module's contributor wrote "the statistical models are several megabytes apiece. This class loads them and caches them by file name. It does not reload them across commits." (2) Every factory's inform() has a comment saying it registers its models in the cache under their file/resource names ("register models in cache with file/resource names" in the tokenizer and lemmatizer factories; "load and register the read-only model in cache with file/resource name" and "load and register read-only models in cache with file/resource names" in the POS and chunker factories). (3) The public op getters (getSentenceDetector(String), getTokenizer(String), ...) take only a name; upstream's TestOpenNLPSentenceBreakIterator registers through a loader and fetches by name, and Solr's NER update processor does the same. (4) Users rely on sharing across collections: LUCENE-10171 (#11208) reported out-of-memory errors when lemmatizer copies were not shared. (5) Word2VecSynonymProviderFactory (#12169, 2023) reused the pattern.

Why it fails for this claim: it shows that caching by name is deliberate, which is why the claim is a documentation defect and not a code bug. None of it addresses two unrelated configurations: the 2012 comment is about not reloading a changed file under one configuration, it lives in a JIRA comment, and no user-facing document states the JVM-wide scope or the masking of a missing file. Verdict: real as a documentation defect (high confidence that the mechanism is by design and that nothing documents its scope; moderate that upstream accepts a documentation change).

Two fixes were built and validated through gradle in a scratch worktree of 645d4fe14a20: the documentation change, which upstream is most likely to accept because the cache is deliberate, and a content-keyed cache that removes the cross-configuration effect while keeping the sharing. Two other options were weighed and not recommended.

Javadoc only, +21 lines in 5 files. The OpenNLPOpsFactory class javadoc gets the note below; each of the four factory javadocs gets one sentence: "Model and dictionary files are cached for the whole JVM by resource name, see {@link OpenNLPOpsFactory}."

 * <p><b>NOTE:</b> models and lemma dictionaries are cached in static maps that are shared by the
 * whole JVM and keyed only by their resource name. The {@link ResourceLoader} passed to a {@code
 * get...Model} method or to {@link #getLemmatizerDictionary} is used only the first time a name is
 * requested: a later request for the same name, through any loader, returns the cached object
 * without reading the resource. So all analyzers in a JVM that name the same file, for example
 * analyzers built from different configuration directories, share the object that was loaded first,
 * and a changed or missing file is not noticed. Give each distinct model or dictionary its own
 * resource name.

Trade-offs

Closes the documentation gap only: the cross-configuration effect and the masking of a missing file remain, now documented. Solr users would also need a sentence in the Solr reference guide's OpenNLP section, which is Solr's documentation.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31).
• :lucene:analysis:opennlp:test: 34 tests, 1 failure, the repro, as expected (the fix changes no behaviour); the module's 33 upstream tests pass.
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest and renderJavadoc: pass.

Every loader-taking getter in OpenNLPOpsFactory goes through one helper that always reads the resource, keys the shared object by type, name and the SHA-256 of the bytes, and still records the first object per name for the public name-only getters. The four factories store what inform(loader) returns and build their ops from it in create() instead of fetching by name. 5 files, +85/-80.

private static <T> T load(Map<String, T> byName, String name, ResourceLoader loader,
    Class<T> type, IOFunction<InputStream, T> parser) throws IOException {
  byte[] bytes;
  try (InputStream resource = loader.openResource(name)) {
    bytes = resource.readAllBytes();
  }
  String key = type.getName() + '/' + name + '/' + sha256(bytes);
  T model = type.cast(byContent.get(key));
  if (model == null) {
    model = parser.apply(new ByteArrayInputStream(bytes));
    byContent.put(key, model);
  }
  byName.putIfAbsent(name, model); // the name-only getters keep returning the first object
  return model;
}

Trade-offs

• Keeps the sharing the cache exists for: identical files under the same name still share one object.
• Every inform() reads and hashes the whole file, even when the object is cached (today a hit reads nothing); the cost per analyzer build was not measured.
• A configuration without the file now fails at build time, as it already does when built first: a behaviour change for setups that relied on the masking.
• A changed file under the same name (for example after a configuration reload) is now picked up; the old object stays cached until clearModels(), like every cached object today.
• The public name-only getters (getTokenizer(String), getNERTagger(String), ...), used by Solr's NER update processor, keep first-loaded-wins; fixing them needs an API change.
• Like today's code, the helper does get, parse, then put, which is not atomic: two concurrent first loads of the same file can parse it twice and leave two factories with different, equal objects. byContent is never trimmed: it keeps one object per distinct file version for the life of the JVM (until clearModels()), so repeated reloads with changed files grow it. No concurrency or memory test was run.
• Side effect: the lemma dictionary's input stream is now closed, which also fixes a separate stream leak in the same method (a 9.1.0 regression of the LUCENE-8188 fix).

What the checks validated

• Same worktree and settings. :lucene:analysis:opennlp:test: 34 tests, 0 failures, the repro included; the repro with -Ptests.asserts=false: passes; the same six format, lint and javadoc tasks: pass.
• The verifiers' wider tests, copied in temporarily: the two-directory tokenizer-model test and the same-named-dictionary and missing-file tests (the second directory without the file now throws IOException): 3 tests, 0 failures; on stock the same classes give 4 failures.

The finder's sketch, for the tokenizer factory only: read the models from the factory's own loader in inform() and keep them, bypassing OpenNLPOpsFactory.

Trade-offs

Gives up the cross-field and cross-collection sharing of multi-megabyte models that the cache exists for (LUCENE-10171 reported out-of-memory errors without it); the adjudication and the devil's-advocate verifier advise against it.

What the checks validated

The finder's repro and TestOpenNLPTokenizerFactory (4 of 4) pass with it.

Use (loader, name) as the key.

Trade-offs

ResourceLoader defines no identity or equals, and CustomAnalyzer.builder(Path) creates a new FilesystemResourceLoader on every call, so two analyzers built from the same directory would no longer share models.

What the checks validated

Not built.

Method: non-formal (code reading, then executed tests).

• A hunt unit on the analysis factory chain audited the inform() and create() of every in-scope analysis factory for the signal "the component used differs from the configured one". The OpenNLP factories register their files through the loader in inform() but fetch by name in create(), and OpenNLPOpsFactory reads the loader only on a cache miss.
• A probe on the stock jars confirmed the signal, then a public-API JUnit with two valid, behaviourally different tokenizer models (one trained in the test). A first cross-model review objected to an earlier version that used malformed model files, so the repro was rewritten with valid models.
• An independent reproducer wrote its own two-directory test with a fresh-cache oracle and a lemma-dictionary variant; a cross-model adversary, reading only, agreed with the root cause, and separate runs for that review added the missing-file case; a devil's-advocate verifier built the "by design" case; the adjudication re-ran the independent test. The repro was later minimized to the one-test lemma-dictionary form.