ICUTokenizerFactory leaves a custom rule file open when reading or decoding it fails

ICUTokenizerFactory can load custom ICU break rules per script from files (rulefiles="Latn:latin.rbbi"). In inform(), parseRules opens each file through the ResourceLoader, reads it line by line through Lucene's strict UTF-8 decoding reader, and calls reader.close() only after the read loop has finished normally. When the file is not valid UTF-8 (for example saved as ISO-8859-1 with é in a comment) or the stream throws an IOException mid-read, inform() fails with that exception, as it should, but the close is skipped and the stream stays open until it is garbage-collected. With FilesystemResourceLoader that is an open file handle, which Lucene's own LeakFS test filesystem reports as a leak. The default configuration (no rulefiles) opens no rule file through the ResourceLoader, a rule file that decodes but does not compile is closed before compiling, and tokens and the index are unaffected. It has been there since rulefiles arrived in 4.1.0; a try-with-resources around the stream and the reader fixes it.

• rulefiles: a comma-separated list of code:rulefile pairs, a four-letter ISO 15924 script code and a resource path, each compiled by ICU's RuleBasedBreakIterator (class javadoc, ICUTokenizerFactory.java:49-66 at 645d4fe14a20).
• Decoding reader: IOUtils.getDecodingReader(stream, UTF_8) reports malformed input instead of replacing it (CodingErrorAction.REPORT, IOUtils.java:192-199); its javadoc says it is "useful to load configuration files, stopword lists or synonym files to detect character set problems".

1. inform() calls parseRules(resourcePath, loader) once per pair (ICUTokenizerFactory.java:116-120, call at :119); without rulefiles it opens nothing through the ResourceLoader (:111-112; the bundled Default.brk and MyanmarSyllable.brk come from the jar, read once by DefaultICUTokenizerConfig's static initializer, DefaultICUTokenizerConfig.java:81-84, :144-153).
2. parseRules opens the stream (loader.openResource(filename), :139), wraps it in a BufferedReader over the decoding reader (:140-141), reads line by line (:143-148) and calls reader.close() at :149, with no try/finally. It then compiles the rules (:150).
3. A byte that is not valid UTF-8 makes readLine() throw MalformedInputException; a failing stream makes it throw IOException. Either leaves the method before :149.
4. Within Lucene, parseRules is the only holder of the stream and inform() returns nothing, so no Lucene code can close it later (the stock FilesystemResourceLoader and ClasspathResourceLoader keep no reference either). What it holds is released only if the stream's own cleanup runs when the garbage collector reclaims it: for a real file, the probe saw the descriptor released after a forced System.gc(). With several pairs, the files read before the failing one were already closed at :149; only the failing stream stays open.

Fault injection on a 76-byte valid rule file: an IOException at each of the 77 read positions (offsets 0-76) and an 0xFF byte at each of offsets 0-75 all leak (153 of 153), as does a failing second rule file; a failing openResource and a file that does not compile do not. With the fix, 0 of 157 runs leak.

• Who is hit. Users of the analysis-icu module who opt in to rulefiles: on the factory directly, through CustomAnalyzer with the SPI name icu, or through a search-server schema (the class javadoc's examples are such schemas). The default factory, without rulefiles, opens no rule file through the ResourceLoader.
• What it takes. A rule file that cannot be read to the end: not valid UTF-8 (a file saved in a legacy encoding such as ISO-8859-1, where one non-ASCII character anywhere, even in a comment, becomes a byte that is not valid UTF-8) or an I/O error while reading. No scale, no unusual parameters; the same with assertions on and off.
• What happens. inform() fails loudly, as it should, and one InputStream per failed load stays open until it is garbage-collected: one file handle with FilesystemResourceLoader, one resource stream with ClasspathResourceLoader. An application that keeps retrying a bad configuration leaks one more each time until the next GC: five failed calls on a real file left 1, 2, 3, 4, 5 descriptors open, and 0 only after System.gc() (0 after every call with the fix). Lucene's LeakFS test filesystem fails a test that does this.
• What does not happen. No wrong tokens, no index change, no data exposure. Not measured and not claimed: running out of file descriptors, or an open handle blocking replacement of the file on platforms that lock open files.
• How likely. Rare: no report since 4.1.0, and the leak hides behind the loud setup failure.

The test lives in the factory's package only to sit next to TestICUTokenizerFactory; it uses public API only (ICUTokenizerFactory, ResourceLoader). It imports org.apache.lucene.tests.util.LuceneTestCase, so as written it compiles on 9.1.0 and later; older releases were checked with a release-agnostic main() harness. The same test is the suggested regression test.

It checks only the contract-backed behaviour: inform() fails with the decoder's CharacterCodingException (unchanged by the fix) and the stream it opened is closed. The I/O-error variant and the real-file LeakFS test have the same root cause and are left out of the repro.

cp $REPRO/src/org/apache/lucene/analysis/icu/segmentation/TestICUTokenizerFactoryRuleFileClosed.java $LUCENE/lucene/analysis/icu/src/test/org/apache/lucene/analysis/icu/segmentation/
cd $LUCENE && ./gradlew :lucene:analysis:icu:test --tests org.apache.lucene.analysis.icu.segmentation.TestICUTokenizerFactoryRuleFileClosed -Ptests.seed=DEADBEEF
# with this seed assertions are on; add -Ptests.asserts=false to turn them off: same result

# gradle, main 645d4fe14a20, seed DEADBEEF (assertions on, the seed's default); the same with -Ptests.asserts=false; verbatim excerpt
TestICUTokenizerFactoryRuleFileClosed > testRuleFileClosedWhenNotUtf8 FAILED
    java.lang.AssertionError: expected inform() to close the rule file stream it opened after failing; actual: left open
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:981894AD4EC96D8F]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.analysis.icu.segmentation.TestICUTokenizerFactoryRuleFileClosed.testRuleFileClosedWhenNotUtf8(TestICUTokenizerFactoryRuleFileClosed.java:54)
:lucene:analysis:icu:test (FAILURE): 1 test, 1  failure

# with the fix (assertions on and off)
:lucene:analysis:icu:test (SUCCESS): 1 test

# supporting checks, not part of the repro: a real file under LeakFS (independent reproducer, -ea and -da)
java.lang.RuntimeException: file handle leaks: [InputStream(.../icu-rules-001/latin.rbbi)]

# five inform() calls on a real file: open descriptors on latin.rbbi after each try, then after System.gc()
FD name=malformed loader=FilesystemResourceLoader tries=5 exc=java.nio.charset.MalformedInputException openAfterEachTry=1,2,3,4,5 afterGc=0
FD name=control loader=FilesystemResourceLoader tries=5 exc=none openAfterEachTry=0,0,0,0,0 afterGc=0

• Ownership. parseRules opens the stream (ICUTokenizerFactory.java:139), is the only Lucene code that holds it and closes it after a successful read (:149). As Uwe Schindler put it on LUCENE-4165 (#5237): "of course we close streams, but only if we open them". Here Lucene opens it.
• A designed failure path. IOUtils.getDecodingReader reports charset problems on purpose: "Decoding readers are useful to load configuration files, stopword lists or synonym files to detect character set problems" (IOUtils.java:185-186). A rule file in the wrong charset is an expected outcome of the reader parseRules chose.
• The sibling loaders close on failure, with one known exception. WordlistLoader.getLines (the stopword loader, the same decoding reader, try-with-resources at WordlistLoader.java:297-299), AbstractAnalysisFactory.getSnowballWordSet (:322), JapaneseTokenizerFactory (:144), the Korean, Morfologik, Word2Vec, Hunspell and hyphenation factories, and the six OpenNLP model loaders. The exception is OpenNLP's lemma dictionary: OpenNLPOpsFactory.getLemmatizerDictionary hands the stream to OpenNLP's DictionaryLemmatizer (OpenNLPOpsFactory.java:162), which never closes it, on success or failure; that is a separate leak, LFV-461 (LUCENE-10171 undid the LUCENE-8188 fix there). An executed probe with injected read failures over 18 analysis-factory configurations (OpenNLP not among them) found leaks only in the ICU rulefiles runs (5 of 54 runs).
• Accepted bug fixes of the same class (lucene/CHANGES.txt, Bug Fixes): LUCENE-7547 "JapaneseTokenizerFactory was failing to close the dictionary file it opened" (6.4.0, 5.5.4); LUCENE-8188, OpenNLPOpsFactory "InputStreams fetched from the ResourceLoader to be leaked" (7.3.0); GITHUB#14727 "Fix several resource leaks" (10.3.0), which replaced a close() at the end of the success path with try-with-resources in smartcn's BigramDictionary and WordDictionary.
• Tests and history. Upstream TestICUTokenizerFactory has 5 tests; the three that load rule files use valid ones, and none covers a failing one. parseRules is unchanged in behaviour since SOLR-4123 (2012), whose 21-comment thread covers the rulefiles syntax, resource paths and rule syntax errors but never mentions closing the stream or a failed read.

The strongest case for "intended" (built by reviewers): no javadoc says who closes the stream after a failed inform() (ResourceLoader.openResource says only "Opens a named resource"); invalid UTF-8 is a configuration error that already fails loudly; the JDK reclaims the handle at GC; and DefaultICUTokenizerConfig.readBreakIterator closes Lucene's bundled .brk files the same close-on-success way (DefaultICUTokenizerConfig.java:144-153).

Why it fails. The contract's silence is answered by ownership: Lucene opened the stream and no Lucene code can reach it after the failure. GC timing is unbounded, which is why Lucene's own test framework fails tests that leave handles open. The sibling loaders and three accepted bug fixes show the project's intent. The bundled-resource reader handles known-good resources inside Lucene's own jar, so it is a side note (optional hygiene, not part of this claim), not a counter-example.

One option was run through gradle in a scratch worktree of 645d4fe14a20; the alternative is the same idea in older style.

Move the read loop into try (InputStream rulesStream = loader.openResource(filename); BufferedReader reader = ...) { ... } and drop the explicit reader.close(); the rules are compiled after the block, as before. 9 lines added, 9 removed, ICUTokenizerFactory.java only.

-    InputStream rulesStream = loader.openResource(filename);
-    BufferedReader reader =
-        new BufferedReader(IOUtils.getDecodingReader(rulesStream, StandardCharsets.UTF_8));
+    try (InputStream rulesStream = loader.openResource(filename);
+        BufferedReader reader =
+            new BufferedReader(IOUtils.getDecodingReader(rulesStream, StandardCharsets.UTF_8))) {
       // read loop unchanged, indented one level
     }
-    reader.close();
     return new RuleBasedBreakIterator(rules.toString());

Trade-offs

• Success path unchanged: the stream is still closed before ICU compiles the rules, so a rule file that does not compile still leaves nothing open.
• On failure the caller sees the same exception, with any exception from close() attached as suppressed; on success a failing close() propagates, as the explicit close() at :149 does today. The stream is its own resource, so it is also closed if wrapping it in the decoding reader fails. Closing the stream after the reader is a harmless second close.
• The same idiom as WordlistLoader.getLines and GITHUB#14727. LFV-719 (a byte order mark kept as rule text) changes the same loop; the two fixes do not conflict.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (2026-11-01): the repro fails on stock and passes with the fix, with assertions on and off.
• The whole :lucene:analysis:icu:test task with the repro in place: 17 test classes, 159 tests (10 skipped), 0 failures, including the 5 TestICUTokenizerFactory tests.
• :lucene:analysis:icu:check -x test (google-java-format, ecj lint, forbidden APIs) passes with the fix and the test in place.
• Earlier checks: the finder's 4 tests pass, fault injection finds 0 leaks in 157 checks (156 fault points and a fault-free baseline), and five failed tries on a real file leave 0 descriptors open.

Keep the code as is and close the reader (or the stream) in a finally block, for example with IOUtils.closeWhileHandlingException.

Trade-offs

Same effect in older style, with more lines; no reason to prefer it. Care is needed: a plain finally with IOUtils.close lets a failing close replace the read exception, so closeWhileHandlingException belongs on the failure path only (the usual success flag idiom). Not run.

Method: non-formal first signal, then formal characterization. On 2026-10-28 an exception-safety probe (FactoryInformLeakProbe) gave 18 analysis-factory configurations a close-tracking ResourceLoader with a read failure injected at byte 0 or 3 and a malformed file: 54 runs, 5 leaking, all on ICUTokenizerFactory rulefiles read or decode failures; the other configurations and an ICU rule file that does not compile closed every stream. The hunt unit then wrote a TLA+ model of the load's resource lifecycle (TLC: 14 traces break the release invariant, all at the read step; the fixed variant holds) and ran model-based fault injection on the real jar (157 checks, 156 fault points and a fault-free baseline: 154 leak, 0 mismatches with the model, 0 leaks with the fix), and reduced it to a public-API JUnit test.