A failed open of a persistent Monitor leaves its query index write-locked until the JVM restarts

new Monitor(analyzer, config) with a query index on disk (MonitorConfiguration.setIndexPath or setDirectoryProvider) builds a WritableQueryIndex. Its constructor first builds an IndexWriter (WritableQueryIndex.java:77), which takes the index's write.lock, and only then opens a SearcherManager (:79), re-parses every stored query (:83) and schedules the purge task (:85-98). Nothing after :77 is inside a try, so when any of those steps throws, the constructor throws with the writer still open. The caller gets no Monitor, so close(), the only release (:322-325), can never run, and NativeFSLockFactory keeps the lock in a static JVM-wide set until the lock is closed. With that default lock factory, every later writable Monitor on that path in the JVM then fails with LockObtainFailedException ("Lock held by this virtual machine"), even with a correct configuration. Four triggers were executed: stored queries the current parser rejects, a non-empty index opened without a serializer, an I/O error while the index opens, and setPurgeFrequency(0, ...). Nothing on disk is lost, and read-only Monitors and other paths keep working. IndexWriter's own constructor releases its lock on any failure; this one never did. Reproduced on nine releases from 8.2.0 to 10.6.0 and on main; garbage collection frees the writer but not the JVM's record of the lock. Low priority: it needs a failed open of an opt-in persistent Monitor and a retry in the same JVM.

// WritableQueryIndex(MonitorConfiguration, Presearcher), condensed
this.writer = configuration.buildIndexWriter();              // :77  new IndexWriter: takes write.lock
this.manager = new SearcherManager(writer, true, true, ...);   // :79  opens a reader: I/O error -> leak
...
populateQueryCache(serializer, decomposer);                  // :83  IllegalStateException -> leak
long purgeFrequency = configuration.getPurgeFrequency();
this.purgeExecutor = Executors.newSingleThreadScheduledExecutor(...);
this.purgeExecutor.scheduleAtFixedRate(..., purgeFrequency, purgeFrequency, ...); // :88  period <= 0 -> leak
// no try/catch anywhere after :77

1. MonitorConfiguration.buildIndexWriter() (MonitorConfiguration.java:93-95) returns new IndexWriter(directoryProvider.get(), ...); setIndexPath opens a fresh FSDirectory for each Monitor (:87-91). IndexWriter obtains write.lock (IndexWriter.java:954). If IndexWriter's own constructor fails, it releases the lock itself (IndexWriter.java:1202-1209, "init: hit exception on init; releasing write lock").
2. Every exception after :77 escapes with the writer open. Four triggers, each executed:
   • I/O error while the SearcherManager opens the index (:79), with a correct serializer and parseable queries (a FilterDirectory that throws on the doc-values file).
   • setPurgeFrequency(0, ...): MonitorConfiguration.java:130-134 stores any value without validation, and scheduleAtFixedRate throws IllegalArgumentException for a period <= 0, even on an empty index.
   • Stored queries the serializer rejects: populateQueryCache collects every parse error and throws IllegalStateException("Couldn't parse some queries from the index") (:173-180). This is the minimal repro.
   • A non-empty index with no serializer: IllegalStateException("Attempting to open a non-empty monitor query index with no MonitorQuerySerializer") (:139-150; non-empty means at least one live query document).
3. The exception leaves Monitor's constructor too (Monitor.java:87-99). The only releases, WritableQueryIndex.close() (:322-325: shut the executor down, then close the manager, the writer and the directory) and Monitor.close() (Monitor.java:142-145), belong to objects the caller never received.
4. NativeFSLockFactory adds the lock path to its static LOCK_HELD set when it obtains the lock (NativeFSLockFactory.java:68, :108) and removes it only when that lock is closed (:145-150); a later attempt throws "Lock held by this virtual machine" (:119). Garbage collection does not help: it frees the unreachable writer and closes its lock file channel, but the path stays in LOCK_HELD, so the retry still fails (an independent verifier's test, five System.gc() rounds; a later probe, ten rounds, with a weak reference showing the writer was collected). One undocumented workaround: subclass MonitorConfiguration, override the public buildIndexWriter() to keep the writer, and close it yourself. Another, also undocumented and executed: a setDirectoryProvider whose FilterDirectory records the Lock from obtainLock; closing that Lock after the failed open lets the retry open with its 1 query, but it releases only the lock, not the writer or its reader.

By reading only, not claimed: the default in-memory Monitor (MonitorConfiguration.java:41, a new ByteBuffersDirectory per open) shows no lock error, because every open gets its own directory.

• Who. Applications of the optional lucene-monitor module that keep the query index on disk. This is opt-in: "By default, Monitor instances are ephemeral, storing their query indexes in memory" (package-info.java:99-100).
• When. An open fails (most realistically an I/O error, or stored queries that a new version of the application's parser no longer accepts) and the application then retries or reopens in the same JVM instead of exiting: a long-running service that reloads its configuration, falls back to a lenient parser, or opens the index writable to delete the bad query; or a test suite. A startup failure followed by a process restart never sees it. Likelihood is rated rare: no report since 8.2.0 (2019).
• What goes wrong (executed probe on main with assertions on and off, 10.5.2 and 9.12.3): every writable open of that path fails until the JVM restarts; 200 retries all failed and added 0 file descriptors. The message says the lock is held, not why, so it looks like a second writer the application does not have. Each failed open leaves an unreachable IndexWriter with its lock file channel and, except for the I/O-error trigger (where new SearcherManager itself throws), a SearcherManager with its reader. Garbage collection frees the writer and closes the channel, but the path stays recorded as locked in the JVM, and the reader's index-file mapping stays (seen for the parser trigger), until a restart.
• What it does not do (same probe): nothing on disk is lost or corrupted (a plain DirectoryReader sees all 3 stored queries); a read-only Monitor on the same path loads all 3 and matches exactly the right ones; Monitors on other paths and in-memory Monitors keep working.

One test class, one method. It starts no threads of its own (each Monitor it opens successfully starts its purge thread, as every Monitor does), and the lock failure reproduced with every seed and assertion mode tried. It sits in org.apache.lucene.monitor next to the module's own tests but uses only public API (Monitor, MonitorConfiguration.setIndexPath, MonitorQuerySerializer.fromParser, MonitorQuery, TermQuery, WhitespaceAnalyzer; the test framework's createTempDir and expectThrows). The rejecting parser models a parser change between application versions. The retry is wrapped only to turn the LockObtainFailedException into a message stating expected vs actual.

The suite-level "file handle leaks" failure is the test framework seeing the same leaked writer's lock channel, not a second bug; it does not appear on 10.5.2 or with every seed, while the lock failure itself reproduced with every seed and version tried.

This same test is the suggested regression test: it fails on stock main and passes with the fix. It passes the module's format, ECJ and forbidden-apis checks unchanged.

# gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true (module prefixes, environment lines and framework frames cut)
TestMonitorReopenAfterFailedOpen > testReopenAfterRejectedStoredQuery FAILED
    java.lang.AssertionError: expected the retry with the working serializer to open the index, but got org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: $LUCENE/lucene/monitor/build/tests-tmp/lucene.monitor.TestMonitorReopenAfterFailedOpen_001/tempDir-001/write.lock
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:5894959657BDB0E2]:0)
        at org.apache.lucene.monitor.TestMonitorReopenAfterFailedOpen.testReopenAfterRejectedStoredQuery(TestMonitorReopenAfterFailedOpen.java:51)
        Caused by:
        org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: $LUCENE/.../tempDir-001/write.lock
            at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:119)
            at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
            at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
            at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:954)
            at org.apache.lucene.monitor.MonitorConfiguration.buildIndexWriter(MonitorConfiguration.java:94)
            at org.apache.lucene.monitor.WritableQueryIndex.<init>(WritableQueryIndex.java:77)
            at org.apache.lucene.monitor.Monitor.<init>(Monitor.java:95)
            at org.apache.lucene.monitor.Monitor.<init>(Monitor.java:77)
            at org.apache.lucene.monitor.TestMonitorReopenAfterFailedOpen.open(TestMonitorReopenAfterFailedOpen.java:32)
            at org.apache.lucene.monitor.TestMonitorReopenAfterFailedOpen.testReopenAfterRejectedStoredQuery(TestMonitorReopenAfterFailedOpen.java:48)
TestMonitorReopenAfterFailedOpen > executionError FAILED
    java.lang.RuntimeException: file handle leaks: [FileChannel($LUCENE/.../tempDir-001/write.lock)]
        at org.apache.lucene.tests.mockfile.LeakFS.onClose(LeakFS.java:63)
        ...

2 tests completed, 2 failed

# the same two failures with -Ptests.asserts=false; the expectThrows on line 47 passes (the rejected open fails first)

• The codebase rule. IndexWriter's constructor catches any Throwable, including configuration it rejects, and releases its lock (IndexWriter.java:1202-1209). It was added as a bug fix: LUCENE-2476, "release write lock on any exception during IndexWriter ctor" (2010).
• Ownership. WritableQueryIndex.close() releases exactly what the constructor acquires (executor, searcher manager, writer, directory; WritableQueryIndex.java:322-325). A caller of a failed constructor has no object to close and no unlock API.
• The documented use is reopening. package-info.java:104-107: stored queries must "be re-parsed by the associated MonitorQuerySerializer when the index is loaded by a new Monitor instance".
• Two triggers break no precondition. An I/O error is an ordinary failure the constructor declares; setPurgeFrequency documents only "the frequency value" and accepts 0 (MonitorConfiguration.java:123-134).
• Nothing says otherwise. No javadoc, test, CHANGES.txt or MIGRATE.md entry says a failed open keeps the lock; the upstream monitor tests never retry after a failed open, and their purge tests use only positive frequencies. The two IllegalStateException messages are code-level validation errors, in no javadoc and no upstream test. The constructor shape came with the module (LUCENE-8766, 2019) and survived the 2022 refactor (LUCENE-10422) without any discussion of failure cleanup.

The strongest "works as intended" case (built by a devil's-advocate verifier and by cross-model reviews, OpenAI GPT-5.6): the package docs make re-parseability of stored queries the application's job, so the parser and null-serializer triggers start from an application error; scheduleAtFixedRate documents IllegalArgumentException for a non-positive period, so 0 is invalid configuration; no javadoc promises that a failed Monitor construction can be retried; Closeable covers only constructed objects; the lock exists to stop two writers; and a restart clears it.

Why it fails: those points justify the first exception, not keeping a native lock on a writer nobody can reach. IndexWriter releases its lock even for configuration it rejects; the I/O-error trigger breaks no precondition at all (it is the strongest case; the parser and null-serializer triggers do start from an application error); the "second writer" is unreachable garbage, and even once garbage collection frees it the JVM still records the lock as held; and a restart is a workaround. The devil's-advocate verifier rated "not intended" high; the earlier cross-model reviews rated the intended argument moderate; a later cross-model review that ran its own repro rated the bug real with high confidence.

Option A was run through gradle in a scratch worktree of 645d4fe14a20, and against the other three triggers' tests. B complements it; C is the narrower first fix and shows why the guarded region must reach the end of the constructor. Building the writer later is not a fix: populateQueryCache reads the stored queries through the searcher manager built on the writer.

In WritableQueryIndex.java:77-98: the rest of the constructor moves, unchanged and re-indented, into the try; the executor goes through a local because a blank final field cannot be read in the catch. Ignoring whitespace the change is +14/-3.

this.writer = configuration.buildIndexWriter();
ScheduledExecutorService executor = null;
try {
  // ... SearcherManager, populateQueryCache: unchanged ...
  executor = Executors.newSingleThreadScheduledExecutor(new NamedThreadFactory("cache-purge"));
  executor.scheduleAtFixedRate(/* unchanged arguments */);
  this.purgeExecutor = executor;
} catch (Throwable t) {
  if (executor != null) {
    executor.shutdown();
  }
  IOUtils.closeWhileSuppressingExceptions(t, manager, writer::rollback, writer.getDirectory());
  throw t;
}

Trade-offs

• Covers all four triggers, because the guarded region includes the executor creation and scheduling.
• Roll back, not close: IndexWriter.close() commits by default (IndexWriter.java:1363-1369); rollback() commits nothing, so a failed open leaves the index as it was (a failed first open of an empty directory creates no commit).
• Same resources and order as close() (WritableQueryIndex.java:322-325). Like close(), it closes the directory the provider returned; for a setDirectoryProvider that hands out a shared Directory that is the existing ownership rule, but a caller who retries with that same instance would then find it closed (the provider API does not say who owns the directory when the open fails).
• The original exception is rethrown, with any cleanup failure suppressed into it, the same idiom as IndexWriter.java:1202-1209; only if a cleanup step throws an Error (and the original is not one) does the helper throw that Error instead, with the original suppressed (IOUtils.java:116-136). The helper exists from 10.4.0 on; a 9.x backport would use IOUtils.closeWhileHandlingException (not run).

What the checks validated

• Stock 645d4fe14a20: the repro fails with assertions on and off.
• With the fix: the repro plus TestMonitorPersistence, TestMonitorReadonly and TestCachePurging pass with assertions on and off (4 suites, 15 tests, 0 failures); the whole :lucene:monitor:test passes with assertions on (33 suites, 182 tests, 0 failures, 1 skipped); checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass.
• The other triggers, against the built main jars with the fixed class shadowed: the purge-frequency-0 check (stock fails, fixed passes), the I/O-error and garbage-collection tests (stock 2 fail, control passes; fixed 3 of 3), the null-serializer and rejected-query tests (stock 2 fail; fixed 2 of 2); no leaked-file report with the fix.

Reject frequency <= 0 in MonitorConfiguration.setPurgeFrequency (or at the start of the constructor) with a clear message.

Trade-offs

• Covers only the purge-frequency trigger and gives a better error; the other three still need A. Open issue #11710 asks for a way to disable the scheduled purge, so maintainers may prefer 0 to mean "disabled".

try/finally around SearcherManager creation and populateQueryCache, with IOUtils.closeWhileHandlingException(manager, writer, writer.getDirectory()) on failure.

Trade-offs

• Passes the minimal repro, but stops before the executor, so setPurgeFrequency(0, ...) still leaks the lock with it (executed twice, by reviewers); it also closes the writer, which commits by default (commitOnClose), instead of rolling it back.

Method: code reading (non-formal), in a hunt unit on the monitor module's lifecycle and persistence whose secondary focus was serializer edge cases, such as a serializer that throws on one query.

• While modelling the persistence lifecycle for operation-level model-based tests, the finder read the constructor's serializer error path and saw that nothing closes the writer built before it.
• A public-API JUnit test (null serializer; a parser that rejects the stored string, then a retry) failed 2 of 2 on main with assertions on and off, 10.5.2 and 9.12.3.
• Independent verifiers added the I/O-error trigger and the purge-frequency trigger; the latter showed the first fix draft was too narrow.