Monitor.match can throw NullPointerException while queries are being registered

QueryIndex keeps, for every open searcher over the Monitor's query index, a QueryTermFilter in termFilters, a plain java.util.HashMap (QueryIndex.java:56). Each register, deleteById or clear commits and refreshes the searcher, and the refresh puts the new reader's filter into the map (TermsHashBuilder.java:38); a read-only Monitor does the same when its refresh runs. Monitor.match reads the map with no lock (WritableQueryIndex.java:221-223, ReadonlyQueryIndex.java:80-82). When a put grows the map past 12, 24, 48, ... entries, HashMap.resize() publishes the new, empty table before it moves the old entries, so a concurrent get can return null for the match's own reader, and TermFilteredPresearcher.buildQuery throws NullPointerException at :122. It needs more than 12 query-index readers open at once (slow in-flight matches spanning refreshes) and fires only at a new peak, so it is rare: a natural workload failed 0 times in 73,064 matches. In every executed run later matches succeeded, and no wrong results were seen. It has been there since the module was added (8.2.0) and is still in 10.6.0 and main; termFilters = new ConcurrentHashMap<>() fixes it.

1. One map, one entry per open reader. QueryIndex.java:55-56: // package-private for testing / final Map<IndexReader.CacheKey, QueryTermFilter> termFilters = new HashMap<>();. Both indexes build their SearcherManager with new TermsHashBuilder(termFilters) (WritableQueryIndex.java:79, ReadonlyQueryIndex.java:48).
2. Writers. TermsHashBuilder.newSearcher runs on the refreshing thread: termFilters.put(reader.getReaderCacheHelper().getKey(), new QueryIndex.QueryTermFilter(reader)); (TermsHashBuilder.java:38) and reader.getReaderCacheHelper().addClosedListener(termFilters::remove); (:39). The writable refresh is manager.maybeRefresh() in commitWithoutNotify (WritableQueryIndex.java:128), reached from register (commit, :102-105), deleteById and clear (:338-351). The read-only refresh is manager.maybeRefresh() in purgeCache() (ReadonlyQueryIndex.java:107-111), called by the application or every purge period by the index's own cache-purge thread (:51-64; default 5 minutes). The remove runs on whichever thread drops the last reference to an old reader, often a match thread releasing its searcher.
3. The unlocked reader. WritableQueryIndex.search holds the purge read lock only around manager.acquire() (:211-217), then calls termFilters.get(searcher.getIndexReader().getReaderCacheHelper().getKey()) with no lock (:221-223); ReadonlyQueryIndex.search takes no lock at all (:73-82). The commit holds commitLock and the read lock (:109-110), the same shared mode that search takes, so nothing orders the put against the get.
4. Why the get returns null. A default HashMap has 16 buckets and load factor 0.75, so a resize threshold of 12; putVal resizes when ++size > threshold. OpenJDK 25's HashMap.resize() publishes the new array with table = newTab; (HashMap.java:712) and only then moves each old bucket, clearing it (oldTab[j] = null;, :717). A lock-free get in that window reads the new, still empty bucket and misses a key that is present. The failing match holds its searcher, so its entry cannot have been removed: the closed listener runs only when the reader's reference count reaches 0. A debugger paused in that window saw exactly this: the published table had 32 slots and 0 entries, while the old 16-slot table still held all 13.
5. Where it throws. Monitor.match calls queryIndex.search(t -> presearcher.buildQuery(reader, t), collector) (Monitor.java:224). The default presearcher, TermFilteredPresearcher (Monitor.java:56-58), filters each document term of a field not in filterFields with termAcceptor.test(field.name, termAtt.getBytesRef()) (TermFilteredPresearcher.java:120-122, reached from the token loop at :127); termAcceptor is the null. A batch with no such term never reads it and does not throw. MultipassTermFilteredPresearcher is a subclass; Presearcher.NO_FILTERING ignores the acceptor.

• In practice a resize happens when a put grows the map past 12 entries, so more than 12 query-index readers are open at once: 12 or more in-flight matches each still holding an older searcher while a refresh opens a new one. A HashMap never shrinks, so after the first resize the next needs a new peak past 24, then 48. (A small HashMap also resizes early if 9 keys collide in one bucket; the keys use identity hash codes, so it needs 9 of at most 12 keys in one of 16 buckets, or of at most 24 in one of 32. It was not measured, and no control run with 11 held searchers ever failed.)
• Threshold controls on stock jars: 11 held searchers (at most 12 entries, no resize) pass 300 rounds (1,589,156 matches) and, in another program, 1,000 rounds; 12 held searchers (13 entries, one resize per round) fail.
• getQuery and scan also call search, but with a builder that ignores the filter (QueryIndex.java:62-81), so a null there is harmless.
• By code reading only, not executed: a reader-close remove racing a refresh put in the same bucket could drop the new reader's entry, and two concurrent removals could leak a filter. By reading, the fix's atomic put and remove would close both; neither variant was run.

• Who is exposed. Applications that share one Monitor between matching threads and threads that register, delete or clear queries, with the default configuration and presearcher (also MultipassTermFilteredPresearcher, a subclass). Read-only Monitors (a read-only configuration with a directory) fail the same way when a refresh, explicit purgeCache() or scheduled, runs while the application matches: it was executed with explicit purgeCache() calls and with only the scheduled refresh at a 2 ms period. With the default 5-minute period, 13 open readers would need matches that span about 12 refreshes of a changing index.
• Effect. A Monitor.match call throws NullPointerException; in every executed run one or a few matches failed and later matches succeeded. No wrong results, corruption or lasting damage were observed, but the tests only check for exceptions, and a HashMap gives no guarantee under such a race.
• Precondition. In practice, more than 12 query-index readers open at the moment of a refresh, and only at a new peak (past 12, 24, 48, ...). Matches that span many registrations get there: long-running queries, big document batches, bursts of registrations.
• How likely. A natural probe on stock jars (default SIMPLE_MATCHER, no blocking, 24 match threads with 200-document batches, one registering thread) went past 12 open readers in 39 of 40 fresh Monitors (at most 20) and failed 0 times in 73,064 matches. Likelihood: rare. No report was found since 8.2.0.

Plain test code has no deterministic form: HashMap.resize() calls no code a test could pause in (a debugger can, see "Independent checks"). So the test repeats a light round (one fresh Monitor, one query f:a, one document f = "a") at most 100 times and stops at the first failure. No latches, sleeps, reflection or shadow classes; it sits in package org.apache.lucene.monitor like the module's tests but uses only public API, and passes checkGoogleJavaFormat, ecjLint and forbiddenApis as is.

It is an amplifier, not a field rate. Through gradle (whose test JVMs run with -XX:ActiveProcessorCount=1), seed DEADBEEF failed with assertions on and off in three separate runs, and 4 of 5 iterations failed with -Ptests.iters=5; on a 32-core machine 20 of 20 iterations failed with -ea and with -da. The seed does not control the interleaving. Nesting depth 11 (at most 12 map entries) passed 2 x 300 rounds; depth 12 failed.

The nested register inside CandidateMatcher.matchQuery only keeps older searchers open on one thread. It is legal public API but unusual. Independently written programs without re-entrant calls (48 slow matches parked on their own threads) fail the same way: an independent verifier's test on main, 10.5.2 and 9.12.3; a tie-break test on main, 10.5.2 and 10.6.0; a plain main() program on main and 10.6.0. A read-only Monitor refreshed only by its own background thread fails on main, 10.5.2 and 10.6.0.

Regression test: this same test; it passes with the fix.

cp $REPRO/TestMonitorMatchDuringRegister.java $LUCENE/lucene/monitor/src/test/org/apache/lucene/monitor/
cd $LUCENE && ./gradlew :lucene:monitor:test --tests org.apache.lucene.monitor.TestMonitorMatchDuringRegister -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false (same failure)

# gradle, scratch worktree of 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (the same with -Ptests.asserts=false)
TestMonitorMatchDuringRegister > testMatchWhileRegistering FAILED
    java.lang.AssertionError: Monitor.match threw while a query was registered
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1F48782182A9B9AD]:0)
        at org.apache.lucene.monitor.TestMonitorMatchDuringRegister.testMatchWhileRegistering(TestMonitorMatchDuringRegister.java:64)

        Caused by:
        java.lang.NullPointerException: Cannot invoke "java.util.function.BiPredicate.test(Object, Object)" because "this.val$termAcceptor" is null
            at org.apache.lucene.monitor.TermFilteredPresearcher$1.accept(TermFilteredPresearcher.java:122)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:52)
            at org.apache.lucene.monitor.TermFilteredPresearcher.buildQuery(TermFilteredPresearcher.java:127)
            at org.apache.lucene.monitor.Monitor.lambda$match$0(Monitor.java:224)
            at org.apache.lucene.monitor.WritableQueryIndex.search(WritableQueryIndex.java:222)
            at org.apache.lucene.monitor.Monitor.match(Monitor.java:224)
            at org.apache.lucene.monitor.Monitor.match(Monitor.java:242)
            at org.apache.lucene.monitor.TestMonitorMatchDuringRegister.lambda$testMatchWhileRegistering$0(TestMonitorMatchDuringRegister.java:47)

# with termFilters = new ConcurrentHashMap<>(): the repro passes (3 of 3 iterations, 300 rounds, assertions on and off)

• Design. The purge-lock comment exists so that searches never wait on commits: "... searches waiting on purge waiting on commit which is problematic since commit is slow and search cannot be slow" (WritableQueryIndex.java:276-290). The query cache is a volatile ConcurrentMap built as new ConcurrentHashMap<>() (:70, :78); the purge log and the new cache are ConcurrentHashMaps (:292, :297). termFilters is the only plain HashMap in the module's main code that changes while matches read it. Both index types start their own cache-purge thread, so a Monitor is multi-threaded by construction.
• SearcherManager. The map is filled by a SearcherFactory plugged into a SearcherManager, "Utility class to safely share IndexSearcher instances across multiple threads, while periodically reopening".
• Upstream treats concurrent register and match as the workload. CHANGES GITHUB#12801: "Remove possible contention on a ReentrantReadWriteLock in Monitor which could result in searches waiting for commits". Its PR body (@daviscook477, merged 2023-11-14) describes latency in Monitor.match "when it is called concurrently with Monitor.register/Monitor.deleteById and QueryIndex.purgeCache" and says commit and search "should be able to execute concurrently".
• The author's statement. In the original library's tracker (flaxsearch/luwak#174, 2018), Alan Woodward (@romseygeek), who later added the module to Lucene, wrote: "You should only have a single Monitor object, but you can index and query it from multiple threads".
• Precedent. LUCENE-9791 (#10830, fixed by #8) fixed a different concurrent-match race as a "Luwak/Monitor bug causing registered queries to sometimes fail to match". In that thread Uwe Schindler (@uschindler) stated the rule this map breaks: a built HashMap may be read concurrently, "but never ever change it again".
• More signals. purgeCache is "normally called from a background thread" (Monitor.java:131); the package docs offer ParallelMatcher and PartitionMatcher "to increase performance in low-concurrency systems" (package-info.java:50-51); upstream TestCachePurging registers queries on a second thread while the test thread purges the cache (TestCachePurging.java:81-127). A null term acceptor has no documented meaning.
• The limit of the claim. Concurrent register and match do not promise that a given match sees a query being registered; the claim is only that match must not throw because of the Monitor's internal state.

Strongest case (from the devil's-advocate verifier, the cross-model reviews and the tie-breaks): no Monitor javadoc says "thread-safe"; ConcurrentQueryLoader says "You can use multiple threads to parse and index queries before starting matches" (ConcurrentQueryLoader.java:37-38), which could mean "register first, then match"; and the test's register inside matchQuery is unusual.

Why it fails: that sentence describes an optional startup helper, not a precondition of register or match, and the helper's own workers call monitor.register from several threads; the code, CHANGES, #12801 and the module's author treat concurrent register and match as supported; a read-only Monitor refreshes on its own thread while the application matches, so the application cannot schedule its matches around the refresh; and tests without re-entrant calls, a plain main() program and a debugger-forced interleaving fail the same way. The cross-model reviews that read the code and the devil's-advocate verifier rated the "intended" argument weak. A later cross-model fact check (reading only) rated it moderate: no Monitor javadoc promises thread safety, more than 12 held readers is uncommon (which lowers the urgency, not the defect), and the re-entrant test is a presentation risk that the non-re-entrant tests answer.

One fix was validated through gradle; the two alternatives were not run.

QueryIndex.java:56: new HashMap<>() becomes new ConcurrentHashMap<>(), plus the import and a one-line comment (1 file, +3/-1). A ConcurrentHashMap keeps the old table published until a resize ends and forwards each moved bin, so a get stays correct during a resize. The put happens in newSearcher before SearcherManager publishes the searcher, so a match that acquired that searcher finds the entry.

import java.util.concurrent.ConcurrentHashMap;
...
  // package-private for testing
  // concurrent: searcher refreshes put and reader-close listeners remove while matches get
  final Map<IndexReader.CacheKey, QueryTermFilter> termFilters = new ConcurrentHashMap<>();

Trade-offs

• By reading, it also closes the two code-reading-only variants (a lost entry, a leaked filter), since put and remove become atomic; those variants were not run.
• Cost: a ConcurrentHashMap.get per match instead of a HashMap.get, on a map with one entry per open query-index reader. No benchmark was run.

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, OpenJDK 25.0.3: the repro fails on the stock tree with assertions on and off; with the fix applied it passes 3 of 3 iterations (300 rounds) with assertions on and 3 of 3 with them off.
• :lucene:monitor:test, the whole module: 33 suites, 182 tests, 1 skipped (a @Nightly test), 0 failures, with assertions on and off, including TestQueryTermFilter, TestCachePurging and TestMonitorReadonly. :lucene:monitor:check -x test passes (google-java-format, ecjLint, forbiddenApis) with the test in the tree.
• Earlier: the same change shadowed onto the jars made the finder's, the independent reproduction's and the tie-breaks' writable and read-only tests pass on main, the intended-behaviour check's writable test pass, and the minimal test pass on 8.2.0; in the debugger check the paused mid-resize match returned normally 4 of 4 times.
• Not run: the full Lucene suite, nightly tests, benchmarks, x86.

Also correct.

Trade-offs

Every match then takes the map's lock, which the purge-lock design tries to avoid for searches.

Create an IndexSearcher subclass in TermsHashBuilder.newSearcher that carries the reader's QueryTermFilter, and read it from the acquired searcher in search.

Trade-offs

No shared mutable state and no close listener, but it changes QueryIndex, both subclasses and TestQueryTermFilter (which reads the map): a larger change for the same effect.

Method: non-formal (code reading, then a directed JUnit), 2026-10-27, main a9779ef3c193. The unit planned Lincheck and Fray model checking of the monitor's query index. Fray switches threads only at synchronisation points, and Lincheck's model checking could not run IndexWriter here, so neither can explore a plain HashMap race. The unit's plan carried the map as a lead that earlier hunt notes had recorded without investigating it.

The finder's JUnit (4 matching threads, up to 48 latch-parked slow matches, public API only) failed on main with assertions on (1 of 4,453 matches) and off (1 of 14,049), on 10.5.2 and on 9.12.3; a read-only variant failed on main; on stock jars 42-56 of 100 single rounds failed (4 seeds), 0 of 400 with the fix. A minimisation step on 2026-10-29 replaced it with the two-thread nested test. On 2026-10-30 a tie-break step paused the registering thread with a debugger right after HashMap.resize() published its new table and ran one match in that pause: it threw in 16 of 16 paused resizes, and returned normally before the put, after the registration and with the fix.