maybeRefresh() swaps in a new reference in two steps: make it current, then release the old one. A cleanup finally releases the new reference whenever the swap did not return normally. If the second step throws, the new reference is already current, so the cleanup closes the manager's own current reference.
acquire() throws. With the fix B keeps its count. Select the still to open it at full size. Generated with AI, checked against the code.- A
ReferenceManagerholds reference A with count 1, its own count. You callmaybeRefresh(): it acquires A (count 2) and gets a new reference B (count 1). swapReferencemakes B current, then releases A, and thatdecRefthrows anIOException, which thedecRefcontract allows.- You expect
maybeRefresh()to throw thatIOExceptionand B to stay current with count 1, so the nextacquire()returns B. - Instead a cleanup
finally, written for a B that was never installed, releases B as well: B drops to count 0, which the manager treats as closed, while it is still current. - Every later
acquire(), and so every later uncontendedmaybeRefresh(), throwsIllegalStateException: The managed reference has already closed - this is likely a bug when the reference count is modified outside of the ReferenceManager.
Only applications with their own ReferenceManager whose decRef can throw before the final release, or a SearcherManager (by reading, also a ReaderManager) whose beforeRefresh listener calls maybeRefresh() again while a reader close fails. Stock managers with ordinary listeners stay usable. The failure is loud, with no wrong results, but the manager must be rebuilt. Contrived, so priority Low.
Mechanism, terms and numbers518 words
ReferenceManager.swapReference makes the new reference current first and releases the old one second (ReferenceManager.java:56-57 on main). doMaybeRefresh marks the refresh as done only after swapReference returns (:172-173), and its finally releases the new reference whenever that mark is missing (:174-178). That finally came with LUCENE-3476 (2011) for a new reference the swap never installed, because the manager was closed during the refresh. When releasing the old reference throws instead (the abstract decRef is documented to throw IOException, :60-65), the new reference is already current, and the finally drops the manager's own count on it and closes it. maybeRefresh() throws the IOException; then every acquire() throws IllegalStateException "The managed reference has already closed - this is likely a bug when the reference count is modified outside of the ReferenceManager" (:112-113), and every later uncontended maybeRefresh() fails the same way in its own acquire() (:164). The application has to build a new manager. It needs a custom ReferenceManager whose decRef throws on a non-final release, or a SearcherManager (by reading, also a ReaderManager) whose beforeRefresh listener calls maybeRefresh() again while a reader close fails; the stock managers with ordinary listeners stay usable. Every release from 3.5.0 to 10.6.0 and main is affected (before 4.7.0 acquire() loops forever instead). The fix decides whether the new reference was installed before swapReference can throw.
ReferenceManager: the base class ofSearcherManagerandReaderManager. It keeps one current reference and holds one count on it;acquire()adds a count for the caller,release()takes it back, and the reference closes when its count reaches 0.maybeRefresh()opens a newer reference and swaps it in.- The swap:
swapReference(ReferenceManager.java:53-58) setscurrent = newReference(:56), then releases the old reference (:57), which calls the subclass'sdecRef.
// ReferenceManager.java:164-181 (doMaybeRefresh, condensed)
final G reference = acquire(); // :164 the refresh holds its own count on the old reference
...
try {
swapReference(newReference); // :172 current = new (:56), then release(old) (:57) throws
refreshed = true; // :173 not reached
} finally {
if (!refreshed) {
release(newReference); // :176 releases the reference that is now current
}
}
...
release(reference); // :181The repro's counts (A is the old reference, B the new one):
| step | A | B | current |
|---|---|---|---|
| start | 1 | A | |
acquire() at :164 | 2 | A | |
refreshIfNeeded returns B | 2 | 1 | A |
current = B, then release(A) throws | 2 | 1 | B |
finally: release(B) | 2 | 0 (closed, for the manager) | B |
release(A) at :181; maybeRefresh() throws the IOException | 1 | 0 | B |
acquire(): tryIncRef(B) fails, B has count 0 and is current | IllegalStateException (:112) |
Expected: B keeps count 1 and acquire() returns it. (A stays at 1 in the repro because the test's decRef throws before it decrements; that is the test subclass's choice, not part of the bug. The independent reproducer's decRef decrements first, like IndexReader.decRef, and fails the same way.)
Before LUCENE-5436 (#6499, 4.7.0) acquire() had no such check and looped forever on the closed reference.
Exposure and likelihood220 words
- Who is hit. (a) A custom
ReferenceManagersubclass (public,@lucene.experimental) whosedecRefcan throw on a non-final release: the repro. (b) A stockSearcherManagerwhoseRefreshListener.beforeRefreshcallsmaybeRefresh()on the same manager, while closing the intermediate reader throws (executed with a throwingIndexReader.ClosedListener); thereclose()of the manager then throwsAlreadyClosedException: this IndexReader is closedtoo. The stockReaderManagerinherits the same refresh code and listeners, so it can reach the state the same way (by reading; not run). - Why the stock path is safe otherwise.
doMaybeRefreshholds its own count on the old reference (:164), so the release inside the swap is never the final one, andIndexReader.decRefthrowsIOExceptiononly when the count reaches 0, fromdoClose()or the reader-closed listeners (IndexReader.java:222-227). A reader close failure then surfaces at the outer release (:181) and the manager stays usable (two controls passed). - Effect. One
IOException, then everyacquire()and every later uncontendedmaybeRefresh()throws until the application builds a new manager. The message says the caller changed the count, which points debugging the wrong way. No search returns wrong hits and nothing is written to the index. - Scale. None: one manager, one refresh, single thread, assertions on or off.
- Likelihood: contrived. No public report of it was found since 2011.
One JUnit test, single thread, deterministic, public API only: an anonymous ReferenceManager over AtomicInteger counters whose first decRef throws IOException. It fails with assertions on and off on main and passes with either fix.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.search;
import java.io.IOException;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.lucene.tests.util.LuceneTestCase;
public class TestReferenceManagerFailedSwap extends LuceneTestCase {
// Expected: acquire() succeeds after the failed maybeRefresh; actual: IllegalStateException.
public void testAcquireAfterReleaseOfOldReferenceFails() throws Exception {
ReferenceManager<AtomicInteger> manager =
new ReferenceManager<>() {
boolean failNextDecRef = true;
{
current = new AtomicInteger(1);
}
@Override
protected void decRef(AtomicInteger ref) throws IOException {
if (failNextDecRef) {
failNextDecRef = false;
throw new IOException("decRef failed");
}
ref.decrementAndGet();
}
@Override
protected AtomicInteger refreshIfNeeded(AtomicInteger ref) {
return new AtomicInteger(1);
}
@Override
protected boolean tryIncRef(AtomicInteger ref) {
return ref.get() > 0 && ref.incrementAndGet() > 0;
}
@Override
protected int getRefCount(AtomicInteger ref) {
return ref.get();
}
};
expectThrows(IOException.class, manager::maybeRefresh);
manager.release(manager.acquire());
}
}
cd $LUCENE
cp $REPRO/TestReferenceManagerFailedSwap.java lucene/core/src/test/org/apache/lucene/search/
./gradlew :lucene:core:test --tests org.apache.lucene.search.TestReferenceManagerFailedSwap -Ptests.seed=DEADBEEF
# assertions off fails the same way: add -Ptests.asserts=false
- Expected
- After
maybeRefresh()throws theIOException,acquire()returns the new current reference. - Observed
acquire()throwsIllegalStateException: The managed reference has already closed ...(ReferenceManager.java:112).
More files, environment, full output91 words
The first decRef call is the old reference's release inside the swap, so that is the one that throws. The test draws no random values. It belongs in lucene/core/src/test/org/apache/lucene/search/ and passes :lucene:core:check -x test (format, lint, forbidden APIs).
Regression test: this same test.
| lucene | main 5adff78bb168 (12.0.0-SNAPSHOT) |
|---|---|
| jdk | OpenJDK 25.0.3 (build 25.0.3+9-LTS) |
| os | Linux, aarch64 |
TestReferenceManagerFailedSwap > testAcquireAfterReleaseOfOldReferenceFails FAILED
java.lang.IllegalStateException: The managed reference has already closed - this is likely a bug when the reference count is modified outside of the ReferenceManager
at __randomizedtesting.SeedInfo.seed([DEADBEEF:9B25C6C40B135ABD]:0)
at org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:112)
at org.apache.lucene.search.TestReferenceManagerFailedSwap.testAcquireAfterReleaseOfOldReferenceFails(TestReferenceManagerFailedSwap.java:60)acquire()'s own comment says a closed current reference "can only happen if somebody outside of the RM decrements the refcount"; here the manager does it.decRefis documented to throwIOExceptionwith no limit to the final release, and nothing forbids callingmaybeRefresh()from a listener.- The cleanup was added (LUCENE-3476) for a new reference the swap never installed.
The full argument190 words
- The manager's own invariant.
acquire()'s comment (ReferenceManager.java:103-111): "This can only happen if somebody outside of the RM decrements the refcount without a corresponding increment since the RM assigns the new reference before counting down the reference." The exception message (:112-113) blames the caller. - The contract allows the trigger.
decRef: "@throws IOException if reference decrement on the given resource failed" (:60-65).RefreshListener(:296-306) sets no rule against re-entry; therefreshLockis aReentrantLock(:43) anddoMaybeRefresh's comment allows the same thread to lock again (:155-160);IndexReader.ClosedListener.onClosedeclaresIOException(IndexReader.java:141-145). - History. The
finallycame with LUCENE-3476 (#4550, 2011), "SearcherManager misses to close IR if manager is closed during reopen": the swap threwAlreadyClosedException(then from an inline null check, today fromensureOpen(),:54) and the new searcher was never installed. None of #4550, #4834 (LUCENE-3761) or #6499 (LUCENE-5436) discusses a failing release of the old reference. - Tests.
TestSearcherManager.testReferenceDecrementIllegally(TestSearcherManager.java:331) expects thisIllegalStateExceptiononly after adecRefmade outside the manager. No test makes the old reference's release fail during a refresh.
The devil's-advocate check167 words
The strongest case: every in-tree manager's decRef fails only on the final release (IndexReader.decRef: "@throws IOException in case an IOException occurs in doClose()", IndexReader.java:208-213; its reader-closed listeners run there too, :222-227), and during a normal refresh the swap's release is never final. A decRef that fails earlier, or calling maybeRefresh() from inside its own beforeRefresh, could be called misuse.
Why it fails: it explains why the bug is rare, not why releasing the reference the manager has just made current would be correct. The abstract decRef contract does not limit failures to the final release, nothing forbids re-entry, and no javadoc, test, CHANGES entry or discussion says a failed refresh may leave a closed reference current; the code comment says the opposite. The cross-model review (OpenAI GPT-5.6) rated the intended-behaviour case weak; a later cross-model fact check rated it moderate and noted that a maintainer could reasonably close the issue as too contrived. What survives is a severity argument, hence priority Low.
Recommended Decide "installed" under the manager's monitor, before the swap can throw
Decide whether the new reference was installed while holding the manager's monitor, before swapReference can throw, and release it in the finally only if it was not. Validated: the repro passes with assertions on and off; TestSearcherManager, TestControlledRealTimeReopenThread, TestLiveFieldValues and TestSearcherTaxonomyManager pass (with the repro, 26 tests, 0 failures, 2 skipped), and so does :lucene:core:check -x test.
boolean installed = false;
try {
- swapReference(newReference);
+ synchronized (this) {
+ ensureOpen();
+ // swapReference makes newReference current before it releases the old reference,
+ // so from here on a failure must not release newReference: the manager owns it
+ installed = true;
+ swapReference(newReference);
+ }
refreshed = true;
} finally {
- if (!refreshed) {
+ if (!installed) {All options and trade-offs2 options ยท 570 words
Validated with gradle in a clean scratch worktree of 5adff78bb168 (OpenJDK 25.0.3, seed DEADBEEF, 2026-11-03). Both fixes keep refreshed false on this path, so listeners still get afterRefresh(false); whether they should is the separate LFV-312 question (a different fix site). Open PR #15985 rewrites doMaybeRefresh into a loop but keeps this code.
Workaround until fixed: let a custom decRef fail only on the final release, as IndexReader.decRef does, and do not call maybeRefresh() from inside beforeRefresh (controls of both shapes stay usable); otherwise rebuild the manager after a failed maybeRefresh().
Decide "installed" under the manager's monitor, before the swap can throw (validated by the checks) (recommended)
Check ensureOpen() and set an installed flag while holding the monitor that swapReference and close() use; release the new reference in the finally only when it was not installed. Once ensureOpen() has passed under the monitor, swapReference (re-entrant on it) cannot fail before current = newReference, so the flag is exact. The case LUCENE-3476 handles (close() ran first) still releases the new reference.
boolean installed = false;
try {
- swapReference(newReference);
+ synchronized (this) {
+ ensureOpen();
+ // swapReference makes newReference current before it releases the old reference,
+ // so from here on a failure must not release newReference: the manager owns it
+ installed = true;
+ swapReference(newReference);
+ }
refreshed = true;
} finally {
- if (!refreshed) {
+ if (!installed) {Trade-offs
About ten lines. It also covers a cross-thread interleaving the one-line fix leaves open, by reading: if close() runs after the failed swap but before the finally, close() releases the new reference as current and the finally must not release it again. Monitor use and lock order are the same as today (swapReference already takes the monitor under refreshLock).
What the checks validated
- Stock
5adff78bb168: the repro fails through gradle with assertions on and off; with fix B it passes, on and off. - With fix B, the repro,
TestSearcherManager(12 tests),TestControlledRealTimeReopenThread(6, 1 skipped) andTestLiveFieldValues(1): 20 tests, 0 failures;TestSearcherTaxonomyManager(lucene/facet): 6 tests, 0 failures, 1 skipped. :lucene:core:check -x test(format, lint, forbidden APIs) with fix B and the test: passes.- The cross-thread interleaving and the stock route (b) were checked by reading only; no test drives them with fix B.
if (!refreshed && current != newReference) in the finally; the hunt's fix sketch, unchanged (comment included).
} finally {
- if (!refreshed) {
+ // release the new reference only if swapReference did not install it: once it is
+ // current, the manager's count on it must stay even if releasing the old one failed
+ if (!refreshed && current != newReference) {
release(newReference);
}Trade-offs
Smallest change. It reads current outside the monitor, so in the interleaving above (a concurrent close() between the failed swap and the finally) it still releases the new reference twice; the stock code does the same there, so it is no worse than today.
What the checks validated
- With fix A the repro passes, assertions on and off; the repro and the same three core suites: 20 tests, 0 failures, 1 skipped;
:lucene:core:check -x testpasses. - Earlier, the hunt's runs of the same one-line change passed the repro (on, off),
TestSearcherManager(12 of 12) and the independent reproducer's two tests.
Downstream products are exposed only if they ship their own ReferenceManager subclass whose decRef can fail before the final release, or call maybeRefresh() from a refresh listener; no query, document or index input can trigger it.
A proof-obligation audit of ReferenceManager's refresh path checked each release of the new reference against the invariant stated in acquire(), and found the cleanup also runs after the swap has installed it. A public-API JUnit test then witnessed it on main; independent checks reproduced it with their own tests.
The method in full118 words
Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).
- The obligation:
doMaybeRefreshmay release the new reference only if the swap did not install it, becauseacquire()relies on the manager never leaving a closed reference current (:103-111). Thefinally(:174-178) keys onrefreshed, which is still false whenswapReferencethrows aftercurrent = newReference(:56-57), so the obligation is not met. - A one-test JUnit witnessed it on main with assertions on and off. An independent reproducer, an executed review by another model family (OpenAI GPT-5.6) and a devil's-advocate verifier wrote their own tests from the claim alone, including two routes through the stock
SearcherManager.
No prior report: three scripted prior-art runs (one on the exact error message), manual GitHub searches in Lucene and downstream projects, the mailing lists, JIRA and the web, plus the changed files of all 409 open PRs (2026-11-03), rechecked the same day.
Searches3 searches
No report found in the documented searches says that ReferenceManager releases the reference swapReference has just installed when releasing the old one throws. Related, not this bug: #4550 (LUCENE-3476, added the release for the not-installed case), #4834 (LUCENE-3761, created ReferenceManager with this code), #5632 (LUCENE-4566, renamed the flag, added listeners), #6499 (LUCENE-5436, added the "already closed" error assuming only an outside decrement causes it), #15831 and open PR #15985 (a lost refresh through tryLock; the PR keeps this code), #15068 (a stack through swapReference(null) from close(), another defect). Downstream reports with the same message (orientdb #8774, #8829, #8832; a 2014 Elasticsearch forum thread) show no failed refresh: symptom match only, not evidence of impact.
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits; Elasticsearch, OpenSearch, Solr, Lucene.NET; dev@, java-user@, issues@, commits@; JIRA archive; web | 2026-11-03 | three scripted prior-report search runs (45, 152 and 38 distinct hits) and manual GitHub searches | only the related items above |
| Open PRs | 2026-11-03 | the changed files of all 409 open apache/lucene PRs; the diff of the one touching the file | only #15985 touches ReferenceManager.java, and it keeps this code |
| Recheck (GitHub) | 2026-11-03 | class, method and message queries; PRs updated since 2026-11-01; upstream file history | nothing new; upstream main 1c2072f0284 still has the same swap and finally |
Introduced by LUCENE-3476 (a234ffd21c9, 2011-10-01) in SearcherManager, first released in 3.5.0; LUCENE-3761 moved it unchanged into ReferenceManager (3.6.0). Executed: 3.5.0 to 10.6.0 and main fail. acquire() loops forever until 4.7.0, then throws. Not fixed upstream.
Versions tested6 rows
Introduced by LUCENE-3476 (#4550; svn era, no pull request), Simon Willnauer, 2011-10-01: a234ffd21c9 on branch_3x (shipped in 3.5.0) and d514a4ac75b on trunk. It wrapped the swap in success / finally { if (!success) release(newSearcher); }, around a swap that already installed the new searcher before releasing the old one. LUCENE-3761 (#4834, Shai Erera, 2012-02-14; 4e5ead2640a on branch_3x, first in 3.6.0, and f4c6a4098ea on trunk) moved it unchanged into the new ReferenceManager; LUCENE-4566 (f6b3aa4054b) renamed the flag to refreshed. Nothing has changed those lines since; all 127 release tags from 3.6.0 to 10.6.0 in the history clone have the same code (8.8.2 to 8.11.x are not tagged there; 8.11.4 was executed). In 3.5.0 the stock SearcherManager held no count across the swap, so any failure closing the old reader hit it; from 3.6.0 only a custom decRef or (from 4.2.0, when RefreshListener.beforeRefresh came with LUCENE-4695) a re-entrant listener reaches it; the 4.1.0 listener has only afterRefresh(), which runs after the swap and the outer release. 3.6.0, 4.6.1, 4.7.0 and 8.11.4 ran plain-Java copies of the same steps and 3.5.0 the stock SearcherManager with a reader-finished listener that throws a RuntimeException (an IOException from doClose takes the same path, by reading), with acquire() bounded by a 10 s join. Upstream main 1c2072f0284 (2026-11-02) is unchanged here. All runs used JDK 25.0.3.
| Version or commit | Verdict | Result |
|---|---|---|
| 3.4.0 | no API | no SearcherManager or ReferenceManager (the probe does not compile) |
| 3.5.0 | first: 3.5.0 | stock SearcherManager with a throwing reader-finished listener: maybeReopen throws, then acquire() loops forever |
| 3.6.0, 4.6.1 | affected | maybeRefresh throws, then acquire() loops forever (3.6.0 with assertions on and off) |
| 4.7.0, 8.11.4 | affected | acquire() throws the IllegalStateException |
| 9.12.3, 10.5.2, 10.6.0 | affected | one-test JUnit variants fail; the stock SearcherManager route reaches the same state on 9.12.3 and 10.6.0 |
| main 5adff78bb168 | affected | the one-test repro fails through gradle, assertions on and off; passes with either fix |
An independent reproducer, an executed review by another model family (OpenAI GPT-5.6), a devil's-advocate verifier and an adjudication all rated it real; the adjudication confirmed it (medium confidence, priority Low). This doc's gradle runs reproduced it on main and validated both fixes.
Checks and verdicts229 words
The cross-model review ran on main ea57360912ed, whose relevant files equal 5adff78bb168's, and its tests were re-run on 5adff78bb168 with the same result; every other check ran on 5adff78bb168 or a release.
| Check | Verdict | What it found |
|---|---|---|
| finder | reproduced | One-test JUnit fails on main (on, off), 10.6.0 and 9.12.3; the one-line fix passes it and TestSearcherManager (12 of 12); a stock SearcherManager probe reaches the same state. |
| independent reproducer | real | Own test (decRef decrements, then throws, like IndexReader.decRef) fails on main (on, off), 10.6.0, 10.5.2 and 9.12.3; own stock-SearcherManager test fails on main; both pass with the fix. |
| cross-model review, refute lens (OpenAI GPT-5.6, executed) | real (high) | Own subclass test fails; own stock test shows the closed current searcher while a non-re-entrant control stays usable; intended-behaviour argument rated weak. |
| devil's-advocate verifier | real, Low | Rare but not intended; a control whose decRef fails only at 0 stays usable. |
| adjudication | confirmed (medium), Low | Re-ran the finder's test (on, off) and both of the reproducer's tests: all fail. |
| this doc (gradle) | reproduced; both fixes validated | Gradle in a clean scratch worktree of 5adff78bb168: the repro fails with assertions on and off; it passes with fix A and with fix B (on, off), and the related suites and :lucene:core:check -x test pass with each fix. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 570 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,709 words
- Registry title
- ReferenceManager.maybeRefresh releases the new reference it has just made current when releasing the old one throws, so every later acquire() throws IllegalStateException (since 3.5.0)
- Area
- core / search: ReferenceManager (SearcherManager, ReaderManager and other subclasses)
- Priority
- Low, rank 947: A failed ReferenceManager refresh can leave the manager holding a closed current reference, so every later acquire() throws an IllegalStateException that blames the caller (before 4.7.0 it spins forever) until the manager is rebuilt, in every release since 3.5.0. It cannot happen with the stock managers and ordinary listeners: it needs a custom decRef that fails on a non-final release, or a re-entrant refresh listener while a reader close fails, so it is contrived.
- Impact class
- minor
- Impact note
- Only applications that run their own ReferenceManager subclass whose decRef can throw on a non-final release, or a stock SearcherManager whose RefreshListener.beforeRefresh calls maybeRefresh again on the same manager while closing the intermediate reader throws (for example a throwing IndexReader.ClosedListener), are hit: after that one failed maybeRefresh, every acquire() and every later maybeRefresh throws IllegalStateException 'The managed reference has already closed - this is likely a bug when the reference count is modified outside of the ReferenceManager', and close() of the SearcherManager throws AlreadyClosedException, so the manager is unusable until the application builds a new one; -ea and -da, no scale, every release since 3.5.0 (before 4.7.0 acquire() spins forever instead); with the stock managers and ordinary listeners it does not happen, nothing is written to the index and no wrong result is returned.
- Summary
- You call maybeRefresh on a ReferenceManager (for example a SearcherManager), and releasing the old reference inside the swap throws an IOException, which decRef is documented to do; you expect maybeRefresh to throw it and the manager to stay usable, with acquire() returning the new current reference. Instead swapReference has already made the new reference current (ReferenceManager.java:56) before it releases the old one (:57), and doMaybeRefresh's cleanup finally, which exists for a new reference the swap never installed, releases it anyway (:174-178), so the manager drops its own count on its current reference and closes it. Every later acquire() then throws IllegalStateException 'The managed reference has already closed - this is likely a bug when the reference count is modified outside of the ReferenceManager', blaming the caller for a state the manager created itself.
- Introduced
- a234ffd21c9131f391646c5f3ced8964d6796a82 (branch_3x, svn r1177943, the code that shipped in 3.5.0; trunk twin d514a4ac75b5374a3c19193137e3bbc972f7bdd2, svn r1177940, ten minutes earlier, first in 4.0.0-alpha): LUCENE-3476 'SearcherManager misses to close IR if manager is closed during reopen' (issue #4550; svn era, no PR) added 'success = false; try { swapSearcher(newSearcher); success = true; } finally { if (!success) release(newSearcher); }' to SearcherManager.maybeReopen around a swap that already installed the new searcher before releasing the old one, for the case where close() ran during the reopen and the new searcher was never installed. The same day f02ae7fc9a5 / 52e8d40c2f2 merged the warm and swap try blocks and kept the shape; LUCENE-3761 ('Generalize SearcherManager', #4834) moved it unchanged into the new ReferenceManager (4e5ead2640a on branch_3x, first in 3.6.0; f4c6a4098ea on trunk, where git blame stops because the file was created there); LUCENE-4566 (f6b3aa4054b) renamed the flag to refreshed [read: git log -S 'release(newReference)' and -S 'release(newSearcher)' on the full-history clone; exec: 3.5.0 SearcherManager form and 3.6.0 ReferenceManager form fail] (2011-10-01 (author date = commit date: 08:10:21Z trunk, 08:20:40Z branch_3x); moved into ReferenceManager 2012-02-14 (12:55:43Z branch_3x, 15:36:14Z trunk))
- First affected version
- 3.5.0 [exec] (the first release with SearcherManager, stock class: maybeReopen throws, then acquire() spins forever; 3.4.0 has neither SearcherManager nor ReferenceManager [exec: compile error]); ReferenceManager, the repro's API, is affected from its first release 3.6.0 [exec]
- Affected versions
- Every release from 3.5.0 through 10.6.0 (4ae946e5a9b, released 2026-10-29) and main 5adff78bb168 (12.0.0-SNAPSHOT); never fixed, not a regression, not new in 10.6. Symptom after the failed refresh: 3.5.0 to 4.6.1 acquire() loops forever on the closed current reference; from 4.7.0 (LUCENE-5436, #6499) acquire() throws the IllegalStateException. [exec] 3.5.0 (stock SearcherManager form, -ea), 3.6.0 (-ea and -da), 4.6.1 (-ea), 4.7.0 (-ea) and 8.11.4 (-da) with plain main() copies of the same steps (core jar only; acquire() bounded by a 10 s join because it never returns before 4.7.0), the one-test JUnit repro on 9.12.3, 10.6.0 and main (-ea and -da), the independent reproduction's own test on 10.5.2, 10.6.0 and 9.12.3 (-ea), and the stock-class route (SearcherManager, a RefreshListener that re-enters maybeRefresh once, a throwing ClosedListener on the intermediate reader) on 9.12.3, 10.6.0 and main (-da). [read] all 127 release tags 3.6.0 to 10.6.0 in the history clone have both the swap order and the finally (8.8.2 to 8.11.x are not tagged there; 8.8.1 read and 8.11.4 executed). Reach by version: in 3.5.0 any failure closing the old reader hit it with the stock class; from 3.6.0 maybeRefresh holds its own count on the old reference across the swap, so the stock managers reach it only through a re-entrant RefreshListener (from 4.2.0, when LUCENE-4695 added RefreshListener.beforeRefresh) plus a failing reader close, and otherwise only a custom ReferenceManager reaches it. Not fixed upstream: upstream main 1c2072f0284 (2026-11-02) still has the code, and the newest upstream commit on ReferenceManager.java is cad76ccfd2d (author date 2025-01-16, comment grammar) [read, 2026-11-03]. Caveats: one seed per run, JDK 25.0.3 on Linux aarch64; the defect is plain Java logic.
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal: proof-obligation audit
- Upstream and prior reports
- none found (2026-11-03, on top of the finder's scripted prior-report search and manual reads; rechecked the same day): three scripted prior-report searches (the finder's 51 queries; run 2 with new terms, 152 distinct hits; run 3 on the exception message text, 38 distinct hits) over apache/lucene issues and PRs, commits, downstream repositories, the dev@, java-user@, issues@ and commits@ lists, the JIRA archive and the web, plus manual GitHub REST searches in apache/lucene and across GitHub (Elasticsearch, OpenSearch, Solr, Lucene.NET) and web reads. The nearest items define the code but not this path: #4550 (LUCENE-3476) added the release of the new reference for the not-installed case only (close() during a reopen; body and all comments read); #6499 (LUCENE-5436) added the 'already closed' IllegalStateException on the stated assumption that only a decrement outside the manager can close the current reference; #4834 (LUCENE-3761) and #5632 (LUCENE-4566) do not discuss release(old) throwing. Open PR #15985 (for #15831, last activity 2026-10-16) restructures doMaybeRefresh into a loop but keeps 'swapReference(newReference); refreshed = true; } finally { if (!refreshed) release(newReference); }', so it neither reports nor fixes this. Downstream reports with the same message (orientechnologies/orientdb #8774, #8829, #8832; a 2014 Elasticsearch forum thread) show no failed refresh and were never traced to this path: symptom match only. All 409 open apache/lucene PRs were checked by changed files (only #15985 touches ReferenceManager.java), and none of the open PRs updated since 2026-11-01 touches ReferenceManager.java, SearcherManager.java or ReaderManager.java; the newest upstream commit on ReferenceManager.java is cad76ccfd2d (author date 2025-01-16, comment grammar).
- Lucene commit tested
- main
5adff78bb168(the one-test repro through gradle in a clean scratch worktree, assertions on and off, unfixed and with fixes A and B; with each fix the repro,TestSearcherManager,TestControlledRealTimeReopenThread,TestLiveFieldValuesand:lucene:core:check -x test; with fix B alsoTestSearcherTaxonomyManager, 2026-11-03). Earlier checks: standalone runs of the repro on main (assertions on and off), an earlier one-test variant on the 10.6.0 and 9.12.3 jars, the stockSearcherManagerwith a throwing reader-finished listener on 3.5.0, plain-Java copies of the same steps on 3.6.0, 4.6.1, 4.7.0 and 8.11.4, and a stockSearcherManagerroute on 9.12.3, 10.6.0 and main. The cross-model review ran on mainea57360912ed, whoseReferenceManager,SearcherManagerandIndexReaderequal5adff78bb168's, and was re-run on5adff78bb168.
- Finding record
lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java:27-36 (at 5adff78bb168) (class javadoc: 'This class ensures each reference is closed only once all threads have finished using it' :29-30; @lucene.experimental :35), :43 (refreshLock is a ReentrantLock), :53-58 (synchronized swapReference: ensureOpen :54, current = newReference :56 BEFORE release(oldReference) :57), :60-65 (decRef: '@throws IOException if reference decrement on the given resource failed'), :84-116 (acquire: 'Obtain the current reference' :85; IllegalStateException when tryIncRef fails, getRefCount == 0 and current == ref :101-113; comment :103-111 'This can only happen if somebody outside of the RM decrements the refcount without a corresponding increment since the RM assigns the new reference before counting down the reference'), :134-142 (close: swapReference(null) releases the current reference), :154-188 (doMaybeRefresh: same-thread relocking allowed :155-160; acquire() of the old reference :164; swapReference :172; refreshed = true :173; the finally that releases newReference whenever !refreshed :174-178, the fix site; outer release(reference) :181), :295-306 (RefreshListener: no rule against re-entry)lucene/core/src/java/org/apache/lucene/index/IndexReader.java:141-145 (at 5adff78bb168) (ClosedListener.onClose throws IOException), :205-232 (decRef: '@throws IOException in case an IOException occurs in doClose()'; closes the reader at refCount 0)lucene/core/src/test/org/apache/lucene/search/TestSearcherManager.java:331 (at 5adff78bb168) (testReferenceDecrementIllegally: the IllegalStateException is asserted only after a decRef outside the manager)Lucene main 5adff78bb168 SECURITY.md:48-50 ('In-scope reports')
- The finding in 30 seconds
lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java:41-58 (at 5adff78bb168) (current, refreshLock, swapReference)lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java:91-116 (at 5adff78bb168) (acquire; the guard at :101-113)lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java:154-188 (at 5adff78bb168) (doMaybeRefresh: :164, :172-178, :181)
- Who is affected
lucene/core/src/java/org/apache/lucene/index/IndexReader.java:222-227 (at 5adff78bb168) (decRef throws IOException only at count 0, from doClose or the reader-closed listeners); StandardDirectoryReader.java:668-671 (the listeners' onClose); ReaderManager.java:33, :87-103 (same refresh code, by reading)
- Why it looks unintended
lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java:43, (at 5adff78bb168) :60-65, :103-113, :155-160, :296-306lucene/core/src/java/org/apache/lucene/index/IndexReader.java:141-145, (at 5adff78bb168) :208-213lucene/core/src/test/org/apache/lucene/search/TestSearcherManager.java:331-361 (at 5adff78bb168)
- Suggested fix
lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java:53-58, (at 5adff78bb168) :134-142, :161, :172-178
- Diagrams
lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java:53-58, (at 5adff78bb168) :91-116, :154-188
- TestReferenceManagerFailedSwap.java (one test, lucene/core):
repro/LFV-1221/TestReferenceManagerFailedSwap.java(62 lines)
A step-by-step picture of the repro's reference counts, made in parallel with this doc from the code at 5adff78bb168 and the executed repro; the PNG is the summary still. Its fix strip shows the one-line fix A, which the diagram's runs executed; the recommended fix B is under Fixes.
assets/LFV-1221/bug.png(570 KB)