ByteBuffersDirectory.rename can silently delete a file that another thread just created under the source name

ByteBuffersDirectory.rename(source, dest) moves the entry with get(source), putIfAbsent(dest, file) and the identity-checked remove(source, file), and then calls files.remove(source) once more (ByteBuffersDirectory.java:191 on main). After the checked remove the moved entry is already gone, so that last call can only remove an entry that another thread put under the source name in the window between the two calls, for example a new file (createOutput) or one renamed onto it. That file disappears without any error, although the other thread's call returned normally and nobody deleted it. The same line was in RAMDirectory's rename after LUCENE-7345 (2016), where it had been the move itself before the checked remove was added, and ByteBuffersDirectory has had it since the class was added (LUCENE-8468, 2018); every release from 7.5.0 to 10.6.0 and main are affected. No Lucene component is known to trigger it; it needs an application that reuses one file name across threads. The fix is deleting the line.

• ByteBuffersDirectory: Lucene's heap-resident Directory. Every file is an entry in one ConcurrentHashMap<String, FileEntry> files (ByteBuffersDirectory.java:93); listAll() lists its keys (:126).
• createOutput(name) creates a new entry only if the name is free: files.putIfAbsent(name, e), else FileAlreadyExistsException (:157-160).
• rename(source, dest) (:178-192) moves an entry to a name that must not exist yet.

1. FileEntry file = files.get(source) (:181).
2. files.putIfAbsent(dest, file) (:185): the file is now visible under both names, which the rename javadoc allows (Directory.java:125-126).
3. if (!files.remove(source, file)) throw new IllegalStateException("File was unexpectedly replaced: " + source) (:188-190). FileEntry has no equals (:226), so this removes exactly the moved entry; once it returns, source is free.
4. files.remove(source); (:191). Alone it removes nothing. If another thread put a new entry under source after step 3, for example with createOutput(source) (:158) or rename(x, source) (:185), this call removes that new entry.

thread R: rename("a", "b")                thread W: createOutput("a") ... close()
  get("a")              -> F1
  putIfAbsent("b", F1)  -> ok
  remove("a", F1)       -> true
                                           putIfAbsent("a", F2) -> ok; write; close() returns normally
  remove("a")           -> removes F2     <- W's new file is gone; listAll() = [b]

Both calls return normally and nobody calls deleteFile, yet listAll() lacks the new a and openInput("a") throws NoSuchFileException. In two-thread stress tests on the stock jar this happens 1 to 222 times per 100,000 hand-offs, in almost every run on the test machine (the finder's test failed 38 of 40 method runs; another test lost 88 of 100,001 with assertions on, 39 with assertions off).

• Who is exposed. Only applications that use one ByteBuffersDirectory from several threads with a reused file name, for example a spool or publish-by-rename hand-off with a fixed staging name, without their own lock around rename and createOutput. No setting, index size or data shape is needed; whether it hits depends only on timing.
• Opt-in. ByteBuffersDirectory is not the default and is @lucene.experimental (ByteBuffersDirectory.java:48); its javadoc says MMapDirectory "is nearly always a better choice" (:43-46), and MIGRATE.md calls it "discouraged in favor of the default MMapDirectory". Wrappers such as FilterDirectory and TrackingDirectoryWrapper delegate rename without serializing it against createOutput.
• Lucene's own paths are not known to be affected. IndexWriter's only rename is pending_segments_N to segments_N (SegmentInfos.java:1002-1005), and nothing re-creates that pending name. NRTCachingDirectory keeps a ByteBuffersDirectory cache (NRTCachingDirectory.java:72) but its rename un-caches the source and renames on the delegate (:181-186).
• Why tests never saw it. MockDirectoryWrapper, which wraps most directories that newDirectory() gives Lucene's tests (rarely a bare, unsynchronized RawDirectoryWrapper instead), declares both rename and createOutput synchronized (MockDirectoryWrapper.java:221, :690).
• What the caller sees. Nothing at the time; later the file is missing from listAll() and openInput throws NoSuchFileException. The content was only on the heap, so it is lost.
• Likelihood: rare. Nothing has been reported since 7.5.0 (2018); publish-by-rename with a fixed name is a real pattern (rename "is used by IndexWriter to publish commits"), so it is not contrived.

Input: one empty, closed file a, renamed to b. The test asserts listAll() is [a, b] afterwards. It sits in org.apache.lucene.store next to TestByteBuffersDirectory, draws no random values, and fails with assertions on and off (20 of 20 with --iters 20).

As a regression test it needs @SuppressForbidden(reason = "swaps the private files map to force the interleaving") (from org.apache.lucene.util) on the method: Lucene's forbiddenApisTest rejects setAccessible without it, and with it :lucene:core:check -x test passes. A maintainer may prefer to take the one-line fix without a reflective test.

Real threads on the stock jar hit the same window: two-thread stress tests (100,000 to 200,000 hand-offs) lose files in almost every run on the test machine (38 of 40 method runs of the finder's test). They show that real scheduling reaches the window; they are evidence, not the repro.

The same file fails unchanged on the 10.6.0, 10.5.2, 9.12.3 and 9.1.0 release jars; before 9.1.0 only the LuceneTestCase import changes (org.apache.lucene.util). Setting a private final field by reflection works on JDK 25; newer JDKs are moving to warn about or restrict it (JEP 500).

Regression test: this same test.

TestByteBuffersDirectoryRenameDropsNewSource > testFileCreatedUnderSourceNameDuringRenameSurvives FAILED
    java.lang.AssertionError: expected:<[a, b]> but was:<[b]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:FD637234E020AE47]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.store.TestByteBuffersDirectoryRenameDropsNewSource.testFileCreatedUnderSourceNameDuringRenameSurvives(TestByteBuffersDirectoryRenameDropsNewSource.java:49)

# with the one-line fix: BUILD SUCCESSFUL (assertions on and off)

• The contract. "Once a file is created it must only be opened for input (openInput), or deleted (deleteFile)." (Directory.java:38-39). The rename javadoc allows one relaxation: "It is permitted for this operation to not be truly atomic, for example both source and dest can be visible temporarily in listAll()" (:125-126). Nothing in it describes deleting a file the rename never moved. The javadoc does not spell out semantics for overlapping calls on one name, so this supports a defect report rather than proving one.
• No serial order explains it. If W's createOutput("a") runs before rename's remove, it throws FileAlreadyExistsException; if after, the new a survives. Even a non-atomic copy-then-delete rename keeps the old a under source until its one delete, so createOutput("a") keeps failing until then; with nobody calling deleteFile, no new a can appear before that delete, so it would not lose the file either.
• The line does nothing on its own. After remove(source, file) returns true, source is already gone, so remove(source) removes nothing without another thread.
• History. Before LUCENE-7345, RAMDirectory.renameFile (renamed rename by LUCENE-7373 before 6.2.0) was fileMap.put(dest, file); fileMap.remove(source);, where the final remove was the move itself. LUCENE-7345 (2016, "RAMDirectory now enforces write-once as well"; c311eb1b7ed on master, branch_6x backport 07920c4fdaa) made it putIfAbsent plus the checked remove(source, file) and kept the old line; LUCENE-8468 (2018) has the same three calls in ByteBuffersDirectory. The history shows the sequence, not what its authors intended. Neither issue discusses rename or concurrent use of one name.
• The author chose a loud failure for a concurrent change to source: the IllegalStateException("File was unexpectedly replaced") at :189, not a silent delete.
• Precedent. LUCENE-8318 (#9365), "The dest file might be deleted later by FSDirectory#rename", a single-threaded case of a later delete hitting a file that legitimately holds the name, was accepted and fixed in 2018.

The strongest case (an intended-behaviour devil's advocate and a review by another model family, OpenAI GPT-5.6): Directory promises no linearizability for overlapping calls on one name, rename may "not be truly atomic", and an application that re-creates a while renaming a away has a race of its own and should synchronize externally (Directory.java:43-45).

Why it fails: the relaxation is narrow (both names visible for a while) and says nothing about deleting a file the rename did not move; no serial order loses it; a writer that only calls createOutput("a") and retries on FileAlreadyExistsException, relying on nothing but createOutput's contract (Directory.java:86-87), still loses files; name reuse after a rename is normal (rename "is used by IndexWriter to publish commits", and MockDirectoryWrapper.rename drops source from its created set, MockDirectoryWrapper.java:257). Verdict: not intended, a defect (low severity).

One option was built and validated in a clean scratch worktree of 645d4fe14a20 with gradle; the alternative is not recommended. A separate, loud race stays after the fix and is out of scope: if another thread deletes and re-creates source between rename's get and its checked remove, rename throws IllegalStateException after dest is already published. The repro and the stress tests involve no concurrent deleteFile(source) or close(); those are separate races.

The checked remove(source, file) already completes the move; the unconditional remove(source) after it can only delete another thread's file.

     if (!files.remove(source, file)) {
       throw new IllegalStateException("File was unexpectedly replaced: " + source);
     }
-    files.remove(source);
   }

Trade-offs

None known: one line, no behaviour change without concurrency, nothing added to any hot path.

What the checks validated

• Stock 645d4fe14a20: the repro fails through gradle with assertions on and off; with the fix it passes, on and off.
• Every suite in org.apache.lucene.store.* of lucene/core (including TestByteBuffersDirectory and its testRename for all three heap variants, TestNRTCachingDirectory, TestFileSwitchDirectory) plus the repro: 29 suites, 777 tests, 0 failures, 17 skipped.
• TestIndexWriter*, TestDirectoryReader*, TestSegmentInfos and TestPersistentSnapshotDeletionPolicy with -Ptests.directory=ByteBuffersDirectory: 25 suites, 426 tests, 0 failures, 22 skipped (a bogus value of that option makes the tests fail, so it reaches the test JVM). This is compatibility coverage only: these tests usually get the directory wrapped in the synchronized MockDirectoryWrapper, so they show the fix breaks nothing, not that it closes the race.
• :lucene:core:check -x test (format, ecj lint, forbidden APIs) with the annotated test: passes.
• Earlier, with the same change shadowed in: the finder's, the independent reproducer's and the devil's advocate's stress tests all pass (20 of 20 method runs with assertions off and 10 iterations).

One lock around rename, createOutput and deleteFile, as MockDirectoryWrapper has, would also close the window.

Trade-offs

Adds contention to every file creation, and the line would still have no effect in single-threaded use. Not built.

Method: semi-formal (the hypothesis came from reading the code while modelling it; TLC then witnessed it).

• The finder, an AI tool, modelled the ConcurrentHashMap steps of createOutput, deleteFile and rename for two threads in TLA+ (BBDir.tla) with the invariant NoLostFile: every created file is either legitimately gone or bound to a name. TLC reported a violation in an 8-state trace on the stock steps; the model without the trailing remove is clean over 13,492 distinct states; a vacuity probe is violated as required.
• A two-thread public-API stress test on the stock jar then lost files in both shapes (a concurrent createOutput("a"), and a concurrent rename(tmp, "a")), and the same programs on FSDirectory kept every file.
• Independent reproducers wrote their own tests from the claim alone. The single-thread, forced-order repro was cut from them on 2026-10-29.