PersistentSnapshotDeletionPolicy never fsyncs the directory of its save file, so an OS crash can lose an acknowledged snapshot

PersistentSnapshotDeletionPolicy.snapshot(), release(IndexCommit) and release(long) are documented to return only once "the snapshot information is persisted in the directory". They write the new save file snapshots_N and fsync its bytes, but never call Directory.syncMetaData(), the call that makes a new file's name durable and that Lucene's own commit makes twice. After an OS crash or power loss on a file system that does not persist a new name together with the file's fsync, the save file can be gone: with the snapshots in their own Directory, getSnapshotCount() comes back as 0 and the reopened IndexWriter deletes the snapshotted commit; with the snapshots in the index Directory the next commit deletes it; a lost release() brings a released snapshot back. The live index stays consistent. It is a side-effect regression of LUCENE-5925 in 5.0.0, and the fix is one line.

• Snapshot: PersistentSnapshotDeletionPolicy (PSDP, @lucene.experimental) wraps another deletion policy and keeps every snapshotted commit until release(). It stores the reference counts per commit generation in a save file snapshots_N in a Directory of your choice: its own, or the index Directory ("It is OK to use the same Directory that holds the index", PersistentSnapshotDeletionPolicy.java:42).
• Durable name: Directory.sync() "Ensures that any writes to these files are moved to stable storage"; Directory.syncMetaData() "Ensures that directory metadata, such as recent file renames, are moved to stable storage" (Directory.java:104-119). On FSDirectory the first fsyncs each file (FSDirectory.java:243-251), the second fsyncs the directory (:268-274). The Linux fsync(2) man page: "Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk."

1. snapshot() (:113-126), release(IndexCommit) (:135-147) and release(long) (:156-159) update the counts and call persist().
2. persist() (:161-184) creates snapshots_<nextWriteGen> and writes the counts (:162-173), calls dir.sync(Collections.singletonList(fileName)) (:175), deletes the previous save file (:177-181) and returns. Nothing in the class calls syncMetaData(), and nothing ever did (git log -S syncMetaData on the file: no commits).
3. Lucene's commit does call it: SegmentInfos.prepareCommit() before writing pending_segments_N (SegmentInfos.java:970) and finishCommit() after renaming it to segments_N (:1005-1007), and by the second call every file of the commit, segments_N included, is already synced (IndexWriter.java:5993, SegmentInfos.java:620): commit does not rely on a file's sync to make its name durable.
4. So, when snapshot() returns, the save file's bytes are durable and its name is not. In the index Directory, the writer's next commit makes the name durable; a separate snapshot Directory is never fsynced by Lucene at all.
5. On restart loadPriorSnapshots() loads the newest snapshots_N it can list (:211-264). If the newest name was lost it loads nothing (a lost first snapshot) or the previous file (a lost later snapshot, or a lost release()).

An independent crash-model test keeps across a crash exactly the names covered by the last syncMetaData() and the bytes of synced files. On main 645d4fe14a20, 3 of 3 tests fail, and its 3 controls pass:

• Separate snapshot Directory: commit 1, snapshot() (writes snapshots_0), commit 2, crash. After the crash the snapshot directory is [], getSnapshotCount() is 0 instead of 1, and the reopened IndexWriter keeps commits [2] instead of [1, 2].
• Snapshots in the index Directory: a crash before the next commit loses the snapshot (count 0); the next commit then deletes commit 1 ([2]).
• release(): snapshots_1 is lost and the not-yet-durable delete of snapshots_0 is undone, so the released snapshot comes back (count 1 instead of 0). This is the less harmful direction: no commit is deleted, but the released commit's files and disk space are kept longer than asked.

On a real FSDirectory, with no crash model, a probe that records every FileChannel.force call sees commit() fsync the index directory (6 forces), while snapshot() and release() force only snapshots_0 and snapshots_1 and no directory, in both setups.

• Opt-in. The default IndexWriterConfig uses KeepOnlyLastCommitDeletionPolicy and is not affected. PSDP is @lucene.experimental (PersistentSnapshotDeletionPolicy.java:47).
• The trigger is a machine-level failure: an OS crash or kernel panic, a power loss, or a hard reset of the machine or VM, after snapshot() or release() returned and before the directory entry reaches disk. With the snapshots in the index Directory that window ends at the writer's next commit; with their own Directory it lasts until the OS writes the directory back.
• Process death is not a trigger. A probe called Runtime.halt() right after snapshot() or release() returned, with nothing closed, and reopened the index in a new JVM: 10 of 10 acknowledged states survived. kill -9 and JVM crashes are the same case by reasoning.
• File systems. POSIX and the Linux fsync(2) man page allow a new file's name to be lost although the file was fsynced; Lucene's commit defends against that (LUCENE-5588, LUCENE-8048 "Filesystems do not guarantee order of directories updates"). Mainstream journaling file systems such as ext4, XFS and btrfs are expected to persist the name along with the file's fsync, so the loss is not expected there; this is reasoning, not tested on a real power cut. On Windows Lucene fsyncs no directory at all (IOUtils.java:526-533).
• What is lost. An acknowledged snapshot record and, with it, the snapshotted commit: deleted when the IndexWriter reopens (separate Directory) or at the next commit (index Directory). A backup or replica still copying that commit loses its source. A lost release() keeps a commit longer than asked. The live index stays consistent and searchable.
• By reading only (not run): reopening with OpenMode.APPEND after losing the only save file throws IllegalStateException("no snapshots stored in this directory") (:101-103).
• Scale: none; one document, one commit and one snapshot() suffice. Likelihood: contrived (it needs all of the above at once).

Both directories are in memory (ByteBuffersDirectory), the test body calls no random() and uses a plain IndexWriterConfig; the test framework's randomized codec, locale and time zone do not touch the syncMetaData() count, so the seed does not change the result. The snapshots live in their own Directory only so that every syncMetaData() call the counter sees can come from the snapshot policy; the javadoc also allows the index Directory. Copy the file into lucene/core/src/test/org/apache/lucene/index/; no build change is needed. It fails with assertions on and off, and unchanged on the 10.6.0, 10.5.2 and 9.12.3 release jars. With the one-line fix it passes.

Regression test: this same test, as written.

TestPersistentSnapshotSyncMetaData > testSnapshotSyncsSaveFileName FAILED
    java.lang.AssertionError: expected >= 1 syncMetaData() call in snapshot(), got 0
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:D187E000AA5DDBBB]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.index.TestPersistentSnapshotSyncMetaData.testSnapshotSyncsSaveFileName(TestPersistentSnapshotSyncMetaData.java:48)

• The contract. "The snapshots are persisted in a Directory and are committed as soon as snapshot() or release(IndexCommit) is called" (PersistentSnapshotDeletionPolicy.java:35-37); "Once this method returns, the snapshot information is persisted in the directory" (:106-108, :128-130, :149-151). SnapshotDeletionPolicy: "This class maintains all snapshots in-memory, and so the information is not persisted and not protected against system failures. If persistence is important, you can use PersistentSnapshotDeletionPolicy" (SnapshotDeletionPolicy.java:35-37).
• Lucene's own protocol. Directory separates sync() (file writes) from syncMetaData() (directory metadata), and commit calls syncMetaData() twice (SegmentInfos.java:970, :1007), the second time after every file of the commit, segments_N included, is synced.
• The test intent. TestPersistentSnapshotDeletionPolicy checks // Make sure we fsync: dir.crash(); (:66-67), where MockDirectoryWrapper.crash() "Simulates a crash of OS or machine" (MockDirectoryWrapper.java:512). It cannot see this bug only because crash() damages or deletes only files that were never sync()ed (:356-358, :526) and never drops the name of a sync()ed file, whether or not syncMetaData() was called; its own TODO: "TODO: we should also mess with any recent file renames, file deletions, if syncMetaData was not called!!" (:339-340).
• The history. The file sync() came in 2013 as a "missing sync" (LUCENE-4976). In 4.8.0-4.10.4 the stock FSDirectory.sync() also fsynced the directory (LUCENE-5588), so the name was durable. LUCENE-5925 (2014, 5.0.0) moved that fsync into the new renameFile(), which only the segments_N publish calls; its discussion never mentions snapshots. PersistentSnapshotDeletionPolicy.java is identical in 4.10.4 and 5.0.0, and LUCENE-7373 (6.2.0, syncMetaData()) did not update it.

The strongest case: "persisted" only means surviving a JVM crash; the class was created (LUCENE-2481, 2010) "in case the JVM crashes". Also argued: sync() of a new file covers its creation; and the design is deliberate.

Why they fail: SnapshotDeletionPolicy sends users to this class for protection against "system failures"; a file sync() adds nothing against JVM crashes alone (the OS keeps the data when only the process dies, as the Runtime.halt probe shows); the upstream test simulates an OS or machine crash; Lucene's commit does not rely on sync() for names; and no issue, comment or CHANGES entry marks the loss as intended, which came as a side effect of an unrelated change. What remains is an argument about reach (mainstream journaling file systems are expected to keep the name), not about intent. Verdict: not intended, a regression.

One fix was built and validated through gradle; the alternatives are listed for completeness.

Make the new save file's name durable before the old file is deleted. The order is safe: if a crash keeps both files, loadPriorSnapshots() always loads the newest generation (PersistentSnapshotDeletionPolicy.java:218), so the new state wins and the delete need not be durable. It usually deletes the older file too (:253-260), but not when the older name sorts after the newer one in listAll() (string order, e.g. snapshots_9 after snapshots_10), which is harmless.

     dir.sync(Collections.singletonList(fileName));
+    dir.syncMetaData(); // make the new save file's name durable before deleting the old one
 
     if (nextWriteGen > 0) {

Trade-offs

One extra directory fsync per snapshot() or release(), operations that run per backup or replication round (a commit already makes two); no benchmark was run. With the snapshots in the index Directory it fsyncs the index directory, which is harmless. On Windows the directory fsync is skipped, as for commit. Applies to every maintained branch (syncMetaData() exists since 6.2.0).

What the checks validated

Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, 2026-10-31: the repro passes with assertions on and off (it fails on the unfixed tree); the repro with TestPersistentSnapshotDeletionPolicy and TestSnapshotDeletionPolicy passes 21 of 21 with seed DEADBEEF, assertions on and off, and 105 of 105 with -Ptests.iters=5 (a random master seed) (unfixed, the two suites also pass 20 of 20); :lucene:core:check -x test (format, ecj lint, forbidden APIs) passes. Earlier, the finder's, the independent reproduction's and the refute reviewer's crash-model tests passed with the same line.

Makes the delete durable too, so a crash never leaves two save files.

Trade-offs

Not needed for the new state to win (the loader always loads the newest file); needed only if the old file's removal must be durable at once. Costs a second directory fsync.

Change the javadoc instead of the code.

Trade-offs

Weakens a documented guarantee ("persisted once this method returns") and leaves existing users exposed.

Call snapshotDir.syncMetaData() right after snapshot() and after release() (public API).

Trade-offs

Every application has to know about it. Under the crash model this control passes (explicit syncMetaData() after snapshot()).

Method: semi-formal (bounded-exhaustive checking of the real jar).

• An AI finder built a recording Directory that logs every create, close, sync, syncMetaData, rename and delete, and a crash model: file bytes are durable only after sync, directory operations only after the next syncMetaData() (the rule Lucene's commit code follows, LUCENE-5588 and LUCENE-8048, and the Linux fsync(2) man page). Every prefix of the recorded operations is a crash point; for each, the not-yet-durable directory operations survive as a prefix in program order (an ordered-metadata model), with content variants for unsynced files.
• The oracle is real Lucene code on each crash image: DirectoryReader.open, CheckIndex, IndexWriter recovery and, for the snapshot workload, the reloaded snapshot state, which must be the last acknowledged one or an in-flight one.
• In the snapshot workload (272 file-system operations, 8 syncMetaData() calls, 5 commits, 4 snapshot operations) 8 of 1,123 distinct crash states failed: 4 right after a snapshot() or release() returned (this bug; crash points op#68, #190, #195 and #262) and 4 inside persist() before its file sync (a sibling bug, in which the loader adopts a broken newest save file; this fix alone does not remove those). Vacuity mutants of the harness (for example, never syncing .cfs files) are caught, and with the unit's fix sketches (this fix plus the sibling's) the workload has 0 failures.
• The finder then wrote a public-API JUnit check; independent reproducers wrote their own tests from the claim. The one-test repro was cut from the finder's protocol check on 2026-10-29.