NRTCachingDirectory: an OS crash during commit can leave an index that will not open

NRTCachingDirectory keeps small newly flushed segment files in RAM and writes them to the wrapped directory when IndexWriter.commit() calls Directory.sync. Since LUCENE-8048 (7.2), a commit fsyncs the index directory in SegmentInfos.prepareCommit so that every name it references is durable before pending_segments_N is renamed to segments_N, the step that publishes the commit. IndexWriter.startCommit calls directory.sync only after that fsync, so with NRTCachingDirectory the new segment files are created on disk after the barrier, and the next directory fsync comes only after the rename. On a file system that persists that rename before the earlier creates (the model LUCENE-8048 adopted; ext4 data=ordered and XFS are not expected to do it), an OS crash or power loss in that window leaves a newest segments_N whose files are missing: DirectoryReader.open, listCommits and new IndexWriter all throw until someone deletes it by hand, which recovers the previous commit. A plain FSDirectory passes the same crash test: its files exist before the barrier. One deterministic test shows it on main, 10.6.0 and older releases; all 71 releases from 7.2.0 to 10.5.2 fail the hunt's longer tests. One directory fsync in NRTCachingDirectory.sync when it wrote cached files fixes it.

1. The barrier. SegmentInfos.prepareCommit (SegmentInfos.java:966-972) calls dir.syncMetaData() (:970, added by LUCENE-8048) and then write(dir), which writes and fsyncs pending_segments_N (:620). Every name that exists at :970 is durable from then on.
2. The order. IndexWriter.startCommit calls toSync.prepareCommit(directory) (IndexWriter.java:5974) and only afterwards filesToSync = toSync.files(false); directory.sync(filesToSync); (:5992-5993).
3. What is cached. NRTCachingDirectory.createOutput sends a file to the RAM cache when doCacheWrite accepts it (NRTCachingDirectory.java:159-163): a flush or merge whose estimated size is at most maxMergeSizeMB while the cache stays under maxCachedMB (:231-245). So a small new segment is not in the wrapped directory at step 1.
4. Where the files are created. NRTCachingDirectory.sync (:169-178) calls unCache for each name; unCache creates the file in the wrapped directory with in.copyFrom (:305-323, the copy at :319). Then in.sync(fileNames) (:177) fsyncs the files; with an FSDirectory delegate, their bytes only (FSDirectory.sync, FSDirectory.java:244-251). The class itself never calls syncMetaData.
5. The publish. SegmentInfos.finishCommit (:996-1012) renames pending_segments_N to segments_N (:1005) and only then fsyncs the directory (:1007).
6. Result. Between :1005 and the end of :1007, the rename and the creates of step 4 are both pending, with no order between them. If the rename reaches the disk first and the machine crashes, the newest segments_N names files that are not there (in the test: _1.si, _1.cfs, _1.cfe). With a plain directory those files already exist at step 1, so the plain control passes.

MockDirectoryWrapper.crash() damages or deletes only files whose bytes were never fsynced and never undoes creates, renames or deletes that lacked a directory fsync: "TODO: we should also mess with any recent file renames, file deletions, if syncMetaData was not called!!" (MockDirectoryWrapper.java:339-340, :526). TestNRTCachingDirectory has no crash test.

• Who. Applications that write through NRTCachingDirectory, the near-real-time setup its javadoc shows (new NRTCachingDirectory(fsDir, 5.0, 60.0), NRTCachingDirectory.java:50-58). Opt-in and @lucene.experimental (:63); not the default Directory. Per the prior-report search notes, Solr's _default configset uses NRTCachingDirectoryFactory; whether Solr's commit path reaches this was not checked.
• Trigger. A machine-level failure (OS crash, kernel panic, power loss) inside commit(), between the segments_N rename and the directory fsync after it, on a file system that persists that rename before the earlier creates. POSIX and Linux fsync(2) do not promise that order; it is the model LUCENE-8048 accepted for plain directories. Mainstream ordered-metadata journaling (ext4 data=ordered, XFS) is not expected to reorder like this, and no physical file system was tested. A JVM crash or kill -9 cannot produce the state: completed creates and renames stay visible to the next process whether or not they were fsynced.
• Effect. Index unusable. In an independent reproducer's probe on the crash state, DirectoryReader.open, DirectoryReader.listCommits and new IndexWriter(CREATE_OR_APPEND) all throw; deleting the newest segments_N by hand recovers the previous commit (after deleting segments_2 the index opens with maxDoc=10). Whoever does that should back up the index first and make sure no reader or writer has it open. Fail-stop: in every enumerated crash state this bug causes, opening throws; none returned wrong or deleted documents. One commit that contains a cached segment is enough; whether a segment is cached depends on the flush or merge context, its estimated size and the cache's free space (NRTCachingDirectory.java:231-245).
• How likely. Rare; no field report attributable to it was found.

The crash state is the one LUCENE-8048's model allows: the rename reached the disk, and no name created since the last directory fsync did. Every file's bytes are kept, which is generous to Lucene. Skipping the rename source while copying is part of the model: with either fix pending_segments_2 is durable before the rename, on stock code it never is.

It is a crash-model test, not a physical crash: no file system was crashed. It covers only the second commit, because a crash during the very first commit may leave a corrupt index by design (LUCENE-4738). The seed only drives the test framework's randomisation; the test also fails with seeds 1234ABCD and C0FFEE01. It passes checkGoogleJavaFormat, ecjLintTest and forbiddenApisTest as is.

Regression test: this same test; it passes with either fix.

cp $REPRO/TestNRTCachingDirectoryCommitCrash.java $LUCENE/lucene/core/src/test/org/apache/lucene/store/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestNRTCachingDirectoryCommitCrash -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same failure)

# gradle, scratch worktree of 645d4fe14a20, -Ptests.seed=DEADBEEF (assertions on; the same with -Ptests.asserts=false)
java.lang.AssertionError: crash state (names durable at the last directory fsync, plus the segments_2 rename)
  should open at commit 1 or 2, but: org.apache.lucene.index.CorruptSegmentInfoException: segment info file:
  _1.si cannot be read - it may be missing or corrupt (resource=BufferedChecksumIndexInput((clone of)
  ByteBuffersIndexInput (file=segments_2, buffers=246 bytes, block size: 1, blocks: 1, position: 0)))
    at org.apache.lucene.store.TestNRTCachingDirectoryCommitCrash.testCrashRightAfterSecondCommitRename(TestNRTCachingDirectoryCommitCrash.java:64)
:lucene:core:test (FAILURE): 1 test, 1  failure

# released jars, standalone runner, seed DEADBEEF
10.6.0 (-ea, -da)          ... but: org.apache.lucene.index.CorruptSegmentInfoException: segment info file: _1.si cannot be read ...
10.5.2, 9.12.3, 9.1.0 (-da) ... but: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. ...

# crash-state files (stock main): [_0.cfe, _0.cfs, _0.si, segments_1, segments_2]
# independent impact probe on that state: DirectoryReader.open, DirectoryReader.listCommits and
#   new IndexWriter(CREATE_OR_APPEND) throw CorruptSegmentInfoException;
#   after deleting segments_2 by hand: opens, maxDoc=10 (commit 1 of that probe's 10-documents-per-commit test)

# with fix A or fix B: 1 test, 0 failures (assertions on and off)

• Contracts. NRTCachingDirectory: "This is safe to use: when your app calls {IndexWriter#commit}, all cached files will be flushed from the cached and sync'd" (NRTCachingDirectory.java:47-48). IndexWriter.commit: commits "and syncs all referenced index files, such that a reader will see the changes and the index updates will survive an OS or machine crash or power loss", and Directory#sync "should not return until the file contents and metadata are on stable storage" (IndexWriter.java:4188-4201). Directory.sync: "Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index" (Directory.java:104-112).
• The design. The commit is published by one rename (Directory.java:131: rename "is used by IndexWriter to publish commits"). LUCENE-8048 (#9096, CHANGES 7.2.0 "Bug Fixes") added dir.syncMetaData() to prepareCommit for this failure on plain directories; its reporter's fix was "sync index directory after all segments have been written", and with NRTCachingDirectory the segments are written to disk only inside sync(), after that barrier.
• History (read, not executed). In 4.8-4.10 FSDirectory.sync itself fsynced the directory after the files (LUCENE-5588), which covered the files NRTCachingDirectory had just created; 5.0 (LUCENE-5925) moved that fsync after the rename for every directory; 7.2 (LUCENE-8048) restored a barrier only before the point where NRTCachingDirectory creates its files. No javadoc, CHANGES or MIGRATE entry, test or discussion exempts NRTCachingDirectory.
• Tests. No upstream test models lost directory updates (the MockDirectoryWrapper TODO above). Scope: only the second-commit case is claimed, because a crash during the very first commit may leave a corrupt index by design (LUCENE-4738, TestIndexWriterOnJRECrash.java:164-170).

The strongest case: Directory.sync promises only file bytes ("any writes to these files are moved to stable storage", Directory.java:105) and names are syncMetaData's job; the crash falls inside an in-flight commit, before commit() returns; the class is @lucene.experimental; and a committer on LUCENE-8048 (Robert Muir) said a file system that behaves this way "is choosing to corrupt users data" and agreed to the barrier "just to be safe" (#9096).

Why it fails: IndexWriter.commit's javadoc asks Directory#sync for "file contents and metadata" (not decisive alone: sync and syncMetaData are separate APIs, Directory.java:104-119), and either way the combination breaks commit's documented promise to survive an OS crash or power loss, and the class's "safe to use"; no javadoc says which commit an interrupted commit leaves, but the pending_segments_N + rename protocol and LUCENE-8048 are built so that it leaves the old one or the new one, and here the index opens to neither, which is what LUCENE-8048 set out to prevent; @lucene.experimental marks API stability, not a durability exemption; and committers still made the barrier part of the protocol. Reachability lowers impact; it does not make the gap intended. Verdict: not intended, a defect. Cross-model reviewers (OpenAI GPT-5.6) rated the case for "intended" moderate and returned "real"; an intended-behaviour devil's advocate found that none of its seven points survives and returned "real". No physical file system was crashed, so the real-world reach may be close to zero on ordered-metadata file systems.

Both fix sites were built and run on a scratch worktree of 645d4fe14a20. A keeps the extra fsync to the affected class; B also covers any other Directory whose sync creates files. Offer both; maintainers may prefer either. Neither was benchmarked.

unCache returns true when it copied a cached file into the wrapped directory (false when the file was not cached), and sync calls in.syncMetaData() after in.sync(fileNames) when it did. Same change as the hunt's and the independent reproducer's fix.

boolean created = false;
for (String fileName : fileNames) {
  created |= unCache(fileName);
}
in.sync(fileNames);
if (created) {
  // files created in the delegate here must have durable directory entries before the
  // commit's segments_N rename (the LUCENE-8048 barrier ran before they existed)
  in.syncMetaData();
}

Trade-offs

• One extra directory fsync per commit, only when the commit had cached files to write, in a class meant to cut small-file I/O. It runs inside startCommit, the first phase of the two-phase commit, which the prepareCommit javadoc says "syncs the index files" (IndexWriter.java:3721-3725).
• Fixes only NRTCachingDirectory; another wrapper whose sync creates files would need the same.
• Variant: call in.syncMetaData() unconditionally (simpler, but a directory fsync on every sync call, including SegmentInfos.write's sync of pending_segments_N, SegmentInfos.java:620: two per commit).

What the checks validated

• The v2 re-run (own scratch worktree of 645d4fe14a20, removed afterwards; gradle, JDK 25, seed DEADBEEF): the minimal test fails on the stock worktree (1 test, 1 failure, the documented CorruptSegmentInfoException on _1.si); with the fix, the minimal test, TestNRTCachingDirectory, TestIndexWriterCommit, TestCrash, TestCrashCausesCorruptIndex and TestSegmentInfos pass (6 suites, 89 tests, 4 skipped, 0 failures); assertions off, the minimal test and TestNRTCachingDirectory pass (61 tests, 2 skipped, 0 failures). The v2 fact check repeated that last pair with assertions on in its own worktree (61 tests, 2 skipped, 0 failures).
• Earlier the same day, in another scratch worktree: :lucene:core:test for org.apache.lucene.store.*, TestIndexWriterCommit, TestCrash, TestCrashCausesCorruptIndex, TestSegmentInfos and TestIndexFileDeleter, assertions on: 34 suites, 816 tests, 19 skipped, 0 failures (repeated by the independent fact check); assertions off: the minimal test and TestNRTCachingDirectory (61 tests) pass; checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass.
• Earlier still: the hunt's, the independent reproducer's and the refute reviewer's tests pass with it, and in the bounded crash enumeration the 31 reader-open failures of the nrtcaching workload disappear.
• Not run: the full Lucene suite, benchmarks, concurrent use (validated single-threaded only). A thread whose unCache finds the file already moved ("Another thread beat us", NRTCachingDirectory.java:313) returns false, skips the directory fsync and may return before the other thread's fsync. IndexWriter's own commits do not race this way (startCommit, IndexWriter.java:3915, runs inside synchronized (commitLock), :3775); the unconditional variant avoids the question.

Call dir.syncMetaData() just before dir.rename(src, dest) in finishCommit (+4 lines with the comment). If it throws, the existing catch rolls the commit back (SegmentInfos.java:1014-1017).

// Directory#sync may create files (NRTCachingDirectory writes its cached files to the
// delegate there), after the directory sync in prepareCommit: make every name this commit
// references durable before the rename publishes it
dir.syncMetaData();
dir.rename(src, dest);

Trade-offs

• Covers any Directory whose sync creates files, not only NRTCachingDirectory.
• One extra directory fsync per commit for every Directory, including plain FSDirectory, in the second phase of the two-phase commit.
• Variant (not run): move the prepareCommit barrier here instead of adding one; that keeps today's fsync count but takes that fsync out of the first phase.

What the checks validated

• Earlier on 2026-10-30, scratch worktree of 645d4fe14a20 (fix A reverted), gradle, JDK 25, seed DEADBEEF: the minimal test passes, assertions on and off.
• The same 34 store and commit suites as for fix A, assertions on: 816 tests, 19 skipped, 0 failures, including TestNRTCachingDirectory (60 tests, 2 skipped); checkGoogleJavaFormat passes.
• Assertions off: the minimal test and TestNRTCachingDirectory pass (61 tests); ecjLintMain and forbiddenApisMain pass. The independent fact check repeated the minimal test and TestNRTCachingDirectory with assertions on and off.
• Not run: the full Lucene suite, benchmarks, the hunt's crash enumeration.

Method: bounded crash-state enumeration of the real jar (the hunt classes it as semi-formal: the first signal came from the executable enumerator, not from a formal tool).

• Enumerator (2026-10-26). A recording Directory under IndexWriter logs every create, rename, delete, file fsync and directory fsync. For every crash point it builds each crash state allowed by a bounded cover of the directory operations pending since the last directory fsync, and checks that the index opens to an allowed commit with the right content, that CheckIndex is clean and that IndexWriter recovers. The nrtcaching workload was added after reading NRTCachingDirectory.sync while modelling which Directory wrappers create files late. Under the unordered model 106 of 6,340 states failed (0 of 2,350 under the ordered model); this bug's fix alone removes the 31 reader-open failures, and the remaining 75 belong to a separate IndexFileDeleter durability issue, tracked on its own.
• TLA+ model. CommitDurability.tla (TLC): ReaderOK is violated for NRTCachingDirectory under unordered persistence and holds under ordered persistence and with the fix.
• Executed tests. The finder's JUnit, an independent reproducer's JUnit and a cross-model reviewer's JUnit fail on stock main and pass with the fix; the minimal repro above is the smallest of them.